How to Score Prompts Without AI Reliably

How to Score Prompts Without AI Reliably

A prompt that sounds precise can still fail the only test that matters: whether its output is usable in a real workflow. To score prompts without AI, treat them less like creative writing and more like an interface contract. Define the expected behavior, supply representative inputs, and measure results with checks a person or script can reproduce.

That approach is less glamorous than asking another model to judge the first one. It is also easier to audit, cheaper to run, and far more useful when a score needs to explain a release decision. A number without a traceable failure reason is not evaluation. It is decoration.

What Are You Actually Scoring?

“Prompt quality” is too broad to be useful. A prompt can be well structured and still produce the wrong JSON shape. It can return technically correct answers while violating a hard privacy requirement. It can perform well on a happy-path example and fail as soon as a field is missing.

Start by separating two related but different targets. The first is the prompt itself: does it contain clear instructions, explicit constraints, required input context, and an output contract? The second is prompt behavior: when run against a defined set of inputs, does it produce acceptable outputs?

Static prompt checks are fast and deterministic. They catch missing constraints, conflicting instructions, undefined placeholders, and ambiguous output requirements. Behavioral checks require executing the target model, but the scoring process itself does not need an AI judge. A JSON schema validator, a test assertion, a regular expression, a compiler, or a human reviewer with a fixed rubric can evaluate the result.

This distinction matters because no amount of wording analysis proves that a prompt will behave correctly. A prompt is executable configuration. Test it like one.

Build a Deterministic Prompt Scoring Rubric

A useful rubric has gates, weighted criteria, and evidence for every deduction. The goal is not to produce a universally meaningful score out of 100. The goal is to make prompt changes comparable and failures actionable.

Set non-negotiable gates first

Some failures should make a prompt unusable regardless of its average score. If a support-assistant prompt exposes secrets, a code-generation prompt emits invalid syntax, or an extraction prompt returns prose instead of required JSON, there is no reason to offset that failure with points for tone.

Define these as gates. A failed gate produces a fail status, even if the weighted score remains high. Common gates include output parsing, required-field presence, schema validity, policy compliance, and maximum response length.

Keep gates narrow and testable. “Must be safe” is not a test. “Must not include an API key present in fixture input” is a test. “Must return valid JSON matching this schema” is a test.

Score the criteria that can vary

Once hard requirements pass, score the dimensions that represent quality trade-offs. For a structured extraction prompt, that might be completeness, field accuracy, normalization, and handling of unknown values. For a coding prompt, it may be build success, test pass rate, minimal diff size, and adherence to an API contract.

Assign weights based on operational cost. A wrong currency code in an internal dashboard may be minor. A wrong package version in a dependency update workflow can break builds or hide a security issue. Weight the second failure accordingly.

A simple formula is enough:

score = sum(weight × criterion result) / sum(weights) × 100

Use criterion results from 0 to 1. A binary test is either 0 or 1. A partial criterion can use a defined scale, such as 0, 0.5, or 1. Avoid granular scales that reviewers cannot apply consistently. The difference between 0.7 and 0.8 usually reflects confidence theater, not measurement.

Make every point inspectable

A score should retain its evidence: fixture ID, expected result, actual result, rule evaluated, and failure message. Store that alongside the prompt version, model identifier, generation parameters, and test harness version.

Without that context, a regression is hard to reproduce. Model behavior can change between versions, and a prompt edit may only expose a problem under a specific input shape. Versioned fixtures turn “the score dropped” into “fixture missing-address-03 now invents a ZIP code.” That is something an engineer can fix.

Use Fixtures That Resemble Production

The quality of prompt evaluation is bounded by the quality of its fixtures. Ten polished examples copied from a product spec will produce a flattering score and little else.

Build a small, deliberate fixture set around the failure modes your workflow actually sees. Include at least these four categories:

  • Normal inputs that represent the common case.
  • Boundary inputs with empty fields, large values, or maximum-length text.
  • Malformed or contradictory inputs that test whether the prompt fabricates an answer.
  • Adversarial inputs that attempt to override instructions, inject irrelevant content, or exploit ambiguous formatting rules.

For each fixture, specify the expected output as precisely as practical. Exact string matching works for stable, narrow outputs. For broader outputs, use machine-checkable properties: valid JSON, required keys, allowed enum values, no extra text, a successful compile, or a matching record count.

Not every requirement can be reduced to an assertion. If you are scoring an explanation intended for a developer, readability and technical judgment may require human review. That is fine. Use a short reviewer rubric with concrete anchors. For example, “identifies the failing command, probable cause, and next action” is more reliable than “is helpful.”

Score Prompt Structure Before You Run It

Behavioral tests are the final authority, but static checks catch cheap mistakes early. A prompt linter can flag missing placeholders, inconsistent variable names, unbounded requests, or output instructions that contradict each other.

Consider a prompt for dependency triage. If it tells the model to “recommend an upgrade” but does not provide the installed version, lockfile context, compatibility constraints, or desired output format, the prompt is under-specified. A model may still return plausible text. That does not make the prompt reliable.

Static scoring can evaluate whether the prompt declares its inputs, identifies its expected output, limits unsupported claims, and specifies behavior for absent or uncertain data. These are not guarantees of correctness. They are contract checks that lower the probability of obvious failure.

Do not reward prompt length by default. Long prompts often hide duplicated instructions and unresolved conflicts. The better question is whether every instruction changes behavior or protects a known edge case. If it does neither, remove it and rerun the fixtures.

Avoid Scores That Hide Regressions

A single aggregate score is useful for dashboards, but it can conceal exactly the failure you need to notice. A prompt that improves formatting across twenty fixtures while failing one critical security case may score higher than the previous version. It is still worse.

Report three things together: gate status, overall weighted score, and per-category results. A compact result might read: PASS, 86/100, schema 100, accuracy 90, abstention 60. The abstention score tells the story. The prompt answers when it should say it lacks enough information.

Also compare candidate prompts against a baseline, not only against an arbitrary threshold. A score of 82 may be acceptable if the prior prompt scored 71 on the same fixtures and no gates changed. It may be a regression if the prior score was 91. Keep the benchmark fixed long enough for comparisons to mean something.

Temperature and sampling settings deserve the same discipline. If a production prompt runs with nonzero temperature, evaluate repeated runs rather than a single output. Score pass rate across runs and track variance. A prompt that passes once but fails one out of five times is not deterministic enough for many automation paths.

Know When Deterministic Checks Are Not Enough

Some tasks have multiple valid answers. Design feedback, open-ended research summaries, and product copy cannot always be judged by schemas or exact fixtures. Trying to force these tasks into a fake numeric precision creates brittle tests and encourages superficial optimization.

Use deterministic checks for what is objective: formatting, factual citations when available, prohibited claims, word limits, required sections, and structured data. Then reserve human review for the subjective part. Reviewers should see the same prompt version, input, output, and rubric, not a detached score.

This hybrid method is slower than fully automated judging, but it prevents a common failure mode: optimizing prompts to satisfy another model’s preferences rather than the user’s requirements. For high-frequency, low-risk tasks, strong mechanical checks may be enough. For customer-facing or high-impact work, a small human sample is usually worth the cost.

The useful endpoint is not a perfect prompt score. It is a prompt change you can ship with a clear answer to three questions: what it was tested against, what it still fails, and whether those failures are acceptable for the workflow.

Promptivo

A deterministic scorer for AI prompts. It runs with no model in the loop, so the same prompt always scores the same, and no model ever reads what you paste.