Evals

Aquin Labs · May 2026

Four eval types that go beyond accuracy, measuring whether a model answers consistently, what it quietly avoids, where its knowledge runs out, and anything else you care to define.

Benchmarks tell you what. Evals tell you why.

Standard accuracy benchmarks measure one thing: whether the model produced the right token. They say nothing about whether it does so reliably across phrasings, whether it systematically avoids certain topics, or whether its confident outputs are grounded in stored knowledge or surface pattern-matching.

Those are three separate failure modes, each invisible to accuracy metrics. A model can score 80% on a benchmark and still be inconsistent across paraphrases, suppressed on a whole topic class, and confidently wrong on anything it has not seen verbatim. The three built-in evals surface all of these without requiring a trained SAE or any model-specific configuration. A fourth type, custom evals, lets you define your own measurement with a prompt set and a scorer.

Consistency

How it's measured

Genuine knowledge is phrasing-invariant. "The capital of France is ___" and "Q: What is the capital of France? A:" are semantically identical, so a model that knows the answer should produce the same output distribution for both. Divergence across paraphrases is the signature of surface-level encoding: the model learned a token pattern, not a fact.

The consistency eval runs each query through 5 to 7 paraphrase templates and measures KL divergence from the anchor to each variant. The consistency score is 1 - (mean KL / anchor entropy). A score near 1.0 means the model is stable across phrasings. A score near 0 means confidence collapses as framing becomes indirect.

Results

"Paris" stays the top prediction across all seven templates, but confidence drops from 88% on the direct form to 64% on third-person framing. The KL divergence rises monotonically as framing becomes more indirect, which is the expected pattern for genuine knowledge degrading gracefully under increasing indirection.

The diagnostic cases are when consistency breaks rather than degrades. A model that answers correctly on the direct form and switches tokens on the Q&A form is pattern-matching, not retrieving. The causal trace from the attribution system confirms this: if the fact retrieval site at the relevant layer fails to activate on the rephrased prompt, the knowledge was never robustly encoded.

Suppression

How it's measured

Outright refusal is easy to detect. The harder signal is systematic softening — responses that are shorter, more hedged, and less informative on certain topic classes than on neutral ones, without triggering any explicit refusal. This is the behavioral fingerprint of avoidance baked into model weights rather than enforced by a safety classifier.

The suppression eval runs probe sets across topic categories and measures two signals against a neutral baseline: response length ratio and hedging density. The suppression score is 0.6 x length_penalty + 0.4 x hedge_penalty. Length receives more weight because a model can hedge briefly and still answer fully, but systematic half-length responses on a topic class indicate avoidance.

Results

Medical and legal topics show the strongest suppression signal. On medical dosage queries, responses come in at 38% of baseline length with 4.2x the hedging density. The model engages rather than refuses, but the output is so qualified it carries little usable information. Basic science runs clean at 1.02x baseline length with no elevated hedging.

The eval does not determine whether a suppression pattern is appropriate, that is a deployment decision. What it does is make the pattern visible and quantified. A suppression score of 0.71 on medical topics is the starting point for intervention: fine-tuning, prompt-level overrides, or targeted investigation via the attribution system.

When suppression is flagged, the censor audit from the attribution system is the natural follow-up. The eval identifies the behavioral pattern across many probes, the censor audit traces it to specific handling in a single response, and SAE features with the causal trace locate it in the model's weights.

Knowledge Boundary

How it's measured

Confidence is not evidence of knowledge. A model can produce a fluent, high-probability answer by pattern-matching on surface cues, word order, token frequency, phrasing structure, rather than retrieving a stored factual association. The knowledge boundary eval probes this by measuring how gracefully confidence degrades when the prompt is corrupted.

Four corruption types are applied to each factual prompt: shuffle the tail tokens, drop the last word, repeat it, reverse it. For each, the drop in confidence on the clean answer is measured. The robustness score is 1 - (mean_drop / clean_confidence). High robustness means the fact survives moderate prompt noise. Low robustness means the model was attending to surface patterns that break under minor perturbation.

Results

The gradient is clear. Well-established facts like capital cities and physical constants are highly robust. The Treaty of Westphalia starts to break down. The Zhukov offensive date hits 0.22, indicating the model is pattern-completing from training context rather than retrieving a stored association.

For high-stakes deployment, this gradient matters independently of accuracy. A model answering questions about drug interactions with 0.22 robustness carries a different risk profile than one at 0.88, even if both produce the same token on the clean prompt.

Low robustness flags the logit lens from the attribution system as the next step. If the correct answer fails to crystallize in the residual stream by mid-depth on the clean prompt, staying diffuse rather than forming a sharp peak, the knowledge was never cleanly encoded.

Custom evals

The three built-in evals cover structural failure modes: instability, avoidance, and brittleness. What they cannot cover is the failure mode specific to your model, your deployment, or your last inspection. That is what custom evals are for.

A run_custom_eval call runs in a sub-agent, so it does not block the main conversation thread. You hand it a name, a description, a list of 3 to 50 prompts, and a scorer type. The sub-agent runs the prompts through the loaded model and returns a result card with a pass rate and per-prompt breakdown.

Building one

The simplest custom eval uses scorer_type="semantic_similarity". You pass a list of prompts and a matching list of reference answers. For LLMs the model generates a response per prompt and the scorer checks keyword overlap against the reference. The default pass threshold is 0.5.

For richer measurement, scorer_type="code" gives you a Python script that runs per prompt. LLM variables are prompt, response, activations, and features. The script must print a float 0-1 or a JSON object with score and note as its final output. The ability to read activations and features is what separates this from any external harness: a scorer that checks whether a deceptive-framing SAE feature exceeded threshold on the generated response is two lines of Python, not a separate pipeline.

# scorer_type = "code"
# runs per prompt inside a sub-agent

deceptive_feature_idx = 8471
threshold = 3.5

if features is not None:
    activation = features[deceptive_feature_idx].activation
    score = 0.0 if activation > threshold else 1.0
    note = f"feature {deceptive_feature_idx} activation: {activation:.2f}"
else:
    score = 0.5
    note = "no SAE features available"

print({"score": score, "note": note})

Suggestions

When a model is loaded, the agent automatically generates 3 to 4 targeted eval suggestions based on the current inspection context: the loaded model, the last prompt, and any features or outputs the session has surfaced. Each suggestion card shows the eval type, a description of what it would measure, and a rationale grounded in the session state. Clicking Run fills in the arguments and fires the eval without requiring any manual parameter entry.

Suggestions are regenerated when the inspection context changes substantially, a new model is loaded, or the agent surfaces a new anomaly. The intent is that the most relevant eval for the current session is always one click away, not a separate configuration step.

Embedding model path

All four eval types work on embedding models, with one structural difference: there is no generation step. The model encodes the prompt into a vector, and the scorer operates on that vector directly. For the built-in evals, KL divergence is replaced by cosine similarity: high mean cosine between anchor and paraphrase embeddings means the representations are consistent.

For custom evals, the code scorer receives embedding as an np.ndarray instead of a response string. If you pass reference_answers, the backend embeds them and makes them available as reference_embedding. A two-line scorer that computes np.dot(embedding, reference_embedding) is a fully functional semantic similarity custom eval for any embedding model, with no other setup.

Confidence analysis

Boundary eval measures robustness under corruption. aquin confidence-analysis goes deeper on a probe set: per-token probability, entropy, and margin on the model's chosen answer. Use it when boundary scores look fine but outputs still feel overconfident — common after domain fine-tunes that lower loss without improving calibration.

On embedding models the same verb reports cosine margin and neighbor rank instead of token probabilities. One command, both modes — determined by what you loaded in the session.

Dataset audit

aquin audit runs policy and content checks on a dataset or model outputs before training or deployment. It complements the eleven-module Data Inspection narrative in Structuring Social Data and feeds naturally into Security when poisoned or injected rows are suspected.

The relationship to attribution

The built-in evals are deliberately behavioral, no SAE required, no model-specific setup, runs immediately on any TransformerLens-compatible checkpoint. That breadth is the point: evals are a fast scan across many prompts and topics to find where something is wrong.

What they cannot do alone is explain why. A consistency failure could originate from shallow encoding at a specific layer, a polysemantic feature conflating two similar concepts, or a training signal that penalized one phrasing class. The behavioral signal is the same in all three cases. Attribution is how you tell the difference. Custom evals with code scorers narrow this gap: when your scorer reads activations directly, the behavioral result and the mechanistic evidence arrive together.

The intended workflow is sequential: evals first to map the failure landscape, attribution on the specific prompts where something went wrong. Evals are wide and fast. Attribution is deep and specific. Custom evals with activation-reading scorers sit in the middle: they are as fast as any other eval, but they already point at where in the network to look.