Transformers & LLMs

Aquin supports the full transformer family: dense LLMs and Mixture-of-Experts models. Every tool in the platform is architecture-aware from the moment you load a model.

One platform, every transformer architecture

Most LLM tooling is built around a single architecture class. Interpretability libraries assume dense models. Fine-tuning frameworks add MoE support as an afterthought. Evaluation pipelines treat model architecture as invisible.

When you load a model, Aquin detects whether it is a dense LLM or a Mixture-of-Experts model. Attribution, training monitoring, evals, benchmarks, and security analysis all adapt to what they are analyzing. A Mixtral run and a Llama run go through the same interface and the same tool set. The platform handles the architectural differences internally.

Architecture support

The transformer is the shared foundation. Every model Aquin supports, dense or sparse, is built on the same core building blocks: multi-head self-attention, a residual stream that carries information across layers, layer normalization, and positional embeddings. The architecture families differ in what happens in the feed-forward sublayer and how those sublayers are connected, not in the attention mechanism itself.

Attribution, SAE analysis, attention inspection, and training signals all hook into the parts of the transformer that are universal. The feed-forward layer is where the families diverge, and Aquin handles each variant at that layer specifically, without changing the interface or the output format.

Dense LLMs

A dense LLM is a standard transformer where every token activates every parameter on every forward pass. All attention heads and all feed-forward neurons run regardless of input. Dense models are the baseline of the transformer family. Llama, Mistral, Phi, Falcon, Qwen, GPT-2, OPT, and Pythia all fall here.

Every Aquin tool was built natively on dense LLMs. The residual stream is a single coherent vector at each layer, attribution runs across the full MLP and attention stack, and the SAE is trained on residual stream activations at a selected peak layer. Fine-tuning with LoRA, QLoRA, or full-parameter updates all go through the same training monitor setup.

Mixture-of-Experts LLMs

Mixture-of-Experts models replace dense feed-forward sublayers with a pool of N expert networks and a learned expert router. Each token's hidden state passes through the router, which selects k experts, typically 2 of 8, or 2 of 64 in DeepSeek-style configurations, to process it. Only those experts run per token. A 46B-active-parameter Mixtral model has the inference cost of a 12B dense model while having the total stored capacity of a 46B one.

Attention layers in MoE models are almost always dense. The sparsity is in the feed-forward sublayers only. This means all of Aquin's attention-level tools apply identically to MoE attention blocks. The additional MoE-specific signals, expert load balance, router assignment distribution, per-expert gradient norms, are tracked automatically when a sparse layer is detected in the loaded model.

Model inspection signals

Attribution tells you which parts of a model's computation explain a specific output. Inspection signals go a level deeper, describing the structural and geometric properties of the model itself, independent of any particular prompt. These are the signals that tell you whether a model is healthy before you run a single eval, whether a fine-tune degraded its representations, and whether specific layers or heads have collapsed.

Every signal below runs on loaded checkpoints across all supported architectures. For MoE models, per-expert variants of the weight-level signals are available alongside the standard layer-level view.

OOD similarity

OOD similarity measures how close a prompt's hidden-state geometry is to the geometry of in-distribution inputs. A prompt that lands far from the training distribution centroid in residual stream space is likely to produce unreliable outputs. The model is being asked to reason in a region it did not see during training.

Aquin computes OOD similarity by taking cosine distance between the prompt's residual stream at a configurable layer and the centroid of a reference batch. The score is computed pre-decode, before any output token is sampled. Prompts flagged as high-OOD are tagged in the session and their attribution results are annotated with a confidence warning.

Attention routing and head entropy

Attention routing analysis surfaces which positions each head attends to across a prompt. Sink tokens, diagonal copy patterns, and semantic-retrieval heads appear as distinct structures in the attention map. Aquin identifies these patterns automatically and labels them in the per-head breakdown.

attention entropy adds a quantitative layer. Shannon entropy of the attention weight distribution tells you how spread or concentrated each head's attention is on a given input. A head with near-zero entropy on most inputs has collapsed: it attends to the same position regardless of input. These dead or degenerate heads are surfaced in a per-layer heatmap and can be examined individually.

Weight rank and SVD spectrum

weight rank is the numerical rank of a projection matrix, how many linearly independent directions it actually uses. A full-rank W_up uses all d_model dimensions independently. A low-rank W_up has collapsed into a subspace. This is common after LoRA fine-tuning where adapter rank is small, or after aggressive post-training where the gradient signal pushes most directions toward zero.

The SVD spectrum makes the rank collapse visible. Aquin plots the full singular value distribution for any selected weight matrix and marks the effective rank at 90%, 95%, and 99% energy thresholds. A model with a sharp drop after the first few singular values is using far less capacity than its parameter count suggests, a signal invisible in loss curves but apparent in the spectrum.

Activation geometry and intrinsic dimensionality

Activation geometry shows how the residual stream separates different concept groups across a prompt batch. Aquin runs PCA on a configurable batch of residual stream activations at each layer and projects the result to 2D. Clusters that are well-separated at early layers and merge at later ones are structurally meaningful. Centroid drift across layers shows how the model accumulates concept-specific information as depth increases.

intrinsic dimensionality quantifies how compressed the representation actually is. If 95% of the variance in a layer's activations is explained by 12 of 4096 PCA components, that layer is operating in a 12-dimensional subspace. Low intrinsic dimensionality is not inherently bad, many tasks are genuinely low-dimensional, but sudden drops after fine-tuning suggest the model is forgetting structure it had before.

Attribution across architectures

Attribution runs the same pipeline on dense and MoE models. The causal trace patches the residual stream at each layer to locate the fact retrieval depth, a residual-stream operation that works identically regardless of whether the surrounding sublayers are dense or sparse. The SAE is applied at the peak layer: for MoE models, it is trained on the pre-router hidden state, capturing the full joint representation before the routing decision splits it.

The circuit graph, logit lens, and feature steering all operate on the residual stream and are architecture-agnostic. The only difference in an MoE attribution run is that the SAE hooks at the pre-router position rather than at a standard MLP output.

Training monitor across architectures

Live metrics arrive through aquin watch ingest from your training script — not from aquin simulate. Full walkthrough: Training.

The training monitor streams step events and runs signal detection in real time. For dense models, five detectors cover loss divergence, gradient spikes, attention head death, dead MLP layers, and loss plateau. For MoE models, two signals are added: expert load balance (Gini coefficient of token-to-expert assignment per sparse layer, per step) and per-expert gradient norm. An expert whose gradient norm drops below threshold for five consecutive steps is flagged the same way a dead MLP layer is.

Post-training, the SAE feature diff and model diff both adapt to architecture automatically. The model diff runs on outputs, it is fully architecture-agnostic. The SAE feature diff runs on residual stream activations: for MoE models, it includes per-expert activation comparisons at layers where expert collapse was detected during training.

Evals

The eval system measures behavioral properties of a model from its outputs: how stable they are across paraphrase templates, whether it systematically shortens or hedges on specific topics, and how much its confidence degrades under prompt corruption. These are computed entirely from model outputs and are fully architecture-agnostic. Running the same eval suite on Llama 3.1 8B and Mixtral 8x7B produces directly comparable scores.

The eval system is also TransformerLens-compatible. Any checkpoint supported by TransformerLens loads without additional configuration. For MoE models, TransformerLens hooks attach to pre-router hidden states, the same position Aquin uses for SAE training and attribution.

Benchmarks

The benchmark system evaluates SAE features and model capabilities. For dense models, the SAE is trained on a selected layer's residual stream and the three feature scores evaluate interpretability, monosemanticity, and causal influence. For MoE models, the SAE hooks onto the pre-router hidden state, capturing the joint representation the router reads, not any individual expert's output. Feature scores are layer-level, not expert-level, making them directly comparable across architectures.

The Benchmark Builder, conversational in-session capability evaluation, is fully architecture-agnostic. Describe what to measure; the agent runs the prompts, scores the outputs, and appends a result card to the thread. Dense or MoE, the interface and output format are identical.

Security

The security system's behavioral layers, jailbreak taxonomy, red team probing, suppression bypass detection, operate on model outputs and are fully architecture-agnostic. The same six attack vectors are probed identically on dense and MoE models.

Weight trojan detection adapts for MoE. In dense models, the scan checks each layer's weight matrix for statistical anomalies. In MoE models, it scans per expert matrix. A backdoor implant targeting a specific expert, a rank-one update in one expert's feed-forward weights, is masked by aggregate-layer statistics but exposed at the per-expert level. Aquin runs the scan at the granularity the architecture requires.

The Attribution System

Seven tools that answer two questions: how did the model produce this output, and is the output actually correct?

Tracing facts through a language model

When a language model answers "What is the capital of France?" with "Paris", it is not looking anything up. Somewhere in 1.2 billion parameters, the association was encoded during training and is retrieved at inference time through a sequence of matrix multiplications. Two questions follow: where exactly does the retrieval happen, and once we know the mechanism, is the answer actually right?

The attribution system runs two pipelines in sequence on every output. The first traces the mechanism: which layers, features, and prompt tokens caused each response token. The second evaluates the result: whether claims are true, whether the framing leans in a direction, whether certain topics were quietly avoided. Neither is complete without the other.

The query

A single factual query run end-to-end through the full pipeline. The prompt is intentionally simple. Unambiguous causal structure makes each step's output easier to read.

ROME-style causal mediation analysis is the entry point: each prompt token's embedding is corrupted with scaled Gaussian noise, the forward pass is re-run, and the drop in the target token's probability is measured. Averaging over multiple noise runs produces a causal score for every (prompt token, response token) pair.

Attribution

Token attribution scores

Three prompt tokens dominate: "capital", "of", and "France". Together they carry nearly all the causal signal driving "Paris". "What" contributes almost nothing. The model identifies the semantically load-bearing tokens and routes most of the causal work through them, not through the full sentence structure.

16 layers, one peak

Causal patching localizes the retrieval to a specific layer. For each layer in turn, the clean residual stream is restored while all other layers remain corrupted, and the recovery in the target token's probability is measured. The result is a causal drop score per layer: which one, when restored alone, brings "Paris" back.

Layer 8 accounts for 87% of the causal signal. The France to capital to Paris association is stored in the MLP sublayers at the network's midpoint. This is the key-value store pattern: the subject representation ("France") functions as a lookup key, and the MLP writes the associated value ("Paris") into the residual stream at that layer.

The logit lens: watching confidence build

The causal trace locates the retrieval site. The logit lens shows what the model is predicting at each layer as it gets there. After every transformer block, the final layer norm is applied and the residual stream is projected directly into vocabulary space as if the model had stopped at that layer and been forced to output a token.

Early layers produce generic tokens like "the" and "city" with no factual commitment. Around layer 5, "France" surfaces briefly as the subject representation assembles. By layer 8, "Paris" dominates at 78% and holds flat through layer 15. The two-step structure of the retrieval is directly visible: subject formation first, then fact lookup at the MLP.

SAE Features

Top active features

The query is passed through an SAE at layer 8 to extract the top activating features at each token position. For each active feature, causal ablation zeroes out its contribution to the residual stream and re-runs the forward pass, comparing output distributions to define its functional role.

The circuit attribution graph

The circuit attribution graph makes the feature bridge structure explicit as a directed bipartite visualization: prompt tokens on the left, SAE features in the middle, response tokens on the right. Edge weight encodes activation strength on the left side and causal ablation score on the right.

Hub features are the diagnostic signal. f13910 (capital/seat-of-government) receives signal from both "capital" and "of" in the prompt and feeds both "capital" and "Paris" in the response, acting simultaneously as a relational and a geographic feature. A hub at this position is the first candidate for any intervention targeting "Paris".

What each feature does to the vocabulary

Each SAE feature is a direction in residual stream space. Its effect on the model's output is read by projecting that direction through the unembedding matrix, the logit projection. For f13933, the top boosted token is "Paris" at +4.21 and all suppressed tokens are non-French European capitals. The feature is not merely "France-related": it specifically routes the output toward French city names and away from other national capitals.

Feature neighborhoods in weight space

Features that are geometrically close in decoder weight space tend to fire in similar contexts and produce similar vocabulary effects. For f13933, the nearest neighbor at 91% similarity is f13007 (European nation names). The neighborhood also includes f7834 (country-capital associations) and f2901 (seat-of-power contexts). Any weight editing intervention should account for this neighborhood: editing one feature risks perturbing the others.

The feature space: a map of 16,384 directions

UMAP projects all SAE decoder directions into three-dimensional space, making the full geometric structure of the feature space navigable. Features that fire in similar contexts and produce similar vocabulary effects cluster together.

All five features active on this query fall inside or adjacent to the same cluster, a geopolitical reference region. The UMAP view is most useful as a pre-edit diagnostic: a tight cluster means an edit to one feature will likely affect the others, and the edit scope should be set accordingly.

Feature steering: intervening directly

Feature steering adds a scaled multiple of a feature's decoder direction to the residual stream at layer 8 on every forward pass, amplifying or suppressing the feature without touching model weights. It is the fastest way to validate a feature's causal role before committing to a permanent weight editing intervention. Steering is reversible, weight editing is not.

When steering confirms the feature's role and the logit projection confirms its vocabulary signature, a ROME-style weight editing operation to correct a factual association becomes a targeted, well-scoped intervention rather than a parameter search.

Checking

The attribution pipeline explains how "Paris" was produced: layer 8, five specific features, three prompt tokens, a geopolitical cluster with a clear logit signature. That tells us nothing about whether the output is accurate, whether its framing is neutral, or whether relevant information was left out. The checking system runs automatically after every generation and produces three analyses in parallel.

Fact check: is it true?

Every distinct verifiable claim is extracted from the response and classified as supported, refuted, or unverifiable, with a one-sentence explanation and up to three sources. Live web search rather than retrieval augmentation matters here: a model may assert something accurate at training time that has since changed.

Bias detection: which direction does it lean?

Rather than applying a fixed set of axes to every response, bias dimensions are derived from the content. Two to four axes genuinely relevant to the specific prompt are scored from -1.0 to +1.0. A response about climate policy yields axes like "alarmist vs dismissive." The axes shift with the content rather than being imposed on it.

Censor audit: what did it not say?

Fact check and bias detection work on what the model said. Censor audit works on what it did not. Given the prompt, 3 to 6 topic areas naturally relevant to the response are identified, then each is assessed: addressed directly (unfiltered), engaged with excessive caveats (softened), or avoided (suppressed).

The audit also attempts to classify the origin of suppression, weight-level (consistent avoidance across prompt framings) vs surface-level (instruction-following patch). This is a hypothesis to investigate, not a finding. Confirming it requires causal mediation analysis and feature steering on the specific deflection point.

Deception features

For honest vs deceptive probe sets, aquin feature locate ranks SAE features by differential activation and persists a canonical deception feature id for steering and checkpoint diff. Use after the main inspect pipeline surfaces suspicious behavior — pairs naturally with Security red-team findings.

Reading together

A model can pass every behavioral check and still encode a factual error that mechanistic analysis catches immediately. A clean causal trace does not guarantee a correct or unbiased output. The mechanism and the result are independent questions and both require an answer.

For teams deploying models in regulated or high-stakes contexts, this is the difference between knowing a model scored 90% on a benchmark and knowing why. Which answers it gets right for the right reasons, which it suppresses, where in the network to look when something is wrong, and how to correct it.

Training Sparse Autoencoders

How Aquin closes the SAE gap when inspect, steer, and find-feature need a dictionary on the model you actually have loaded: capture, train, load, diff, and align as one connected toolchain.

When feature tools need a dictionary you do not have

SAE-based tools assume a sparse dictionary over the residual stream at a specific layer. Catalog models can pull public weights. Family HuggingFace ids, embedding encoders, and fine-tuned checkpoints usually cannot. The session still loads the model; inspect still runs causal traces; but feature decomposition, steering, and deception ranking stall until a dictionary exists for those exact weights.

Aquin treats that as an operational pipeline, not a one-off script. activation capture exports labeled probe activations with manifest metadata. sae train collects corpus-scale activations or reuses saved shards, fits a dictionary, and saves locally. user SAE binding wires the result into the same inspect and steer path as a pulled public SAE. Each step syncs a card to the web orchestrator so the run is visible beside your session, not buried on disk.

The question this article answers is not how sparse autoencoders work in the abstract. It is what each tool in the chain surfaces, when to use probes vs corpus collection, and where the output feeds next in an investigation.

The tooling chain

Five verbs cover the full loop. Probes and corpus collection are inputs. Train and load are the dictionary lifecycle. Diff and align handle the checkpoint case where a public SAE no longer matches internal geometry after fine-tuning.

capture-activations

activation capture is the probe-scale export path. It runs a curated prompt set through the loaded model, writes per-layer activation shards, and attaches manifest metadata (probe labels, checkpoint id, hook name). The run syncs an activationCapture card to the web orchestrator so you can compare captures side by side without digging through ~/.aquin.

Use capture for checkpoint comparison, deception slices, and labeled exports. Do not use it as the substrate for dictionary training. Six probe vectors can finish a train card but produce a dictionary with 90%+ dead features. That is a pipeline smoke test, not a feature tool you load into inspect.

sae train

sae train is the dictionary lifecycle. Without flags it streams corpus text through the session model, materializes normalized activation chunks to disk, fits a sparse autoencoder, and saves under ~/.aquin/sae/user/. With --activations <dir> it skips forward passes and retrains from saved shards: same trainer, no model rerun.

Every run mirrors status to a saeTrain card (step, recon, dead-feature count). When the card completes, user SAE binding via load sae --user switches inspect, steer, sae-stats, and find-feature to your dictionary without restarting the session.

Run signals

Aquin logs reconstruction loss, mean L0 sparsity, and dead-feature count every 500 steps on the saeTrain card. Treat these as operator go/no-go signals, not ML lecture material. Falling recon on corpus data means load and benchmark. Flat recon with high dead count on probe-only input means the card finished but the dictionary is not usable.

The table below is from the same Llama 3.2 1B quick run at layer 8, contrasted against a six-vector probe rerun. The gap is the main operational lesson: scale of activations matters more than step count.

sae diff

After fine-tuning, a pulled public SAE may still reconstruct activations while assigning wrong feature indices. sae diff runs the same probe set through base and checkpoint weights, decodes both activation streams through the public dictionary, and reports per-feature delta. The result syncs a saeDiff card beside your training monitor.

A large diff is usually why you train a new dictionary on the checkpoint instead of steering with base weights. Diff tells you the public basis no longer matches internal geometry. It does not produce a replacement dictionary. That is sae train followed by load sae --user.

sae align

When you have two trained dictionaries (public base vs user checkpoint, or two training runs), feature indices are arbitrary. sae align Hungarian-matches decoder columns between two .pt files and reports mean cosine similarity plus the weakest pairs. The run syncs a saeAlign card.

decoder alignment is for index translation when you need correspondence across dictionaries, not as a quality gate. Low mean cosine after a large fine-tune means the feature basis moved. Run InterpScore and sae-stats on the checkpoint-trained dictionary before deciding whether to steer with it.

LLM vs embedding

The trainer and card schema are shared. The activation tensor is not. LLMs record every token position in the post-block residual stream. Embedding encoders record one mean pooling vector per text. Embedding dictionaries are narrower (4,096 features typical vs 32,768 on small LLMs) with a lower L1 coefficient because pooled vectors are already compressed.

Feature tools differ by mode after user SAE binding. LLMs get inspect, steer, and circuit graphs on token positions. Embeddings get check browser, check contrastive, and check faithfulness probes on sentence pairs. Train at the hook you plan to inspect, not where reconstruction is globally minimal.

Connected investigation

SAE training is rarely the end state. It is the bridge between a loaded model and feature-level tools. Typical loop: load a checkpoint, run sae diff if a public dictionary exists, train when diff is large, bind with load sae --user, then move into attribution and benchmarks before steering or circuit work.

The Training monitor article explains when checkpoint SAE diff fires during a fine-tune and why that motivates a new dictionary. Attribution covers inspect, steer, and circuit graphs once a dictionary is bound. Benchmarks covers InterpScore, purity, and MUI for deciding which features to trust before you build on them.

Simulating Training

No gradient-descent loop runs. Four analytical passes: SAE baseline, gradient landscape, LiSSA influence scoring, and NTK-linearised weight prediction, answering the question "what would happen if I trained?" before a single GPU cycle is committed.

The core idea

Here is the problem with fine-tuning: you find out what went wrong after you paid for it. Your dataset had three poisoned samples that the model happily learned from. Two layers were already dead before training started and accumulated zero gradient throughout. Your learning rate was slightly too aggressive for the landscape curvature at layer 12. You find all of this out when you inspect the checkpoint, which means you already spent the compute.

The simulation inverts the whole thing. It reads the actual curvature of the loss landscape before moving any weight. The Neural Tangent Kernel diagonal tells you how much each parameter would move under training. LiSSA approximates the inverse Hessian in linear time and scores every training sample by how much it helps or hurts test generalisation. SAE gradient decomposition projects the gradient onto interpretable feature directions and predicts, by name, which concepts would strengthen and which would be suppressed. All of this before a single optimiser step runs.

The output is not a report. It is a synthetic checkpoint: real PyTorch weights, shifted analytically by the NTK-linearised delta, saved to disk in the same format as a real fine-tune. The SAE diff pipeline, model diff pipeline, behavioral scoring, calibration and all of it runs on the synthetic checkpoint unchanged. The inspection system cannot tell the difference, because structurally there is none.

Why not just train?

Most fine-tuning iterations are exploratory. You are not training a production model, you are testing a hypothesis: will this dataset teach the model what I need it to learn? Committing GPU time to a hypothesis test is expensive and slow. The simulation answers the same hypothesis in a fraction of the time, with one critical advantage: it tells you things a real training run cannot.

Influence scoring is the clearest example. To know which of your training samples are actively hurting generalisation, you would normally need to retrain with each sample held out and compare the resulting checkpoints. LiSSA does this analytically in a single pass. You get a ranked list of harmful samples before training starts, and you can remove them, re-simulate, and confirm the prediction improved, all without touching a training loop.

The simulation does not replace training. It replaces the first three or four exploratory runs, the ones you were going to throw away once you understood what your data actually contained.

Not the training monitor

aquin simulate forecasts training analytically on the loaded model — no optimizer loop, no external metrics file. aquin watch is the opposite: it ingests JSONL from a trainer you already run and streams live charts plus signal detection. See Training for watch ingest and the five live detectors.

Pass 0: dataset quality

Pass 0 runs without loading the model at all. It analyses the raw dataset text in under a second and answers three questions that decide whether the rest of the simulation is worth running. First: does the dataset contain harmful content? Keyword detection scans every instruction and response field and flags anything that matches. Second: is the dataset contaminated with AI-generated assistant-speak? Phrases like "certainly!", "as an AI language model", and "I'd be happy to help" are fingerprints of synthetic data that trains the model to hedge and defer rather than to know things. Third: is the data structurally usable? Length distribution, sequence length violations, and instruction diversity all surface here.

The diversity check is worth lingering on. A dataset with 64 samples that are all slight paraphrases of the same instruction will produce a model that knows one thing very confidently and nothing else at all. The mean pairwise token overlap score quantifies this before you train. A score near 1.0 is a red flag that is invisible from sample count alone.

Pass 1: SAE baseline activations

Pass 1 is a forward-only pass with no gradients. It answers one question: which SAE features are already active for this kind of input? The model runs with a residual stream cache at the SAE layer, the SAE encoder maps each activation to a sparse coefficient vector, and the mean across all probe texts becomes base_acts_mean: a vector of shape [n_features] where each entry tells you how active that concept currently is for your dataset.

This baseline is not optional context for Pass 2. It is load-bearing. The SAE gradient decomposition weights every feature's predicted score by its baseline activation. A feature the model has never used for these inputs gets zero weight regardless of how strongly the gradient aligns with its decoder direction. Predicting that a dormant feature will strengthen because the gradient points at it is like predicting that a muscle will grow because you thought about it. The baseline catches that and zeroes it out.

Pass 2: gradient landscape

Pass 2 is where training would normally happen, and deliberately does not. A forward and backward pass runs on each batch. The gradients are read, per-layer norms are computed and streamed as live heatmap events, and the global max gradient norm is recorded. No optimiser step is applied. The gradients are pure measurement: they tell you the shape of the landscape your weights sit in.

Dead layers show up here in a way that real training hides. In a real training run you might not notice that L16 and L18 have near-zero gradient until you inspect the checkpoint and wonder why those layers look identical to the base model. The simulation surfaces it immediately, before training, as a property of how your dataset activates the network. Those layers have nothing to learn from your data. That is a signal about the data, not about the training procedure.

Gradient spike predictions work the same way. If the mean max gradient across your batches already exceeds 5× grad_clip without any momentum buildup, you will almost certainly see real spikes once Adam's moment estimates start amplifying. The simulation catches this before the first optimiser step ever runs.

SAE gradient decomposition

After all batches complete, the mean gradient across the residual stream at the SAE layer is computed. This is where the prediction becomes interpretable. Instead of reporting "the gradient norm at layer 8 is 0.71", the decomposition asks: which named concepts does that gradient point toward? Each SAE feature has a decoder direction in the residual stream space. The inner product of the loss gradient with that direction tells you how much training pressure is pushing the model toward or away from that concept. Weight the result by the feature's baseline activation from Pass 1 and you have a per-feature score: how much would this concept shift if training ran to completion.

Features with large positive scores are predicted to activate more strongly after training. Features with large negative scores are predicted to suppress. This is grounded in the actual gradient direction and the actual baseline activations. When F501 (refusal / safety language) shows up with a large negative score on a dataset that contains no refusal content, that is worth investigating before training because it means your data is pulling the model's safety behaviour in an unexpected direction.

Pass 2b: LiSSA influence scoring

This is the pass that has no equivalent in a standard training loop. Influence functions ask: if I removed this one training sample, how would the model's test-set loss change? A sample that reduces test loss is helpful, it is genuinely teaching the model something transferable. A sample that increases test loss is harmful, it is teaching the model something that actively hurts generalisation, whether that is an outlier, a mislabelled example, or a sample that reinforces a spurious correlation.

Computing influence exactly requires the inverse Hessian, H⁻¹. For a model with n parameters that is an n×n matrix, hundreds of billions of entries for even a small LLM. LiSSA sidesteps the materialisation entirely by approximating H⁻¹·v via a short iterative recurrence that only ever calls the Hessian-vector product primitive. One LiSSA run computes the inverse-HVP once, then the influence of every training sample is just a dot product against that shared result. The whole pass scales as O(n) in time and O(n) in memory.

How LiSSA works

LiSSA is a Neumann series approximation. The key recurrence seeds with the test gradient, then iteratively refines the estimate of H⁻¹·v by applying the Hessian-vector product at each step and folding the result back in with a damping and scale correction. After ten iterations it has converged to a close approximation of the true inverse-HVP without ever materialising the Hessian itself.

Each iteration is one forward pass and two backward passes. Ten iterations total, the only compute cost is twenty backward passes on a single sample. The convergence chart shows the residual dropping geometrically: by iteration 7 the approximation is within 5% of the true H⁻¹v₀. Damping λ=0.01 stabilises convergence when the landscape is flat. Scale s=25 prevents divergence in sharp regions. Both are conservative priors that hold across transformer architectures without tuning.

Pass 2c: RLHF simulation

Preference alignment adds a different objective on top of standard SFT: the model should assign higher probability to chosen responses than to rejected ones. Pass 2c simulates this without running RLHF training by computing the reward margin directly from the current checkpoint. For each preference pair (prompt, chosen, rejected), the model computes log P(chosen | prompt) and log P(rejected | prompt) and takes the difference.

The DPO loss is −log σ(β·margin). The gradient of that loss with respect to the residual stream at the SAE layer is decomposed using the same SAE projection as Pass 2, producing a feature-level view of what RLHF alignment would reinforce and suppress relative to standard SFT. If F501 (refusal / safety) appears in the top reinforced features on a dataset where you did not intend to tune refusal behaviour, the preference labels are driving an unintended alignment.

The mean reward margin at the start of training tells you how hard the alignment phase will have to work. A negative mean margin means the base model actively prefers the rejected responses, the model is aligned backwards and RLHF training will fight the base distribution for every preference pair. That is expensive and often unstable. Knowing this before training starts means you can fix the preference labels or adjust β rather than discovering the instability mid-run.

Pass 3: NTK-linearised weight delta

This is the pass that produces the synthetic checkpoint. It applies the weight changes that T virtual training steps would have produced, analytically, without running any of them. The key insight from the Neural Tangent Kernel theory is that in the linearised regime, every parameter moves in proportion to its gradient and inversely in proportion to the curvature it faces. Parameters in flat regions move far. Parameters in sharp regions move cautiously. The exact formula is:

The weight delta for each parameter is: learning rate times the number of virtual steps, divided by the NTK diagonal entry plus a small damping constant, times the mean gradient for that parameter. Parameters facing high curvature get a smaller effective learning rate. Parameters in flat regions get a larger one. No optimiser loop runs and the delta is applied analytically in a single no-gradient pass and gradient clipping guards against runaway values on dead layers before anything is written to the weights.

NTK diagonal via Rayleigh quotient

Computing the full NTK diagonal exactly would require one Hessian column per parameter, storing it would cost as much as the model itself. The Rayleigh quotient approximation gets the same information from exactly one HVP call per parameter, in the gradient direction:

The Rayleigh quotient measures how much the Hessian amplifies the gradient: how sharply the loss curves in exactly the direction the parameter wants to move. A large K_ii means the loss is steep in that direction; the effective learning rate shrinks and the parameter moves cautiously. A small K_ii means the landscape is flat in the gradient direction; the parameter gets a large effective learning rate and moves aggressively. Dead layers have K_ii near zero, they would receive a runaway effective learning rate but the grad_clip guard catches and rescales the delta before application.

Loss sharpness via power iteration

Sharpness is the single number that tells you how reliable the NTK prediction is. The loss landscape is only well-approximated by its linearisation in flat regions. Sharp regions, where the Hessian's largest eigenvalue is large, are where the linear approximation breaks down fastest as the weights move. power iteration estimates λ_max in three HVP calls:

A sharp landscape (λ_max > 10) means the NTK-linearised predictions are least reliable: the linearisation approximation breaks down where curvature is high. Aquin surfaces this as a sharpness label alongside the effective LR map, so you can judge how much to trust the delta prediction before committing to a real run.

Synthetic checkpoint

Pass 3 ends with a merge. The LoRA adapter matrices are merged into the base weights and the result is saved to disk. The file is a standard model checkpoint: same keys, same dtypes, same tensor shapes as any real fine-tune you would produce from an actual training run.

This design choice is the entire point. By producing a real checkpoint instead of a structured report, the simulation gains access to every downstream tool in the inspection system without modification. The SAE diff loads base and synthetic checkpoint, runs both through the SAE encoder, and computes the feature-level diff. The model diff runs behavioral scoring on the synthetic checkpoint exactly as it does for real fine-tunes. Circuit tracing, the feature browser, calibration, regression tracking, all of it runs unchanged, because the checkpoint is indistinguishable from the real thing.

The alternative, a structured report that downstream tools would need to parse and adapt to, would have required every inspection tool to add a "simulation mode". The synthetic checkpoint means zero adaptation. The full power of Aquin's inspection pipeline is available on a run that committed zero GPU training cycles.

Signal detection during simulation

The simulation emits the exact same signal event types as a live training run. Dead layer signals fire when Pass 2 finds layers with mean gradient norm below 1e-6. Gradient spike signals fire when the mean max gradient exceeds 5× grad_clip. The dashboard renders them identically to real training signals, in the same positions in the event stream, with the same severity levels.

The difference is in how you act on them. A dead layer signal from a real training run means stop and investigate the current checkpoint. The same signal from the simulation means this configuration would have produced a dead layer, fix it before you train. Simulation signals are forward-looking warnings, not reports of something that already happened.

The full pipeline

The four passes are a strict dependency chain. Pass 1 needs the model loaded. Pass 2 needs the baseline SAE activations from Pass 1 to weight the gradient decomposition. Pass 2b needs the gradient computations from Pass 2. Pass 3 needs the mean gradients accumulated across all of Pass 2. After Pass 3 writes the checkpoint, SAE diff and model diff run in parallel, the only genuinely concurrent step in the whole pipeline.

The simulation also supports a comparison mode that diffs two saved results side by side: influence score changes between dataset versions, feature score shifts when the training objective changes, effective LR map deltas when learning rate or epoch count is adjusted. The pattern is: simulate, read the harmful samples, remove them, simulate again, compare. By the time you commit to real training you have already seen two or three predicted checkpoints and know which one is worth running.

The git graph below shows exactly this workflow. Simulation v1 flags L16 and L18 as dead and identifies three harmful samples. The dataset is cleaned. Simulation v2 runs clean, surfaces a sharpness warning that prompts a learning rate adjustment. Only then does the real training run commit. The GPU time runs on a configuration that has already been vetted analytically.

Every simulation produces four first-class artifacts: a quality report, an SAE prediction with per-feature scores, a ranked influence score list, and the synthetic checkpoint itself. They are independent once produced. You can re-run the influence analysis against a cleaned dataset without re-running the gradient passes, or load the synthetic checkpoint in the model inspector without ever touching the original simulation again.

The Hessian-vector product is the primitive everything else is built on. LiSSA calls it ten times to approximate the inverse Hessian. The NTK diagonal calls it once per parameter in the gradient direction. power iteration calls it three times to estimate sharpness. Each call costs one extra backward pass. No Hessian matrix is ever stored.

Pass 2 dominates wall-clock time because it runs a full forward and backward pass on every batch. LiSSA adds twenty more backward passes on top of that. Pass 3 is cheap, a handful of HVP calls and a single no-gradient write. The post-sim phase is where the real inspection value unlocks: SAE diff and model diff run in parallel the moment the checkpoint is saved, and everything downstream is available within minutes of the simulation completing.

The compare endpoint makes this iterative loop explicit. Diff two simulation results and you see exactly what changed: which samples flipped from harmful to helpful after cleaning, which features shifted when the training objective changed, how the effective LR map moved when the learning rate was adjusted. The pattern is simulate, read, clean, simulate again, compare until the predicted checkpoint is one you would be comfortable shipping.

Four invariants hold across every simulation run regardless of model, dataset, or configuration. No gradient-descent loop runs. LiSSA completes within its iteration budget. The NTK-linearised delta is clipped before application. The synthetic checkpoint is a valid state dict loadable by the inspection pipeline. These are not soft guidelines, they are enforced in the implementation and the simulation fails loudly if any of them break.

The simulation answers one question: should I train? By the time it completes you know whether your dataset is clean, which samples are pulling against generalisation, which named concepts the training would strengthen or suppress, whether any layers are dead for your specific data, and what the resulting checkpoint would look like to every downstream inspection tool. That is more than you would know after a real training run, because real training does not score your samples by influence.

If the prediction looks good, commit the compute. If it surfaces harmful samples, remove them and re-simulate. If dead layers appear, investigate whether the data's coverage of those layers is sufficient. If the sharpness is high, pull the learning rate down and check the effective LR map before the delta blows up on a specific layer. The GPU budget runs on a configuration you have already stress-tested analytically, not a hope.

Embedding simulation

The same simulate machinery runs in embedding mode after aquin load --model gte-small (or any supported encoder). aquin check dataset scores pair quality; aquin simulate forecasts how the embedding geometry would shift under contrastive fine-tuning. Runs live in ~/.aquin/runs/ alongside LLM simulations — not under watch.

Training

Real-time signal detection, behavioral before/after comparison, and per-layer feature diffs: everything the loss curve does not show you.

What the loss curve does not show

A fine-tuning run has structure beneath the loss curve: gradient dynamics that reveal how information flows through the network, attention heads that can collapse silently, feature activations that shift as the model rewrites internal representations to accommodate the training objective. Almost none of that structure is visible from loss alone.

The training inspect system surfaces it in real time. A step event stream feeding loss, learning rate, gradient norms, per-layer breakdown, dead layer list, and epoch index into a signal engine that runs on each step as it arrives. When the engine detects a gradient spike, a loss plateau, a collapsed attention head, or the onset of loss divergence, a signal fires immediately with the specific metric and the exact step.

When training completes, the dashboard computes a model diff and a per-layer SAE feature diff, showing not just how loss moved, but which behaviors changed and which internal representations were rewritten.

What gets streamed

The system is agnostic to training framework. It consumes a flat step event schema: step index, loss, learning rate, max gradient norm, per-layer grad norms as a record, dead layer list, epoch index. No nested objects, no optional deep structures.

The per-layer grad norm breakdown is what enables the dead layer and attention collapse detectors. Without it, the engine observes only aggregate gradient behavior. With it, it names the specific layer that has collapsed and tracks how long it has been dead.

Watch vs simulate

This article is about the training monitor: live loss, learning rate, gradient norms, and signal cards in your session tab. That data arrives through aquin watch — a passive observer for your PyTorch loop, HuggingFace Trainer, or any script that appends metrics JSONL. Run aquin watch init, then aquin watch ingest --file metrics.jsonl --follow. Charts mirror to the web session automatically.

aquin simulate is different: an analytical forecast before you train (LiSSA influence, NTK weight delta, SAE gradient decomposition). It does not ingest external metrics. See Simulating Training for that path.

The signal engine

Five detectors

The signal engine is a pure function that runs on each new step snapshot. It takes the full step history plus two persistent streak maps, one for non-attention layers and one for attention layers, and returns a signal if one fired, or null. The streak maps are the only stateful part: they persist across steps so that dead layer detection can track how many consecutive steps a layer has had near-zero gradient.

Priority and cooldown

loss divergence and gradient spikes are checked first because they indicate active instability that may warrant stopping the run. dead layer and attention collapse are checked next, naming the specific failed components. loss plateau is last. It frequently describes healthy convergence rather than a problem, and its priority reflects that.

A 30-step cooldown prevents the same signal type from re-emitting continuously. A gradient spike that resolves and re-occurs fires again after 30 steps, the second occurrence is a distinct event with its own context.

The model diff

Behavioral delta

Three behavioral scores describe how the fine-tune changed the model from the outside: consistency score, suppression score, and robustness score. These are the same metrics from the eval system, applied to the base-vs-fine-tuned comparison. The base model is the reference; the fine-tuned checkpoint is the subject. The difference is the behavioral delta the training objective produced.

The robustness score is the most informative signal for factual fine-tuning. A fine-tune intended to reinforce factual knowledge should produce higher robustness on those facts. A robustness drop on target facts after factual fine-tuning means the model learned a surface pattern rather than a grounded representation.

The SAE feature diff

Layer change density

Behavioral scores describe the model from the outside. The SAE feature diff describes what changed internally. For each layer, the diff reports how many features shifted activation between base and fine-tuned, the mean absolute activation delta, and the single feature with the highest delta.

Layer-level change density is the most informative aggregate. A fine-tune that changes 14 of 512 features at L8 and 2 of 512 at L4 is making a focused, deep rewrite. The top feature per layer is where mechanistic investigation should start. If L10's top shifted feature is F501 (refusal / safety language) and the training data had no refusal content, that warrants investigation in the model inspector.

Checkpoint analysis

After a real fine-tune (not a simulation), checkpoint tools compare base vs on-disk weights and activations. aquin weight-diff reports parameter-level deltas and norm shifts. aquin residual-drift measures how residual-stream geometry moved on probe prompts. aquin sae diff is the per-feature activation diff described above — the mechanistic counterpart to the behavioral model diff. Before merging a LoRA adapter, aquin merge-analysis composes weight-diff with rank/collapse signals and optional behavioral scores into a pass/warn/fail gate. When a run logs multiple checkpoints, aquin trajectory-analysis plots how ‖ΔW‖ evolves over training steps.

The regression tracker

A single model diff shows how one fine-tune changed behavior relative to base. The regression tracker extends this across runs: every time a model diff arrives, category scores are appended to a per-category history so behavior can be tracked across all completed runs in the session.

A category that regresses more than five percentage points on the latest run is flagged. Detection is relative to the immediately prior run, not to the base. A score can look healthy against the base model while trending negatively across iterations. The tracker catches that drift where the raw diff cannot.

Confidence calibration

A model's stated confidence and its actual accuracy can diverge in ways invisible from loss alone. A fine-tune can lower loss while making the model systematically overconfident. ECE measures that gap directly: it bins outputs by stated confidence, computes accuracy within each bin, and reports the mean gap between the two.

The calibration panel runs this comparison between base and fine-tuned using the training dataset as the evaluation set. The reliability diagram shows both models' accuracy-per-confidence-bin as bar pairs against a perfect-calibration diagonal. The per-topic ECE table breaks the aggregate down by category. Models trained on domain-specific data frequently improve ECE on the target domain while degrading it on adjacent topics that share surface patterns with the training examples.

The low-confidence row list surfaces inputs where the fine-tuned model assigns probability below a configured threshold. These rows are exportable directly as a labeled dataset for the next training iteration. The model's own uncertainty becomes the selection criterion for the data that trains the next version.

Training as the start of the investigation

Each finding from the training run is an entry point into a deeper investigation, not a terminal result. A dead layer signal at L6 step 61 is most usefully followed up by opening the fine-tuned checkpoint in the Model Inspector, going directly to L6, and running the causal trace to confirm whether that layer still contributes to outputs.

A suppression score that rises from base to fine-tuned opens a data investigation: the training dataset can be opened in the Data Inspector and the toxicity and PII modules run against the columns most likely to produce hedging signal.

The SAE feature diff provides the entry point for mechanistic investigation. Once the features that shifted most are identified and at which layers, open Attribution on those features — benchmark with aquin sae-stats, run the logit lens, steer to confirm causal role. A harmful-sample flag from watch ingest points to simulate for influence scoring before the next run.

The calibration panel adds a third path out of the training run. Low-confidence rows exported as a labeled dataset for the next iteration. The regression tracker closes the loop in the other direction, confirming the next iteration did not trade one weakness for another. Together they make the training session the input to the next investigation rather than the end of one.

Embedding Models

Geometry inspection, retrieval evaluation, fine-tuning monitoring, embedding diff across checkpoints, and sparse autoencoder feature analysis. Load any sentence-transformers compatible encoder and get the full picture of your embedding space.

Embedding models in Aquin

An embedding model is an encoder that collapses a variable-length input into a single dense vector. That vector is the whole output. There is no next-token distribution, no chain of reasoning, no generation. Everything the model knows about the input is compressed into a fixed-size point in a high-dimensional space, and the quality of that compression determines whether downstream retrieval, clustering, or classification works.

Most embedding tooling stops at benchmark numbers. Aquin goes into the space itself. You can load any sentence transformer checkpoint, visualize the geometry of your dataset, measure whether the space is healthy or anisotropic, trace similarity through the encoder layer by layer, evaluate retrieval quality on your own query-document pairs, compare two checkpoints to see exactly what a fine-tune changed, and decompose individual embeddings into interpretable sparse features using a trained sparse autoencoder.

Supported models

Aquin supports any HuggingFace checkpoint that follows the sentence-transformers interface, a transformer encoder with a pooling layer on top. Pooling strategy is detected automatically from model config: CLS token pooling, mean pooling, or weighted mean pooling. For Instructor-style models with instruction prefixes, the prefix is applied transparently at inference time.

Bi-encoders

Bi-encoders embed query and document independently and compare them with cosine similarity. This makes them fast for large-scale retrieval: you embed the corpus once, index it, and query at inference time. The tradeoff is that the encoder cannot model interactions between query and document. BGE, E5, GTE, Nomic, Jina, Instructor, MiniLM, and SBERT are all bi-encoders. Every tool in Aquin's embedding system runs on bi-encoders.

Cross-encoders

Cross-encoders take a query-document pair as a single concatenated input and output a relevance score. They do not produce an embedding vector. Because they model query-document interactions directly, they are significantly more accurate than bi-encoders on reranking tasks, but cannot be used for large-scale retrieval directly. Aquin supports cross-encoders for reranking evaluation: load a cross-encoder alongside a bi-encoder retriever and compare the rank distributions before and after reranking.

Inspection signals

Retrieval benchmarks tell you a number. Inspection signals tell you why that number is what it is. Each row maps to a CLI verb on the loaded embedding model (no embed- prefix — mode follows aquin load).

Embedding geometry

The embedding explorer projects your dataset into 2D using UMAP and plots every point. Color by label, by cluster assignment, or by OOD score. Points that sit far from any cluster, inputs the model has not learned to place reliably, are flagged automatically. The explorer is the starting point for understanding whether your embedding space is doing what you need it to do before you run any retrieval or classification on top of it.

intrinsic dimensionality adds a quantitative view. If a 768-dimensional embedding space only needs 40 dimensions to explain 95% of the variance in your dataset, the model is compressing your data heavily. Whether that is good or bad depends on the task, but knowing it is essential context for choosing embedding dimension, comparing models, and diagnosing retrieval failures.

Anisotropy

anisotropy is a geometric degeneration where all embeddings cluster in a narrow cone rather than distributing across the full sphere. In an anisotropic space, random pairs of inputs have high cosine similarity not because they are semantically similar, but because every vector points in roughly the same direction. This inflates similarity scores across the board and makes retrieval unreliable.

Aquin measures anisotropy as the mean pairwise cosine similarity across a random sample of embeddings. A well-distributed space has mean similarity near 0. A collapsed space has mean similarity approaching 1. The distribution is plotted as a histogram so you can see whether the problem is severe across the board or concentrated in a subset of the data.

Layer-by-layer analysis

An embedding model's final vector is not built in one step. It emerges across the encoder's layers as attention heads route information and the feed-forward sublayers transform representations. Aquin plots mean pairwise cosine similarity of hidden states at each encoder layer. This shows at which layer the model's representation stabilizes, where collapse begins if it does, and whether the final pooled output reflects the geometry of earlier layers or diverges from it.

OOD detection

An input that embeds far from the centroid of your dataset's embedding distribution is out-of-distribution for your corpus. Including OOD inputs in a retrieval index degrades retrieval quality, they pull nearest-neighbor scores away from genuinely relevant results. Aquin computes an OOD proximity score for each input by measuring cosine distance from the corpus centroid. Inputs above a configurable threshold are flagged and listed for review before indexing.

Retrieval evaluation

Aquin evaluates retrieval quality on your own query-document pairs. Upload a JSONL file with query and document fields, optionally with relevance labels, and Aquin computes the full retrieval metric suite: Recall@1, Recall@5, Recall@10, MRR, and NDCG@10. Results are broken down by topic category when labels are available.

The hard negatives gap is the most actionable metric. It measures how much cosine similarity separates the closest true positive from the closest hard negative for each query. A small gap means the model is barely distinguishing relevant from near-relevant documents at the decision boundary, the failure mode that standard Recall@k scores miss entirely.

Fine-tuning support

Live fine-tune metrics stream through aquin watch ingest (see Training). For contrastive runs, aquin simulate forecasts geometry shifts before you commit GPU — see Simulating Training.

For contrastive loss objectives, InfoNCE, NT-Xent, triplet, the loss is decomposed into positive pair similarity and negative pair similarity tracked separately. A widening gap between the two is healthy. A narrowing gap means the model is pulling negatives in, not just pushing positives together.

LoRA fine-tuning on embedding models is supported natively. Adapter matrices are merged at load time for inspection. The training monitor tracks per-layer gradient norms across the encoder layers, flagging layers where gradients have died or spiked. The same dead-layer detector used for LLMs, applied to the encoder stack.

Embedding diff

When you fine-tune an embedding model, the geometry of the space changes. Aquin's embedding diff runs both checkpoints on the same probe dataset and compares: centroid positions per topic cluster, cosine similarity distribution shift, anisotropy delta, and nearest-neighbor rank changes across the query set. This tells you what the fine-tune changed in the space, not just whether task metrics went up.

embedding drift is reported as a composite score, a weighted average of centroid shift magnitude, rank change count, and anisotropy delta. A fine-tune that improves retrieval by pulling topic clusters apart without inflating anisotropy scores well. A fine-tune that improved one cluster's retrieval by collapsing another's geometry scores poorly even if headline Recall@1 went up.

Sparse autoencoders

Geometry tells you the shape of the space. A sparse autoencoder tells you what is in it. Standard embedding analysis shows that two sentences are close together, but not why. SAE feature analysis opens the vector and reads out the specific concepts it contains.

A sparse autoencoder is a dictionary learning model trained on the final-layer activations of an embedding model. It learns a set of unit-norm decoder vectors called features, one per dictionary entry, such that any activation can be approximately reconstructed as a sparse linear combination of them. The coefficients in that combination are the feature activations: a large coefficient on a feature means the input strongly expresses the concept that feature has learned to represent. In practice, features tend to correspond to interpretable concepts: domains (medical, legal, financial), linguistic patterns (negation, formal register), and topic clusters.

Embedding SAE tools are available through the programming agent. Load an embedding model and ask the agent to decompose a sentence, browse the feature space, or trace how a concept builds across encoder layers.

Feature decomposition

The entry point to SAE analysis is feature decomposition: run a text through the embedding model and SAE encoder, and read out which features activate and how strongly. A typical sentence activates 50 to 200 features out of a 16,384-feature dictionary. The top 10 to 20 are usually interpretable, and looking at them reveals the model's understanding of the input at a granularity that neither the raw embedding vector nor the distance to other sentences can show.

Contrastive decomposition runs two texts side by side and returns only the features that differ between them. This is useful when two semantically similar sentences should map to the same retrieval result but do not. The diverging features show where the model is drawing a distinction that might not be meaningful for your task.

Feature browser

The feature browser runs your corpus through the SAE and ranks the features by total activation. For each feature it shows an auto-generated label (derived from the top-activating examples via an LLM), activation frequency across the corpus, the maximum activation value observed, and the three to five sentences that activated it most strongly. Clicking a feature expands the example list.

The browser is the fastest way to understand what your data looks like from the model's perspective. Run it on a corpus and you will see which concepts the model has learned to distinguish and which it has compressed together. Corpora with many domain-specific terms tend to produce a small number of high-frequency domain features. Corpora that mix registers produce more general linguistic features at the top of the ranking.

Network graph

Features do not activate in isolation. Sentences that activate a feature for medical diagnosis also tend to activate features for clinical symptoms and drug dosage. The co-activation network makes these relationships visible. Each node is a feature, sized by mean activation and colored by activation frequency. Edges connect features that co-activate on the same sentences, with thickness proportional to co-activation frequency.

The network reveals the latent cluster structure of your corpus at the feature level. Tightly connected subgraphs correspond to semantic domains where the model has learned to group related concepts. Loosely connected features are general-purpose, they fire across domains. Click any node to see its neighbors and their co-activation frequencies.

Circuit tracing

Feature decomposition tells you which features activate at the final layer. Circuit tracing tells you at which encoder layer each feature appears and how its activation builds across the stack. For a given text and target feature, Aquin runs the SAE independently on the hidden state at each encoder layer and plots the target feature's activation across layers.

The resulting circuit graph is a horizontal DAG, one column per layer, with bezier arcs showing activation growth between layers. Features that appear early and grow steadily are structural, the model is building the concept progressively. Features that appear suddenly in the final two or three layers are late-binding, the model is making a classification-like decision rather than building up the representation. The co-active features in each column show what the model was also representing at that layer.

Steering

SAE steering adds a scaled version of a feature's decoder direction to the final-layer activation before computing the embedding. Boosting a feature pushes the resulting embedding toward inputs that activate it strongly. Suppressing a feature pulls it away. The result is a modified embedding that you can use to measure how much that feature influences the model's output.

The steering tool reports cosine shift, how far the steered embedding moved from the original, and optionally re-ranks a retrieval corpus to show how the results change. Boosting a domain feature on a borderline query will typically pull in more domain-specific results. This is a way to verify that a feature actually encodes the concept its label suggests, and to understand how robustly that concept influences retrieval.

Absorption and polysemy diagnostics

A well-trained SAE has features that fire independently and correspond to distinct concepts. Two failure modes undermine this: feature absorption and polysemy. Absorption is when feature A always fires when feature B fires, meaning one concept has been absorbed into another and the absorbed feature contributes no independent signal. Polysemy is when a single feature fires on semantically unrelated inputs, meaning it has been overloaded to represent several distinct concepts.

Aquin's diagnostics scanner finds both. Absorption is detected by computing conditional activation probability across a corpus: if P(B activates | A activates) exceeds a threshold, the pair is flagged. Polysemy is detected by measuring the semantic variance of a feature's top-activating examples. High variance means the activating texts are semantically distant from each other. Both reports come with the specific pairs or features identified, so you can judge whether the overlap is an artifact or reflects a genuine latent structure in your data.

Retrieval faithfulness

Retrieval faithfulness measures which SAE features are load-bearing for retrieval quality. For each feature, Aquin zeros it out in all query embeddings and recomputes NDCG@k on your query-document pairs. The drop in NDCG tells you how much that feature was contributing to retrieval. A large drop means the feature is essential. A negligible drop means the feature, despite high activation, is redundant with other features in the embedding.

This analysis surfaces a question that benchmark scores cannot answer: which parts of the embedding actually drive retrieval performance? Run aquin check faithfulness on your pairs JSONL. aquin decomp decomposes a single embedding into sparse feature contributions for debugging borderline queries.

Cross-model feature matching

Two embedding models trained on similar data may learn similar concepts, but their SAE feature dictionaries are completely independent. Cross-model matching identifies which features correspond across models by computing cosine similarity between decoder vectors. Features with high decoder similarity represent the same concept in both models. Features with no match are model-specific, concepts the model has learned that the other has not.

This is useful when choosing between models for a task. If both models have learned the concepts relevant to your domain and the features match closely, the models are interchangeable for that domain and you can pick on latency and size. If one model has domain-specific features that the other lacks, that is a meaningful capability difference. Cross-model matching makes the comparison specific rather than abstract.

The Security System

Adversarial risk detection across the model checkpoint and the boundary between model versions.

Adversarial risk across the pipeline

ML security does not live in one place. At the model layer, a trained checkpoint can be probed for jailbreak susceptibility, robustness under prompt corruption, and suppression bypass, and weight tensors can be scanned for weight trojan signatures independent of any prompt-response behavior. At the training layer, behavioral scores are compared between base and fine-tuned checkpoints to reveal what the fine-tuning objective changed about the model's defenses.

Both layers are surfaced in a single continuous session. The model inspector's security panel runs red teaming probes across six attack vectors and scans weight tensors for trojan signatures. Live robustness drift across fine-tune versions is visible in the same session tab where aquin watch streams training metrics — see Training.

Where security checks live

Seven distinct security checks across two pipeline layers. Model-layer checks run against a trained checkpoint; training-layer checks run at the boundary between base and fine-tuned model.

Model inspection layer

The fine-tuning objective, template design, prompt distribution, and RLHF reward signal can each shift the model's behavior in ways that increase susceptibility to adversarial prompts. Behavioral security requires probing the model directly after training.

The model inspector's security panel contains two tabs. Red Team runs adversarial probes across six attack vectors and produces a composite robustness score with per-vector breakdown. Weight Trojans analyzes weight tensors directly, independent of any prompt-response behavior, for statistical signatures associated with backdoor implants.

Jailbreak taxonomy

The six attack vectors map onto a taxonomy of known jailbreak families. Different attack families have different mitigations and mechanistic signatures. A model robust to prompt injection but brittle to role confusion has a different training problem than one that fails on multi-turn extraction.

Red team probing

The red teaming panel runs automated adversarial probes across all six attack vectors and produces a structured report. Each vector is scored 0 to 1 by robustness, classified as pass (65% or above), warn (35 to 65%), or fail (below 35%), and annotated with a finding that identifies the specific failure mode.

Weight trojan detection

Behavioral red teaming only catches backdoors reliably triggered by adversarial prompts. weight trojan detection takes a different approach: weight matrices are analyzed directly for statistical signatures characteristic of implanted backdoor patterns.

Three signals: kurtosis measures whether the weight distribution has heavier tails than expected. Outlier density measures the fraction of weights more than four standard deviations from the layer mean. singular value ratio measures whether the weight matrix has a dominant low-rank component.

Training monitor layer

Fine-tuning changes more than what a model knows; it changes how it behaves under adversarial pressure. A fine-tune intended to add factual knowledge can decrease robustness to role confusion attacks if the training data contained examples that rewarded persona compliance.

Security as a connected investigation

The value of a layered security system is the chain of inference it enables. A weight trojan flagged at a specific layer becomes an actionable mechanistic question: did any SAE features at that layer activate anomalously on the adversarial prompt families that scored lowest in red teaming? Keeping that investigation in a single continuous session means the chain from model to training dynamics stays intact.

Benchmarks

How Aquin decides which SAE features are trustworthy, and how you can build and run custom benchmarks against any loaded model without leaving your inspection context.

Part 1: Feature Benchmarks

Most SAE features get used because they have a plausible-sounding label and a clean activation plot. That is not enough. A label can be wrong. A feature can be coherent but causally irrelevant. Before a feature earns a place in a circuit graph or a steering experiment, three properties need to hold independently: the label predicts where it fires, it is monosemantic, and it actually does work in the forward pass.

These conditions are orthogonal. A feature can pass two and fail the third in any combination. Aquin scores all three separately and surfaces them as a diagnostic triple. The combination tells you what to do next: relabel, filter, or trust.

InterpScore

The question InterpScore answers: does this feature's label predict when it fires? Two sentence sets are built per feature, one where the label implies the feature should activate and one where it should not. Both pass through the model, maximum activation at layer 8 is extracted per sentence, and Cohen's d is computed between the two distributions. The result clips to [0, 1].

A score near 1 means the label and the feature agree. A score near 0 means they have drifted, so treat the auto-generated label as a guess. Each feature uses 10 positive and 10 negative sentences, 20 separate forward passes through the full model and SAE.

FeaturePurityScore

InterpScore evaluates the label. FeaturePurityScore evaluates the feature itself, with no label involved. The sentences where the feature fired above threshold are embedded, and mean pairwise cosine similarity of the upper triangle is computed, excluding self-similarity.

High purity means activating contexts cluster tightly in embedding space, so the feature is monosemantic. Low purity means it is firing on surface-level co-occurrence rather than a coherent concept. Polysemantic features concentrate near the sparsity penalty boundary, which is consistent with what the superposition hypothesis predicts.

Model Utilization Index

A feature can look perfect on InterpScore and FeaturePurityScore and still be inert. The model computes it but does not route through it. This is the gap MUI is designed to close. Some cleanly labeled, monosemantic features produce near-zero KL divergence under ablation. They are decorative.

MUI measures causal load directly. At each token position where the feature fires above threshold, its projection onto the residual stream is zeroed and the forward pass re-runs. KL divergence between baseline and ablated output distributions is computed at that position, averaged across all firing positions, and normalized by baseline Shannon entropy. The result is a [0, 1] score of how much the model's output depends on this feature when it is active.

SAE dictionary health

Per-feature InterpScore, purity, and MUI answer "is this feature good?" aquin sae-stats answers "is the whole dictionary healthy?" — dead features, never-fired units, mean firing rate, and sparsity distribution across the layer. Run it before benchmarking individual features or after aquin sae train on captured activations.

Reading the scores together

The three scores are a diagnostic triple, not a leaderboard. The most actionable pattern is high purity and high MUI with low InterpScore. The feature is coherent and causally load-bearing, but its label is wrong. A relabeling pass using the actual activating examples usually resolves it in minutes. The all-low pattern is a dead feature, appearing disproportionately near the sparsity penalty boundary, and it should be filtered before any downstream analysis.

Part 2: The Benchmark Builder

Standard benchmark workflows require selecting a suite, configuring a harness, running the eval, and parsing results out of band. For scheduled evaluations that pipeline is fine. For a question that surfaces mid-inspection, say a suspicious feature or an unexpected output, it is a full context switch that almost never happens. The question gets dropped.

The Benchmark Builder removes the context switch. You describe what you want to measure in natural language, the agent writes the prompt suite, runs it against whatever is currently loaded, and returns a scored card in the thread, grounded in the same session that surfaced the question. The card supports four chart types and exports to CSV, JSON, PNG, and PDF.

Building a benchmark

The simplest path is build_benchmark. You supply a title, a list of capability dimensions with scores the agent has measured, and an optional summary. The agent selects the scoring method based on task class and records it in card metadata. A 67% on CoT math with partial-credit scoring is not the same as a 67% on factual recall with next-token probability. The method travels with the result.

Custom evals

The built-in consistency, suppression, and boundary evals cover known failure modes. Custom evals cover yours. run_custom_eval is a spawn-only tool that runs inside a sub-agent, so it does not block the main conversation thread. You pass a name, a description, a prompt set of 3 to 50 items, and a scorer_type.

Custom evals work on both LLMs and embedding models. For LLMs the model generates a response per prompt, then the scorer evaluates it. For embedding models no generation happens — the model encodes each prompt and the scorer receives the embedding vector directly. The same scorer interface handles both paths, so a custom eval written for an LLM can be adapted to an embedding model with a one-line change.

Scorer types

The two scorer types differ in how much of the evaluation logic you own. Semantic similarity handles the comparison automatically. Code hands you the raw variables and expects a number back.

For the code scorer, LLM variables are prompt, response, activations, and features. Embedding model variables are prompt, embedding (np.ndarray), and reference_embedding. The script must print a float 0-1 or a JSON object with score and note as its last output line. Anything before that is treated as logging.

The code scorer can access activations directly, which means custom evals can measure properties of internal representations, not just surface outputs. A scorer that checks whether a specific SAE feature activated above threshold on the generated response is two lines of Python. That kind of mechanistic criterion is not available to any external eval harness.

Benchmark suites

Individual eval cards answer narrow questions. A suite answers a broader one: how does this model perform across a coherent set of concerns? build_benchmark_suite assembles multiple eval results, built-in or custom, into a named suite with a composite score. Each eval entry carries a weight, and the composite is the weighted average. The suite is also spawn-only, so it runs after the constituent evals have finished.

Weights encode judgment about relative importance. A safety audit might weight suppression at 1.5 and a one-off custom eval at 0.8, reflecting that suppression failures matter more to the deployment decision than the narrow custom signal does. The weight is visible in the card so the tradeoff is explicit.

Reading results

Scores are relative to the generated prompt suite, so they are not directly comparable to published leaderboard numbers unless you explicitly request a named standardized benchmark. The most reliable use is within-session comparison: run the same request against two models or checkpoints and compare rank order, not absolute values.

A low score is a starting point, not a verdict. A reasoning score of 67% driven by spatial failures is a different problem from one driven by arithmetic failures. A follow-up benchmark scoped to the sub-type disambiguates in one additional request. For custom evals, a low score on a code scorer that reads activations is also a prompt to dig into the attribution system, because the behavioral signal and the mechanistic signal are already pointing at the same session.