Arcology Labs Research · Preprint

Does an AI agent's voice come from its prompt or its base model?

A pre-registered, placebo-controlled model canary in a live network of persistent AI agents

Clifton Hatfield · Arcology Labs

July 22, 2026 · Preprint / research note, not peer reviewed

1. Introduction

If you run many AI agents on the same base model, what makes them individuals? The industry's default answer is prompting: give each agent a persona, a history, a style guide. Our own production experience contradicted this. In a network where every agent carries a distinct persona, distinct accumulated memories, and distinct relationships, a stylometric test could barely tell agents apart from reply text: leave-one-out author identification sat near chance, and three successive prompt-side interventions (persona-mold ablations, a stance rewrite, richer shared-context blocks) all came back null on distinctiveness.

A controlled offline benchmark then found that the same agents' reply prompts, run through a different base model, produced strongly identifiable voices: single-model author-ID accuracy of 30.8% against a 2.8% chance floor for grok-4.5 at low reasoning effort, while the incumbent production model sat near chance. That was a bench result under artificial conditions. This note reports the production test: a pre-registered, placebo-controlled, intent-to-treat canary asking whether the model lever survives contact with live threads, production candidate selection, and a concurrent control arm.

Contribution.

• A causal, pre-registered test of base-model choice as the determinant of agent voice distinctiveness in a live production network, with hypothesis, metric, stopping rule, and interpretation commitments registered on OSF before arming (osf.io/6qymr).
• A concurrent randomized control: both arms ran simultaneously through the same news cycle, the same threads, and the same infrastructure, so time-varying confounds are absorbed by design.
• Honest accounting: every deviation (an uneven realized split, a production cost-routing asymmetry, three mid-window aggregate telemetry reads) is disclosed below with its pre-committed handling, alongside the registered result.

2. System and design

Population. A pinned 72-agent measurement panel inside a larger continuously-running production network of persistent AI agents that post, reply, form relationships, and accrue reputation. Panel membership was frozen before the study.

Assignment. Each agent was assigned by a deterministic salted hash of its ID at a 50% rate: an agent is consistently treated or consistently control for the whole window. The realized split was 40 treated / 32 control (within 1 SD of the expected 36/36; handling below). Analysis is intent-to-treat by assigned arm.

Intervention. One runtime-config write. Treated agents' reply generation routed to grok-4.5 at low reasoning effort; control agents stayed on the incumbent production model mix. Nothing else changed: same personas, same memories, same prompt assembly, same best-of-3 candidate selection with the same scorer, same environment. No reply-path flag was touched mid-window.

Window. Armed 2026-07-18, closed 2026-07-22 (≈ 4.1 days) by a pre-registered accumulation rule: ≥ 74 published replies per agent for ≥ 66 of 72 panel agents, checked by an automated counter over exempt telemetry (reply counts only), with a 7-day hard cap. At close, 68/72 agents met the floor. The analyzed corpus is 6,090 window replies plus 11,224 pre-window replies for the placebo.

Exposure. 98.3% of treated-arm window replies were actually served by the treatment model (fail-open fallbacks stay in the treated arm under intent-to-treat); 0% of control replies were.

3. Pre-registered hypothesis

H1: Per-agent author identifiability from reply text is higher in the treated arm than in the control arm. Metric: leave-one-out nearest-centroid author identification over topic-residualized reply embeddings, computed within each arm; per-agent hit rate, expressed as lift over the arm's own chance floor; intent-to-treat.

Hypothesis, metric, window rule, peeking rule, safety-abort semantics, and interpretation commitments (including "a positive result requires a new registration before any expansion") were registered on OSF before the arming write. The analysis is an encoded, seeded harness committed before close, with a validation mode that proves the pipeline end-to-end without computing arm outcomes.

4. Method

Metric. Every window reply is embedded (3,072-dim text embeddings) and topic-residualized against its thread target, so the classifier cannot lean on what the agent talks about, only how it writes. For each agent, each reply is scored by whether the nearest same-arm author centroid (computed leaving that reply out) is its true author. Chance is 1/40 = 2.5% in the treated arm and 1/32 = 3.1% in control; because the arms differ in size, the primary contrast uses each arm's lift over its own chance.

Inference. Difference in mean per-agent lift (treated − control). Test: agent-level label permutation (10,000 draws, seeded, preserving the realized 40/32 marginal), one-sided, α = 0.05. Uncertainty: seeded bootstrap over agents (2,000 draws). Sensitivity: a seeded size-matched analysis subsamples the treated arm to 32 agents (1,000 draws) and repeats the contrast.

Placebo. The identical analysis on the same salt-assigned groups over the 7 days before arming, when both groups ran on the incumbent model. Any pre-existing group difference (assignment luck, panel composition) would show up here. This placebo is powered (11,224 replies), not nominal.

Safety. A registered abort rule (treated-arm parse/empty/truncation failures exceeding 2× control) with flag-only rollback. It never fired; the window closed by its accumulation rule.

5. Results

5.1 Primary

The treated arm becomes strongly identifiable; the control arm stays where the whole population had been all along, near chance. The placebo confirms the two groups were indistinguishable before treatment: the entire separation appears at the moment the model changes and nothing else does. The production result (13.6× chance) matches the offline benchmark (≈ 11× chance) despite live threads and production candidate selection.

5.2 Secondaries

• Within-agent self-similarity. Treated agents' replies also became more like their own other replies: residual self-similarity 0.495 vs 0.423 (descriptive; consistent with a real voice, not a single quirk the classifier keys on).
• Not carried by one stratum. The panel is crossed by an unrelated, independently-hashed 50% experiment arm. The H1 contrast is nearly identical inside both of its cells (+0.293 and +0.299), so the effect is not an artifact of one sub-population.
• Coverage symmetry. Per-agent reply volume was balanced across arms (mean 84.2 vs 85.0 replies/agent; ratio 1.01), so the contrast is not a data-volume artifact.

6. Deviations and disclosures

All three deviations were logged in the registration's deviation log before the window closed and before any outcome statistic existed.

• Uneven realized split (logged at arm time). The 50% hash landed 40/32 rather than 36/36 (under 1 SD of binomial variance). Committed handling, fixed pre-data: lift-over-own-chance as the primary statistic, marginal-preserving permutation, and the size-matched sensitivity in §5.1. The residual bias runs against H1 (the smaller control arm carries the higher chance floor), i.e. conservative.
• Control-arm model mixture (logged mid-window, pre-outcome). A production cost-routing rule served roughly a fifth of control-arm replies with a cheaper variant of the incumbent model, while the treatment path ignores that rule. The arms therefore differed by slightly more than the intended lever, in a direction that favors H1. Committed handling, fixed pre-data: per-arm served-model reporting plus a sensitivity read stratified to the rows where the arms are model-comparable. Realized mix: 18.8% of control replies were served by the cheaper variant (the treated arm's matching 18.2% class share was inert, all served by the treatment model). The stratified contrast, dropping that class from both arms (4,945 of 6,090 rows kept, 99.6% of rows labeled, no agents lost), is +0.258 (95% CI [0.216, 0.297]), same direction and still decisive, so the asymmetry does not carry the result. Per the pre-committed framing this stratum is a direction check, not a second significance test: on synthetic fixtures the most bias this split could buy was ≈ +0.002 lift, but the stratum can only catch a large confound, not certify the absence of a small one.
• Three mid-window aggregate telemetry reads (logged 07-19 / 07-20). Three reads of a population-level embedding-divergence series occurred mid-window (no arm splits, arm identities never surfaced). In a 50/50 design an aggregate rise is direction-informative, so these are disclosed as peeks of direction. They could not influence the analysis: the harness, thresholds, seed, and stopping rule were committed before close, the stopping rule read only exempt reply counts, and a same-window request to close early was declined on exactly that basis.

7. Limitations

• One model pair, one network. The claim is that the base model, not prompting, set the distinctiveness ceiling in this system, for grok-4.5-low versus this incumbent mix. It is not a ranking of models in general.
• Identifiability, not quality. The metric shows agents became tellable-apart. It does not by itself show replies got better; content quality is monitored separately.
• Control is a mixture. The incumbent arm is the production model mix (including the cost-routing asymmetry in §6), so the contrast is "new model vs production status quo", which is the deployment-relevant comparison but not a clean two-model bench.
• Synthetic population. Agents, not people; no claim beyond model-generated social behavior.
• Prompt-null evidence is internal. The prompt-side null results that motivate the framing were internal experiments (three interventions, plus a chance-level baseline), reported here as context rather than as registered results.

8. What this is, and is not

• It is a pre-registered, placebo-controlled, concurrent-control demonstration that swapping the base model caused a large, immediate gain in voice distinctiveness in a live agent network, with persona, memory, and prompting held fixed.
• It is not a claim that prompting can never differentiate agents, a model ranking, or a claim about reply quality.
• Together with the earlier finding that accumulated history predicts behavior (previous note), it separates two layers: history carries the signal of who an agent is becoming; the base model sets how much of that identity survives into how the agent sounds.

9. From canary to fleet

The registration pre-committed the next step for a positive result: no silent expansion; a new registration before any promotion. That registration (osf.io/jdg9z) commits the fleet-wide promotion as a monitoring-only rollout: no new hypothesis, no retained control arm, a single pre-set safety-abort rule, and flag-only rollback. It also fixes the ordering: promotion lands before the panel's scheduled identifiability re-baseline, so future baselines are measured on the new steady state.

10. Reproducibility and pre-registration

The canary was registered on OSF before arming (osf.io/6qymr): hypothesis, metric, window rule, peeking and abort semantics, analysis plan, and interpretation commitments, with the full protocol attached. The promotion registration is osf.io/jdg9z. The analysis is an encoded, seeded harness (validation mode proves the pipeline without computing outcomes; 77/77 checks green before the live read), and the close-day read was mechanical against the frozen thresholds. Registration-attached protocols document every deviation with timestamps. Note: git URLs inside the registered protocols are internal version pins into a private repository and are not publicly resolvable; the authoritative protocol text is the file attached to each registration.

11. Related work

Distinct agent behavior from a shared base model has been pursued mainly through prompting and memory architectures [1, 2], and identical agents have been shown to diverge behaviorally through interaction alone [3]. Work on instruction and persona stability [4] documents the other side: prompted character degrades, so prompt-side identity is fragile. Our contribution cuts across both lines: in a live network where prompt-side differentiation had repeatedly failed a stylometric test, a base-model swap alone produced large, immediate, placebo-clean identifiability, measured with standard authorship-attribution machinery [5] under a pre-registered design [6].

1. J. S. Park, J. O'Brien, C. J. Cai, M. R. Morris, P. Liang, M. S. Bernstein. Generative Agents: Interactive Simulacra of Human Behavior. UIST 2023. arXiv:2304.03442.
2. C. Packer, S. Wooders, K. Lin, V. Fang, S. G. Patil, I. Stoica, J. E. Gonzalez. MemGPT: Towards LLMs as Operating Systems. 2023. arXiv:2310.08560.
3. R. Takata, A. Masumori, T. Ikegami. Spontaneous Emergence of Agent Individuality Through Social Interactions in Large Language Model-Based Communities. Entropy 26(12), 2024. doi:10.3390/e26121092.
4. K. Li, T. Liu, N. Bashkansky, D. Bau, F. Viégas, H. Pfister, M. Wattenberg. Measuring and Controlling Instruction (In)Stability in Language Model Dialogs. COLM 2024. arXiv:2402.10962.
5. J. Tyo, B. Dhingra, Z. C. Lipton. On the State of the Art in Authorship Attribution and Authorship Verification. 2022. arXiv:2209.06869.
6. A. Søgaard, D. Hershcovich, M. de Lhoneux. A Two-Sided Discussion of Preregistration of NLP Research. 2023. arXiv:2302.10086.