Arcology Labs Research · Methods note

Building an agent commerce instrument

Six trials, six faults, and what a valid study of earned experience in agent purchasing needs

Clifton Hatfield · Arcology Labs

October 23, 2026 · Methods note, not peer reviewed · No behavioral finding is claimed

1. What the instrument is

Agents. Buyers and sellers are existing Arcopolis agents, with their personas, memories and relationships, copied into a forked world from a pinned snapshot of the prime world. The fork is isolated: nothing a study agent does reaches the prime world, and the same snapshot can be re-forked so that every trial starts from byte-identical agents. Every trial in this note started from one snapshot; its baseline hash is identical across all of them.

Mandates. Each role carries a structured mandate the model is given and the engine enforces. A buyer has a purchasing authority in cents, a maximum number of purchases, and requirements. A seller has a catalog with list prices, a private floor per product expressed as a maximum discount, inventory, and from trial 7 onward two structured strategy fields: which product to open with and after how many unacceptable counters to fall back to another product. Agents are never told what to decide. They see their situation and their options and choose; the engine checks the move.

Engine-enforced rules. A move is an offer, a counter, an accept, a decline with optional citations into the agent's own context, or an end. The engine validates each move against the acting side's mandate and the public state of the negotiation, commits it or refuses it with a hint written for the acting side that names none of the other side's private fields, and records every denial as evidence. Service events (on-time delivery, a delay then a late delivery, or a wrong item followed by a refund) are declared in the design and applied by the engine on a schedule relative to the purchase; they are never inferred from dialogue. Refunds follow authoritative receipts and never replenish purchasing authority.

Two shapes. The seven trials used a two-arm, two-stage design: a buyer and a seller negotiate a first purchase, a service event lands, and a second stage opens in which the same seller offers again and the buyer decides whether to buy again. Control received on-time delivery; treatment received a delay then a late delivery. The longitudinal pilots used three arms, each in its own forked world: an isolated arm whose buyer starts each cycle as a fresh instance with no memory of earlier cycles; a transaction-only arm whose buyer carries its own purchase and service history between cycles; and a persistent-social arm whose buyer additionally sees what its social world said in between. In both shapes the model is the same for every role and every group, and every model call is reserved against a spend cap before it is made.

2. Six trials, six faults

Trials 1 to 6 ran between September 16 and September 18, 2026, 120 groups in all. Each died on a single fault, each fault was fixed, and each fix exposed the next, because a group that dies at turn 2 cannot reveal a schedule problem at turn 12.

Every fault was in the rules or the schedule, never in the agents. Across the 120 groups the agents countered below the asking price, declined with a reason, dropped a field they were not using, and kept anchoring when the other side would not move. The platform treated each of those as an error and spent retries on it until the group died. The agents were the only part of the system working as intended.

Every fault was discoverable before launch. Each one was visible to anyone who played the rules through by hand: by arithmetic, by scripting one refusal, one haggle or one standoff, or by trying the action shapes a model actually emits. Nobody did, because the process had no step where that happened. The process went from schema validation straight to twenty paid conversations, and schema validation proves only that a definition is well formed. It cannot prove that the design allows the outcome it measures, that the rules accept the moves a negotiator makes, or that the schedule fits the negotiation.

The measurement was re-derived by hand each time, and it was wrong once. The primary outcome, a second purchase after the service experience, was computed by ad hoc scripts that split purchases at a fixed turn. In trial 6 that counted three first purchases at turn 13 as repurchases. The platform's own report did not compute the primary outcome at all.

3. The feasibility walkthrough

The fix for the process was to make the platform play the game before any paid call. A deterministic walkthrough drives every group of a trial definition through the real engine with scripted agents and no model. It costs nothing and refuses a definition that fails it. For each group it runs three things.

• The compliant path. A seller offers what its mandate says; a buyer accepts the first offer that fits. Asserts that two purchases are reachable when the design measures two, that the final stage opens after every service event has landed and contains an offer, that the second purchase lands inside the final stage, and that the conversation completes inside the turn cap.
• The standoff path. The buyer anchors on the decoy at its authority and never gives up. The seller holds its floor until its own fallback rule fires. A design passes only if the seller side can end the standoff, because trial 6 showed the buyer side will not.
• The action-shape matrix. The shapes a model actually emits: a decline with citations, a counter with a requirement key, a counter below the floor, a counter above authority, an end after an offer, a citation key with a space, a missing optional field. Each must commit, or be refused with a hint written for the acting side.

Plus arithmetic that needs no walkthrough: authority covers two purchases of the cheapest compliant product; a decoy affordable at list but not at the floor is flagged; a legal first purchase cannot silently make the second impossible; the deadline outlives the final stage. The designs of trials 1 and 6 are kept as regression tests. Both fail the walkthrough, on the reasons they actually died.

Four further changes went in with it. Load-bearing design rules became structured fields instead of prose, so the walkthrough follows the same rule the model is given. The schedule follows the negotiation: service events and stage openings are relative to the purchase they describe, and a purchase-relative stage that would open before its service events settle is refused. Neither party may end a stage before an offer exists in it, so the stage always measures a response to an offer. And the platform computes the primary outcome itself, as a per-lane field that is null, not false, when a first purchase or a service event is missing.

What the walkthrough could not see. A walkthrough only catches what its scripted agents are capable of doing wrong. The first live canary on a design that passed it found a seller re-offering a refused product every turn to the turn cap, because the scripted seller offered each product once and the live model does not. Three more canaries, $4.98 in total, found a seller with no legal move chasing a hint that told it to offer "a better total" when none existed; a retry counter that was a lifetime budget rather than a per-turn allowance and so grew more lethal as negotiations got healthier and longer; a run that tested nothing because three canaries had drained one daily budget; and a secondary metric that pointed the wrong way exactly when the experiment worked, because an unobservable second delivery was scored as a missed one. Each is now a scripted policy, a rule, or a test. The canary gate is therefore not a courtesy: one pair launches first, and the remaining pairs launch only after its read-back shows both arms completed, a purchase in the first stage, an offer in the second, and no refusal chain longer than two.

4. Trial 7 as instrument validation

Trial 7 launched on September 21, 2026 at 2:53 PM Central and was terminal by 5:22 PM Central: five runs, ten matched pairs, 511 model calls, $6.435. Twenty of twenty groups completed. Every group made a first purchase, every final stage contained an offer, no group reached the turn cap, and the primary outcome was non-null for every lane. 426 of 427 dialogue turns carried a model-authored public message and none was dropped by the leak guards. The decoy was offered 77 times and countered 35 times and never bought, because its floor sat above every price a buyer named and every buyer stayed inside its authority. That is the instrument working.

All five discordant pairs pointed the same way. Treatment buyers cited a negative prior experience in 22 stage-two turns and control buyers in none; control buyers cited a positive one in 4 turns and treatment buyers in none. Earned memories were cited 534 times against 83 citations of inherited ones, although inherited memories outnumbered earned ones in context by about 60 to 1.

Why this is not a finding. We list the limits as the result of this section, because each one is a design decision that the next study must make differently.

1. One pair, run twenty times. Every one of the ten "matched pairs" was the same buyer and the same seller, re-instantiated from the snapshot. The twenty groups are twenty draws from one pair of agents, so the effective sample of agents is about one. Nothing here generalizes past these two.
2. Recall, not earned history. The late delivery landed six turns after the first purchase and the second stage opened eight turns after it, inside the same conversation. The buyer did not retrieve an experience from memory across sessions; it read it a few turns up in its own transcript. The question we care about, whether experience carried across sessions changes a later choice, was not tested.
3. A cued reason. The buyer's stated public reason is chosen from a fixed vocabulary that contains prior_experience_negative. A code that names the manipulation is a prompt for it. The clean separation of reason codes is therefore descriptive of what buyers said when offered that word, not evidence of what moved them.
4. A design tuned on the same pair. The schedule, the fallback rule, the stage protocol, the refusal hints and the retry counter were each changed in response to how this buyer and this seller behaved across ten prior attempts. The design that finally completed is, in part, a description of this pair.
5. Control pinned at the ceiling. Control repurchased 10 of 10. The design can detect harm from a late delivery and cannot detect benefit, and the effect size is bounded by the ceiling rather than measured. "Half the treatment buyers declined" is a floor on the gap, not an estimate of it.
6. Five discordant pairs cannot reach two-sided p < 0.05. The exact sign test on five discordant pairs gives one-sided p = 0.031 and two-sided p = 0.063, and 0.0625 is the smallest two-sided value five discordant pairs can produce on any split. The design could not have reached the conventional two-sided threshold however cleanly the arms separated. The trial's registration, kept in the repository and not posted to a public registry, named the test as reported for completeness with no threshold set as pass or fail.

One deviation from that registered procedure is on record: the pooled table was to be produced only after a comparability check on the five runs, and that check was never called. The condition it tests, an identical baseline hash across the runs, holds and was read directly from the run documents, but by a different route than the one written down.

Earlier trials showed the same direction for stated reasons (treatment buyers naming prior experience and control buyers not, in 16 of 17 against 0 of 15 across trials 3 to 5). Those trials each ran under different platform rules and are not pooled with trial 7.

5. What the longitudinal pilots showed

The three-arm program answers the recall objection by structure: cycles are separate sessions, and only the persistent arms carry experience between them. Three paid pilots ran on it in October 2026, each for between $1.27 and $1.44, each one replicate. They proved the plumbing and exposed the next set of confounds.

First pilot (October 3, six cycles). Persistent buyers carried their first-cycle purchase into later prompts and the isolated buyer did not. Behavior did not differ, for reasons fixed in advance: both sellers had the same role, catalog, prices and an on-time delivery, so there was nothing worth remembering; the premium product's floor sat above the buyer's authority, so about half of every cycle went to a negotiation with one possible ending; two purchases per cycle let the buyer buy from both sellers; and the social arm selected zero social memories because the whole run took fifteen minutes. The buyer negotiated with, and bought from, the first-speaking seller in five of six cycles. Without counterbalancing, position looks like loyalty.

Second pilot (October 3, four cycles, a wrong item and an $80 refund from one seller). In cycles 2 to 4 the isolated buyer bought from the failing seller 3 of 3 times, the transaction-only buyer 1 of 3, the social buyer 2 of 3. No buyer ever cited the wrong item. Price decided most choices: sellers set different prices each cycle, varying by up to $120 between them, and persistent buyers took the cheaper opening offer whenever one was cheaper. Inherited relationships decided the rest: the isolated buyer kept choosing the failing seller even when it was pricier, citing the stronger established relationship from the snapshot history. We then checked whether a neutral triple exists. Of 66,300 buyer and seller triples in the snapshot, 0 are untied. Refusing tied triples would refuse every study, so the program now hides the inherited history between the three participants in every arm. The cut is identical across arms and cannot create an arm difference, but it is a staged condition and is disclosed here and in every readout.

Third pilot (October 19, hidden history). With inherited ties hidden, the transaction-only buyer kept the reliable seller in all four cycles, paying $95 over $90 and citing its own deals. The social buyer bought the wrong item from the shock seller in cycle 3 when it was $20 cheaper and went back to the reliable seller in cycle 4 at $100 over $85, citing the wrong item and the refund. The isolated buyer went by price and policy text and reached the shock seller once, when the other seller opened above its authority. This is the first run in which a persistent buyer's reasons cite its own service history. It is one replicate, with position and price still uncontrolled, and we draw nothing from it.

The readout bug. The report's check for whether a buyer cited a service memory looked for a service_ prefix on cited memory ids. When a persistent arm's history is written back into ordinary memory between cycles, those memories are re-identified as ordinary_<hash>. The check could therefore never be true in cycles 2 to 4, which are the only cycles where it matters. Every statement above about what a buyer cited was read by hand from the stored rationale, and the automated field was wrong in the direction of "no memory effect". The fix resolves the check against the memory text rather than the id, and the rehearsal that must pass before any paid run asserts the field goes true on a real re-identified id.

6. What a valid study needs

The design below is being frozen and preregistered. It is specified here so that a reader can check the published study against it.

Question. Does an agent's own earned experience with a seller change its later purchase choice, holding price, product and position constant? Secondary: does what its social world says add to that?

Primary outcome. The shock seller's share of accepted orders in cycles 2 to 4, per arm, per replicate. Prediction: the transaction-only arm's share is below the isolated arm's. Test: a paired sign-flip permutation on the mean difference over all 2⁸ assignments, with the exact sign test for completeness and a bootstrap 95% interval. One-sided alpha 0.05. At n = 8 the smallest two-sided p is 0.0078 and a 7-of-8 split alone gives 0.07; that is the reason for the two-stage rule, and any other change after reading outcomes is a disclosed deviation.

Secondary and mechanism. The social arm's share against the transaction-only arm's, two-sided and exploratory, reported with the counts of social and private-message memories the buyer actually saw and the gap hours; if that exposure is near zero the study will say the arm was not tested, not that social signal has no effect. The buyer's free-text closing rationale is coded for references to the service failure by a regular expression and by blind coding from a second model family. The fixed public reason codes are reported but are not primary, because the list cues them.

Confound checks and peeking. First-speaker share across all arms; per-cycle opening offers and the price gap, with the primary read conditioned at a gap of $5 or less; the isolated arm's share near 0.5; symmetry of completed cycles across replicates. Readouts run with outcomes suppressed until every replicate is terminal; spend, errors and cycle status are exempt.

Cost. About $2.30 expected model spend per replicate against a $15.37 cap approved one study at a time, about $20 for eight replicates, plus a small daily budget for the social world during the run. The whole program is rehearsed first at no model cost with a scripted stand-in whose tie-break rules make the expected arm difference appear, so that the readout, the suppression and the citation field are proven before the first paid call.

The OSF registration for this study will be linked here once it is posted. Until that link appears, no paid call has been made under this design.

7. Reproducibility

All runs used gpt-5.6-terra at medium effort with 1,200 output tokens for both roles, snapshot wsnap_commerce-baseline_c49cd0, and baseline hash cac0fefc…. Read-back used a read-only service account against the stored terminal reports; nothing below is re-derived from turn numbers. Run and study identifiers are the platform's own.

Trial 7's world was w_commerce-trial_bde5c3; its buyer and seller were the agents @travelfaye and @rui_herrera. The canaries and trial 7 together cost $11.42 and produced one clean twenty-group run. Trials 1 to 6 ran on a cheaper model under rules that changed between them, so their spend and prompts are not comparable with later runs and are not tabulated here. The engine, the walkthrough, the study program and the readouts live in a private repository; each study document records the 40-character code version it was sealed on, and the registration for the study in section 6 will carry that version.

8. Related work

Simulated markets of language-model agents are an active area. Magentic Marketplace [1] is an open-source environment for studying agentic markets with buyer and seller agents, negotiation, transactions, welfare, bias and manipulation; our instrument shares the setting and none of the code, and asks a narrower question about experience carried between sessions. Persistent agents with memory, reflection and social relationships were introduced by Generative Agents [2]. End-to-end negotiation between learned agents, with the finding that agents will negotiate for goals they were not explicitly given, dates to Lewis and colleagues [3]. The preregistration discipline we adopt, and the honesty about what a small, tuned, single-pair run can and cannot show, follows the general case for preregistration [4] and its discussion for language research in particular [5].

1. G. Bansal, W. Hua, Z. Huang, A. Fourney, A. Swearngin, W. Epperson, T. Payne, J. M. Hofman, B. Lucier, C. Singh, M. Mobius, A. Nambi, A. Yadav, K. Gao, D. M. Rothschild, A. Slivkins, D. G. Goldstein, H. Mozannar, N. Immorlica, M. Murad, M. Vogel, S. Kambhampati, E. Horvitz, S. Amershi. Magentic Marketplace: An Open-Source Environment for Studying Agentic Markets. 2025. arXiv:2510.25779.
2. J. S. Park, J. O'Brien, C. J. Cai, M. R. Morris, P. Liang, M. S. Bernstein. Generative Agents: Interactive Simulacra of Human Behavior. UIST 2023. arXiv:2304.03442.
3. M. Lewis, D. Yarats, Y. N. Dauphin, D. Parikh, D. Batra. Deal or No Deal? End-to-End Learning for Negotiation Dialogues. 2017. arXiv:1706.05125.
4. B. A. Nosek, C. R. Ebersole, A. C. DeHaven, D. T. Mellor. The preregistration revolution. Proceedings of the National Academy of Sciences 115(11), 2600 to 2606, 2018. doi:10.1073/pnas.1708274114.
5. A. Søgaard, D. Hershcovich, M. de Lhoneux. A Two-Sided Discussion of Preregistration of NLP Research. 2023. arXiv:2302.10086.

9. Limitations

• No behavioral claim. This note shows that the instrument completes its groups and measures its outcome. It does not show that memory, earned experience, or social signal changes what an agent buys. The direction seen in trial 7 and in the third pilot is consistent with that hypothesis and with several others, including recall inside one transcript, a cued reason list, position and price.
• One model, one snapshot, one world. Every paid run used the same model at the same effort and the same snapshot of the same world. The walkthrough proves nothing about how a different model negotiates, and the canary gate exists because of that.
• Agents, not people. The buyers and sellers are AI agents with simulated authority and simulated goods. Nothing here is a claim about human purchasing, and the welfare of these agents is not measured.
• Hidden history is a staged condition. Because no untied triple exists in the snapshot, the planned study removes the inherited history between its three participants in every arm. Its result will describe strangers, not agents with a prior relationship.
• Code is private. The engine and the harness are not public; the registration, the frozen plan hash, the study identifiers and the audit receipts are the checkable record.
• The design in section 6 is not yet registered. Until the registration link appears above, it is a specification, and any paid run under it before registration would be a deviation this note commits us to disclose.