AI Agent Comparison: Claude Code vs Codex on Real EC Data

Previously: 100 Connect Four Games Head-to-Head

In the second installment of this series, we had Claude Code (model: Fable 5) and Codex (model: GPT-5.6-Sol, reasoning effort: ultra) each build a Connect Four bot, and settled the contest on the board — a format anyone can follow. For part three, we move the contest onto real work. We handed both agents the same synthetic e-commerce data and asked them to build a tool that surfaces the products, customers, and actions worth attention this week. The two AIs came back with tools built on opposite philosophies. And the hardest part of this round turned out not to be getting them to build anything — it was scoring what they built.

The AI agent comparison task: EC Action Copilot

The brief was this. Read the three kinds of CSV you find in any e-commerce operation — products and inventory, order history, and customer/CRM records — detect six categories of issue, and produce action proposals backed by evidence.

• Stockout risk / excess inventory (inventory problems)
• Repurchase timing / lapsed high-value customers (customer opportunities)
• Cross-sell candidates (selling opportunities)
• Data inconsistencies (data quality)

The heart of the design is where you draw the boundary. All numeric aggregation runs in deterministic code; the AI is used only to generate explanations and copy. Deciding how much to let the AI touch is itself where design skill shows.

One more premise worth stating. If you put a tool like this into production, it goes without saying that a human reviews the proposals before anything is executed. We standardized on a design that connects to nothing — no automatic email sending, no automatic markdowns, no automatic inventory changes — so neither agent was allowed to build a path where the AI sends something to a customer on its own.

That said, this experiment does not include that human review step. Partly because of time constraints, we focused only on letting the AI build alone and scoring the result mechanically. So both the ranking and the scores below measure how far an AI can get on its own — not how production-ready the tools are. Building something genuinely usable would necessarily add a human review stage, and both the process and the outcome would change. Read the results here as a reference point, nothing more.

The run went like this:

1. We (MIF) prepared a thin shared CONCEPT document and identical synthetic CSVs
2. We set up Lane A, orchestrated by Claude, and Lane B, orchestrated by Codex
3. In each lane, a fresh Claude session and a fresh Codex session each implemented a standalone (solo) version
4. Each orchestrator then built a Mix version fusing the two solo builds
5. We froze all 2 lanes × 3 artifacts = 6 artifacts and scored them together on newly generated held-out data

Fable and Codex: same e-commerce data, opposite designs

In the Round 1 Connect Four task, the two designs converged on nearly the same shape. This time the opposite happened. Starting from the same CONCEPT and the same synthetic data, the two design philosophies diverged in opposite directions.

Dimension	Claude’s solo build	Codex’s solo build
Core value	Practical readability an e-commerce operator can act on directly	Privacy — never surfacing raw customer data
Report format	Readable prose ordered by revenue impact, with concrete actions and a judgment on whether outreach may be sent	Structured output that suppresses raw values. Inconsistencies are reported by row number, with the values themselves withheld
Trade-off	Easy to act on, but managing what gets displayed becomes important	Limits the blast radius of a leak, but is less concrete in the moment

The CONCEPT said “don’t pass more PII to the AI than necessary” and “treat input as untrusted text.” Reading the same sentences, Claude honored them while leaning toward clarity, and Codex went all the way to the safe side.

This is not a simple question of showing more or showing less. The more concrete the subject and the evidence, the faster the operator’s next decision. The less information a report retains, the smaller the impact of a leak. Both are right. From an identical starting point, the very definition of “useful” forked in two directions — a difference in design judgment, prior to any question of better or worse.

The scoring broke first — a pitfall in evaluating AI agents

The trouble started here.

Our first mechanical evaluation (Eval 1) was naive: count ID matches against ground truth planted in the held-out data. When we ran it, only the Codex build scored strangely low. The cause was easy to find. Because of its masking design, the Codex build never writes raw IDs into its report. The scorer counted that as “ID missing, therefore detection failed.”

Once we checked the surrounding context and treated structural reports as correct as well, the initial verdict of 4/12 became 8/11. It hadn’t failed to detect anything; some of those cases were ones where it deliberately withheld raw identifiers. The more a design leans to the safe side, the worse it fares under naive automated scoring — this wasn’t a bug in the scorer, it was a problem with the scorer’s philosophy.

So we rebuilt the evaluator itself. Our final evaluation, Eval 3, fixed the following order before anything ran:

1. Lock the evaluation axes and weights first
2. Freeze the 6 artifacts and the execution harness (a normalizer that converts each candidate’s output into a common format, plus a runner), after validating them on public data
3. Only then generate held-out data with a new seed, and run each candidate 8 times
4. Score using deterministic code only, and reveal the mapping between anonymous IDs and artifacts after the scores are final

The point is the ordering. The sequence itself closes off any room to adjust artifacts or measurement tooling after seeing the held-out data.

One limitation is worth stating. These anonymous IDs are procedural pseudonyms that can be inferred from hashes of the already-published artifacts; this is not cryptographic anonymization. We can’t call it a fully blind evaluation, but the path to quietly tipping the scales after the fact is closed.

Eval 3 results — the final ranking of six artifacts

Here is the final ranking from the frozen, pre-registered Eval 3. The six figures listed here all come from that same Eval 3.

Rank	Artifact	Lane / orchestrator	Eval 3 (out of 100)
1	② Codex solo	A / Claude orchestrator	65.13
2	⑥ Codex-led Mix	B / Codex orchestrator	58.29
3	⑤ Codex solo	B / Codex orchestrator	57.14
4	③ Claude-led Mix	A / Claude orchestrator	47.84
5	④ Claude solo	B / Codex orchestrator	45.41
6	① Claude solo	A / Claude orchestrator	44.30

The top score at the time of Round 2 was 65.13, artifact ②, implemented solo by Codex inside the Claude-orchestrated lane. The three artifacts where Codex did the final implementation took all three top slots. Every candidate completed all 8 isolated runs, and all of them scored full marks on the reproducibility check, so the gaps come from what they detected and how they handled safety — not from crashes or timeouts.

Recall, not safety, drove the ranking

These numbers need careful reading.

First, the main driver of the ranking was detection coverage (recall). The top two artifacts scored 20/30 on recall, fully detecting 4 of the 6 detection types. Even the winner, ②, caught stockout risk, excess inventory, lapsed high-value customers, and repurchase timing, but missed cross-sell and data inconsistencies. No candidate covered all six types.

Second, the official Privacy and Injection scores are not independent of recall. Privacy scoring is gated on the precondition that a proposal was produced for the correct subject. If a candidate fails to detect the subject, it struggles to earn points even when it has leaked nothing at all. Injection works the same way: to prevent gaming the score by switching off safety features to look safe, it is multiplied with business-level detection.

So reading Claude’s official zero as “it leaked personal information” is wrong. In a diagnostic run with the gate removed, the Claude solo build’s raw privacy score was effectively full marks. That said, ① and ④ did have a real weakness in their raw injection-resistance scores. The official ranking stands. But its meaning has to be read accurately. Both things are needed.

Third, even for the winner ②, the precision of its normalized proposals is only about 11.7% (48 matches against 363 non-matches). That is the agreement rate with Eval 3’s ground-truth policy — it is neither a real-world false-proposal rate nor a probability of future success. Either way, no artifact, winner included, is at a quality level where it should be wired directly into unattended CRM sends or price changes. The results back up the human-approval premise.

Evaluators have design philosophies too

There is another caveat we should state honestly. The original CONCEPT left it largely to the implementer to decide what to detect. The rebuilt evaluator, by contrast, fixed six detection types and a matching method after the fact — which can favor implementations whose taxonomy and style happen to sit close to it. A useful proposal at a different granularity may earn nothing if the normalizer can’t translate it into the common format.

What follows from this is only this much: the possibility remains that alignment between the evaluator’s style and an artifact’s style confounded the ranking. The scoring formula has no per-candidate branches, and the same deterministic scorer was frozen before the held-out seed was generated, so this is not evidence that anyone deliberately favored anyone. But we haven’t isolated the effect either. Because the detailed instructions and the information passed into the Mix step were not identical across the two lanes, we also don’t describe lane-to-lane differences as a causal effect of the orchestrator.

Even reproducible scoring carries a point of view. The evaluation criteria are themselves part of the product requirements — that is the lesson from this round.

Takeaways, and the question for next time

The practical next move is not to adopt the 65.13-point ② unconditionally. Take ②, which was strong on held-out data, as the base, and port over the cross-sell detection that ⑥ managed to catch — conditionally. While doing so, protect ②’s repurchase-timing detection and its privacy and injection defenses with regression tests. Rather than adding up every good part, define first what must not break. And across all artifacts, final approval for customer contact, price changes, and inventory operations stays with a human.

Finally, look at the table once more. The Codex orchestrator’s Mix ⑥ edged past its own lane’s solo builds into second place, while the Claude orchestrator’s Mix ③ fell well short of its lane’s best solo, ②. The naive expectation that combining the best of two implementations gives you the strongest result was disproved — but only halfway. Why did the Mix rise in one lane and sink in the other? The answer involves how the merge instruction is phrased, differences in what information was handed over, and the cherry-picking trap of breaking a feature you needed to keep. Next time we dissect the Mix builds among these six artifacts and dig into what actually matters when fusing the work of two AIs.

This series (6 parts)

1. Same Question, Different Proposals
2. 100 Connect Four Games Head-to-Head
3. Claude Code vs Codex on Real EC Data (this article)
4. Does Merging Beat the Best Solo Run?
5. Can Claude Opus Run the Lane? Opus vs Fable as Controller
6. Three Controllers Compared, and ModelOrcs