In 30 seconds
01 · Scorecard
Scorecard
Every number comes from the instruments described below. "Sealed" means scored against the official table's cells, for every filing status its two columns cover and through the worksheet's second lookup, and against the publication's worked examples. The table itself was not in the bundle the models received (two of its cells appear in one worked example); the examples, with their answers, were.
02 · The table
The table is not the formula
The rule the table actually follows
03 · Readings
The reading register
The points found so far where the sources, or the sources and this schema, can be read two ways: the readings, the one the reference takes, what decides it, and the passages, each quoted from the sealed sources and checked against them. The points this schema takes as given facts (whether the filer is another taxpayer's qualifying child, the tiebreaker, the disallowance period, and the others the register's own description lists) are not in this register.
Which reading each formalization implements
04 · Disagreements
Where the formalizations disagree
05 · Coverage
Clause coverage
For each clause, the generator builds cases that toggle that rule (some toggles touch two or three clauses, and coverage credits each). A check mark means the formalization's answer changed when the rule was toggled; a cross means it never did, so the rule is missing, dead, or unreachable from the schema.
06 · Mutation
Mutation testing: what the test suite cannot see
07 · Sealed misses
Sealed-set misses
Cases with a sealed answer (the table's printed cells, the worksheet's second lookup derived from them, and the worked examples) where a formalization returned something else.
08 · Method
How it works
The formalization is a black box: a taxpayer's situation goes in, a credit comes out. Five instruments test it against the law; four treat it as a black box, and mutation testing rewrites its Python source.
Every rule, cited
Separate versions disagree
The table's cells, withheld
Break it on purpose
Hundreds of one-line changes are injected into each formalization. A change the suite passes is a survivor; a survivor some known input can see is a suite gap, and only gaps count against the suite (section 06).
Hashes on the sources
Every file under the never-edited folders (this provision's sources and sealed files, the two replications', and what each model wrote), the clause map and the reading register are listed with their SHA-256 in SHA256SUMS at the repository root. Two rules are checked: the tree matches the manifest (a changed, missing or unlisted file fails, and each formalization still hashes to what its run recorded), and the manifest is append-only for sealed files against every manifest main has ever had (a sealed file's line may be added, never changed or removed, so rewriting the manifest in the same commit as an edit fails too, and a bad commit on main is not cleared by the next one; only the clause map and register lines may be rewritten, deliberately). This page is built only after both pass, and prints the hashes from the manifest at its foot. Result files are not hashed; they are what this page is built from.
09 · Limits
What this does and does not show
- It finds holes; it cannot prove absence. A formalization that passes everything here can still be wrong in ways no instrument tested.
- Facts are inputs. Whether someone is "the qualifying child of another taxpayer" is given, not derived. The models were tested on applying the law to facts, not on finding facts.
- Disagreement locates, it does not judge. When versions differ, at least one is wrong. The reference and the inferred readings name the outlier; the sealed set holds no answer for a generated case.
- Models from two labs. The OpenAI models ran through Codex from an empty folder with a read-only sandbox, the Anthropic models through the Claude CLI with no tools; all five got the same bundle: the statute, the parameters, the Publication 596 text with its worked examples and their answers, the schema, and the table's construction rule stated in full (the bracket algorithm and the worksheet procedure). The bundle held no other model's code, none of the generated cases, and not the table itself (two of its cells appear in one worked example). What is observed and what is not: the Claude runs started from an empty folder with no tools, so the bundle is all those models could read, and each reply is kept beside its formalization; the Codex runs started from an empty scratch folder in a sandbox that blocks writes and network but not reads of this machine, and each session record is kept, holding the instructions Codex adds before the prompt and showing no tool call in either run. Model ids are as each tool recorded them: the Claude command line's reply names the model it used; Codex's session record names the model it was set to use.
- Some of the law is ambiguous. The table-versus-formula finding is the first entry in an ambiguity register; the register is itself a deliverable.