Engineering log

Most project histories list what was shipped. This one lists what was wrong — because in a system whose whole promise is calibration, the interesting events are the ones where the system, or its measurements, turned out to be lying.

This page is safe to hand-write, which most of the documentation here is not. Everything else in this repository describes the code as it is now, so it rots the moment the code moves — which is why those pages are anchored to the sources they describe and checked on every push. A dated record is different. It says what was true on a day, and days do not change.

Who caught what

The figure covers July and August, and it is not redrawn as the page grows — a dated record says what was true on a day, and the months below it have their own sections. Twenty defects fell in that window, sorted by what actually surfaced each one. The shape matters more than the count: automated checks caught the infrastructure problems, measurement caught the data problems, and the single most consequential defect — the one that had passed every automated check for weeks — was caught by one sentence from a person.

⚠ The count is a hand count, and the page states elsewhere why it has no guard: what counts as an entry here is ambiguous — a heading, or a bold paragraph? — and a counter would pin that ambiguity into code. So the number below is not extended month by month. September is not tallied, and its section says why the tally would mislead: its entries are not defects the system had, but readings of it that were wrong.

---

July — the checks that were not running

The tests that had never run

2026-07-10 · ci(nexus): run the DB-backed tests — and repair the fixtures that never ran

The database-backed integration tests looked healthy in the sense that nothing was red. They had never executed: a fixture scope mismatch made them fail during setup, which the runner reported as an error the suite tolerated. Months of “passing” told us nothing about the paths that touch a real database.

The repair added a Postgres CI job that runs the full suite against a real server with migrations applied. The rule it left behind: a skipped test is not a test, and a net you have never seen catch anything has not been shown to work. Since then, when a guard is added, it gets deliberately broken once to confirm it goes red.

A test run destroyed a real corpus

2026-07-10 · feat(nexus): document lifecycle across web, MCP and CLI + a guard against wiping a real DB

The suite truncates tables between tests, which is correct for a scratch database. It was pointed at the development database, and the corpus in it was gone.

The fix was not “be careful with the connection string”. A database now has to declare itself disposable — the suite looks for a marker table and exits with a hard failure if it is absent. Care is not a mechanism; a refusal is.

---

August — the measurements that were wrong

The figure in the document was not the figure the run produced

2026-08-04 · docs: correct the coverage figure — 232 chunks was wrong, the run says 10

A coverage number had been written into a document from memory rather than from output. Re-reading the run showed it was off by more than an order of magnitude. Small on its own, and the reason it is here is that it was the first of a pattern: over the following two weeks, the instrument was wrong more often than the system was.

A control that had been specified and skipped overturned a verdict

2026-08-06 · fix(ko-eval): run the calibration control that was specified and skipped - it overturned a judgement

An evaluation design called for a negative control — an input that should produce the same result if the effect being claimed were not real. It had been written into the plan and then not run, because the primary result looked convincing. Running it reversed the conclusion.

Standing rule since: a result without its control is not a result, and the control is run before the verdict is written down, not after it is doubted.

A partner’s fingerprint came back a month after it was scrubbed

2026-08-07 · chore(scrub): remove a partner organisation's fingerprint, and count it on every push

The repository was scrubbed of identifying details before it went public. A month later they had returned through ordinary work — an address here, a real page identifier there, and, in the blind spot nobody was watching, inside commit messages and pull-request bodies.

A scanner now runs on every push. It reads the API for the current title and body rather than the frozen webhook payload, so an edit cannot slip past it, and it carries negative controls so that a scanner which has stopped detecting anything fails loudly instead of passing quietly. The allowlist takes reasons, not exceptions: when a test later tripped it, the fix was to use synthetic identifiers, not to add a line to the allowlist — a list that grows one exception per incident is a hole with a schedule.

The repository had two schemas

2026-08-09 · fix(ci): the recall job never ran migrations, so the repo had two schemas

One CI job created its database from the baseline file and never applied the migrations on top. It had therefore been testing a schema that no deployment has ever run, and passing.

Ingestion wrote into a column that nothing searches

2026-08-11 · fix(nexus): the corpus never said which generation serves it, so two processes disagreed

The embedding model, its dimension, and the vector column it writes to are one unit — a generation. The documented command, run on the host, resolved that unit from configuration defaults; the container resolved it from environment variables. They disagreed, so a perfectly successful ingestion wrote vectors into a column no query reads. Nothing failed. The documents were simply not findable.

A corpus now declares its generation in the database, append-only, and ingestion is refused before a single document is collected if the running process resolves a different one. The same change also established that when chunk text changes, every vector column is invalidated — a bug an earlier decision record had declared dead came back the moment a second column was added.

The number existed, and nobody was ever shown it

2026-08-11 · fix(nexus): the coverage number existed, and no one was ever shown it

Index coverage — how many chunks each retrieval path can actually see — had been computed correctly all along. It was written to the API start-up log, where no human looks. The command people actually type showed nothing, and a gap of 51 chunks sat open for a day.

This is the most frequently repeated shape in this log, and it has now appeared four separate times: the detector exists, the delivery does not. The rule that came out of it is that a signal is worth zero until something a person or an agent actually reads is showing it, and that a check on the detector alone stays green when the delivery is deleted — so the test has to run the surface.

Half the failures were mine

2026-08-12 · feat(nexus): measure a second corpus, and find out half the failures are mine

Running the evaluation against a second corpus split the failures cleanly: some were the system’s, and the rest were defects in the grader itself — a label pointing at a document that did not contain the answer, an abstention detector that a fourth phrasing walked straight through, a citation format the verifier could not resolve.

The abstention detector is the instructive one. It had been a list of known refusal phrasings, and the next run produced a phrasing that was not on the list, so an honest refusal was scored as a hallucination. A list of observed strings is not a detector. It was replaced with a structural rule: a refusal names the evidence and negates it, which does not depend on having seen the sentence before.

The grader could no longer tell systems apart

2026-08-13

Both evaluation packs reached their ceiling. The scores had gone up over the preceding week — and every point of that gain traced back to fixing the instrument. Retrieval and generation had not been touched.

The two packs were reclassified as regression suite, not quality evidence, and the repository stopped citing their totals as a measure of how good the system is. A measure that cannot separate two systems is not measuring them.

The grader passed a defect it was structurally unable to see

2026-08-18 · feat(nexus): tell the user when the evidence does not fit the question

The automated score said 7.7 out of 8. The team it was deployed to was not using it. Asked why, one person said the answers felt off — and that sentence opened a defect that every automated check had passed for weeks.

Reproduced: asked where a tool’s name came from, the system filled all ten evidence slots and answered at length with a technology table and an API response shape. Not a hallucination. The citations resolved, the grounding check passed, the fact check passed. No available measurement could see it.

The cause was in the fusion step. Reciprocal rank fusion scores a result by 1/(k + rank), so it carries rank and discards magnitude — and both retrieval paths had already computed magnitude to sort by, then thrown it away on return. Restored, the two populations separate cleanly:

	vector distance	keyword score
answerable	0.191 – 0.455	2.0 – 4.5
topic present, answer absent	0.377 – 0.470	0.8 – 1.2
outside the corpus	0.544 – 0.575	0.1 – 0.6

Weak evidence does not block an answer — it changes the narration contract, so a wrong threshold costs a short answer rather than a wrongly withheld one. The thresholds are recorded as a hypothesis from seventeen authored questions, and every request now stores the two magnitudes so that real usage, not invention, can eventually set them.

A correction belongs with this entry: two of the six questions written to represent topic present, answer absent turned out to be answerable, and the system answered them correctly with citations. The absence check had been run with the author’s vocabulary rather than the corpus’s. Re-split, the reading is stronger — but the lesson is that an absence proved with your own words is not an absence.

A document that no retrieval path could read

2026-08-18 · fix(nexus): keep a chunk's generation key with the document it belongs to

Reviving a soft-deleted document restores “only the current generation” of its chunks, decided by comparing a key on the chunk against the document’s content hash. Re-ingestion updated the document’s hash and never moved the chunk’s. So any document that had been edited once carried chunks stuck in the past, and a later delete-then-revive stood the document back up with zero readable chunks — listed, counted, and reported healthy while no retrieval path could read a word of it.

The live corpus the team queries had one, with eight more waiting on the same trigger, which is not a human command but a scheduled reconciliation job. The blind spot that hid it is worth naming: coverage is computed over chunks, so a document with none is outside the population entirely and reports as fully covered.

The front page had been describing a retrieval path that changed

2026-08-21

The illustration at the top of this site said a query fans out to three retrievers which fuse via reciprocal rank fusion. Two legs fuse. The graph lookup runs separately and attaches after the diversity cut, contributing nothing to the ranking.

That exact error had already been found and corrected six days earlier — in the agent instruction file, which is anchored to search/hybrid.py and therefore watched. The correction never reached the public page for one reason: the home page was not in the anchor list. The most-read description of the retrieval architecture was the one thing the drift net was not watching, so it drifted the longest — thirty-four commits to the code it describes.

Both language versions were corrected and both are now anchored. The generalisation is uncomfortable and worth keeping: a net protects exactly what is registered with it, and the pages most likely to be omitted are the ones that feel like marketing rather than documentation — which are also the ones most people read.

Fifteen out of fifteen, with the change switched off

2026-08-26 · feat(nexus): the answer-fact ruler measured mention, not assertion — split them

A new grader was written to measure something retrieval metrics cannot see: whether the answer actually contains the value the question asks for. It read 15 out of 15. Then the retrieval change it had been built to evaluate was switched off, and it still read 15 out of 15.

Both facts had one cause. A substring test cannot distinguish a value that is claimed from a value that is merely listed. The failing answer wrote 4,000 into a table of what each source says, then closed with “until this is confirmed, no figure can be asserted.” The grader counted that as correct.

The replacement asks where the value stands. Either in the lead, meaning the prose before the first table, quote or heading, which is the place the system’s own prompt reserves for the answer. Or in a verdict segment, one opened by a conclusive connective. Everything else is laying evidence out.

Three rounds per arm, majority per question, noise floor read before any test:

	with the change	without it	band	gap
substring	15 · 15 · 15	15 · 15 · 15	0	0
assertion	14 · 13 · 14	9 · 9 · 10	1	5

The gap clears the noise. The sign test still does not run: five discordant pairs against a pre-registered minimum of six. All five point one way and none the other, and the claim stops there. More rounds cannot fix it, because discordant pairs are a property of the question set rather than of the sampling.

The rule it left behind is now applied before measuring anything: ask whether the grader can see the treatment before reading its score. A grader pinned at 1.000 is not a strong result. It is an absent one. The same pass turned up a second defect in the substring version, which accepts a value that appears only inside a quotation of a superseded policy while the answer concludes something else entirely.

The instrument was counting our own re-runs

2026-08-26 · fix(nexus): the re-ingest signal counted our own re-runs — give it a denominator

One of the entropy signals counts re-ingest overwrites: documents whose content changed under an existing identity. An accepted architecture decision designates it a demand-pull trigger, and several deferred items are gated on it, so this number decides what gets built.

The day before, that view had been repaired. It aggregated globally, so throwaway evaluation tenants were drowning the live signal: 61,425 duplicate pairs globally against 0 in the live corpus. Splitting it per tenant was the repair.

It was half the repair. The live number reads 53 events, spread across 18 documents, 38 of them inside three days, in a corpus of 126. Those three days are days the ingestion pipeline was being changed and re-run. The signal was measuring how often we re-ingested the same eighteen documents, not how much the corpus was churning.

The correction adds a denominator instead of redefining the metric. The accepted decision record defines the signal as events, and quietly changing what that column means would put an approved document and the code out of step. Whoever read 53 alone now reads 53 across 18.

What it does not fix is stated in the migration itself. Our re-ingest and a genuine edit both change the content hash, so the two cannot be separated from stored data. Separating them means recording the cause at write time, which is a specification, not a migration.

The uncomfortable part is the sequence. The person who split that view per tenant and the person who missed the second layer are the same, one day apart. Cleaning half of an instrument makes the other half look clean.

The only path that spends money had no ledger

2026-08-26 · feat(nexus): book what the screenshot reader spends — the only path that costs money had no ledger

An ingestion run was reported as costing nothing. It had sent 39 images to a vision provider.

The report was not a lie. It was an absence. A spend module existed and was wired into two evaluation scripts, the paths that had once burned a day of paid API calls, and into nothing on the ingestion path. So the figure came from counting by hand, and the hand count missed the pages the loop had died on, which were exactly the pages that generated the calls.

The reads are now booked the way the answer path already books its own: token usage returned from each backend, priced once from the same rate table, accumulated per run, printed when the run ends. Three distinctions are kept apart on purpose, because collapsing any of them produces a number that reads as reassurance:

• a cache hit is not a call, and counting it inflates the spend
• a failure is a call, and not counting it breaks “how many did we send”, which is the exact figure that was wrong
• an unknown price is not a free one, so the run names the model that has no rate rather than inventing one

Wiring it surfaced a latent failure worth recording. Passing the new argument unconditionally makes any reader that does not accept it raise a type error, and the extraction path catches every exception per image, so that error would have been swallowed as a read failure. Images would have silently stopped being read while every counter reported a clean run. With no ledger attached the call is now byte-identical to before, and a test pins that property.

A number produced by hand is not a measurement. It has no failure mode that anyone can see.

---

What the log adds up to

Improvements came from removing defects, not adding technique. Seven retrieval techniques were tried and measured — multi-hop retrieval, model-driven query rewriting, frequency-based expansion, corpus merging, and others. All seven were rejected on measurement. Every real gain in this period came from removing something broken: a diversity cap that was truncating the correct passage, extraction markers polluting the search index, magnitude discarded at fusion.

The same shape kept recurring: the detector existed, the delivery did not. Coverage was computed and never shown. Document-to-code anchors were written for weeks with nothing reading them. Refusal reasons were recorded where only one view could see them. The current rule is that a check on the detector alone is not enough — the test has to run the surface a person actually looks at, and it has to be deliberately broken once to prove it goes red.

The instrument was wrong more often than the system. That is not a complaint about the instrument; it is the reason the instrument is treated as a first-class artifact here, with signed labels, pre-registered verdict rules, and a standing prohibition on editing a grader after seeing the score it produced. The sharpest case is the most recent: a grader that read a perfect score and could not see the change it had been written to evaluate, and an entropy signal that was counting our own re-ingests one day after being repaired for a different contamination. Both were found by checking a claim against the thing it describes, which is cheap and is now done on a schedule rather than when something feels wrong.

This page is a record, not a status board. For what is currently open, see OPEN.md, which counts unresolved items so that it is possible to tell whether they are going up or down.

The harness was measuring a path no user takes

2026-08-29 · fix(nexus): the harness was measuring a path no user takes

Two labels passed 5/5 and 4/5 in experiment scripts. Run through the evaluation harness, the same labels failed outright.

Same corpus, same questions, same scoring rules. One thing differed: how the evidence was assembled. The experiments called the function the answer paths use; the harness assembled its own. So the harness was missing both enrichments added that week. Every number it produced described a system nobody uses.

Pointing it at the shared function flipped both labels back to passing. That is the proof.

This repo has recorded the shape before — registration behind an execution guard, a copy outside the regression checks, wiring attached to one of three paths. What is new is that the list now includes the measuring instrument itself. When measurement takes a different road than production, it stays green while guarding nothing.

The instrument invented a defect, and I went to fix it

2026-08-29 · fix(nexus): a value has surface forms, and one missing form invented a defect

A label listed its expected value in one spelling. The system answered correctly in another. The grader scored three correct answers as misses.

That much is an ordinary label defect. The cost came next. I read the failure as a system defect — “it holds the evidence and picks the stale side” — designed a remedy for that defect, wrote a prompt clause, and measured it over five runs per arm. The pre-registered adoption rule rejected it. Of course it did: there was nothing to fix.

Reading one failing answer ended it in three minutes. Until then I had been reading only scores.

So a rule was added: read the failing answer before diagnosing the failure. A score says which value is missing; it does not say whether that value is really missing. Re-measured with the spellings the system actually writes, the enrichment reads 0/5 → 5/5, not the 3/5 first reported.

A change meant to make the harness honest buried a real contradiction

2026-08-29 · eval: labels say what they are measured against, and a "fix" that hid a conflict

The corpus is documents, yet the questions were phrased as “how does the product behave”. Under that mismatch, a correctly cited answer that is wrong about reality scores full marks. So every label was made to declare its authority, and the questions were rewritten to say “according to the document”.

Putting that change under measurement produced this:

	conflicting document in evidence	answer
natural phrasing	present	both values, disagreement flagged (2/2)
authority prefix	absent	one value, stated confidently (2/2)

The prefix narrowed retrieval and dropped the document holding the contradiction out of the candidate set entirely. The answer was faithful to what it received. Evidence arrival does not vary between runs, so this is not noise.

Reverted. Authority stays metadata. A label’s wording is not a free variable — change it and retrieval changes. A phrasing meant to make the harness honest had come close to hiding exactly what this tool is built to surface.

Built, then never read

Asking “how many characters can a nickname be” returned two documents with two different numbers. Opening the code settled less than expected: the server does not validate that length at all, and only the column caps it at 20. It was not a question of which document to believe.

What the owner asked for was not a verdict but a juxtaposition — an implementation may have broken the document, or the document may not have caught up, so tell me both. Checking whether that capability existed found three of four layers disconnected:

layer	state
code value resolver	present
code tree mount	present
registered concepts	zero rows — the seed file targeted a sample app
wired into answers	absent — reachable only from one dedicated CLI command

Fixing it surfaced three more. The resolver read the wrong shape: static final constants only, while in real Java the “how many characters” limits live in @Size(max = 100) and @Column(length = 20) — so it could reach none of the values we wanted to compare. I looked for the cost in the wrong place: the first lookup took 55.9s and I assumed the new parsing was to blame, added a pre-filter, and measured no improvement at all. Split out, it was listing files 50.70s, reading them 5.28s, parsing 0.18s — the cost of walking a bind mount. Seeding made dead claims look alive: it printed only 11 loaded, and a claim that fails to bind is still stored, just without a value.

What comes out now:

50 characters — but the code disagrees, so both are given.
⚠ The server's request validation caps this at 500, ten times the document's 50.
Which one is correct cannot be settled from this evidence.

Three values diverge: party name 30 vs 100, party introduction 50 vs 500, nickname 12 vs 20. The system picks neither. They may both be right at different layers — a product rule of 30, a server guard of 100 and a column of 255 is a real shape in this codebase, not a contradiction.

Correction (2026-08-31). Above, I wrote that the nickname had no server validation at all. That was false. The owner asked “no limit? not even in the test code?”, and checking properly found it in four places: the user-facing UpdateMyBioRequest @Size(max = 20), a domain value object Nickname that throws above 20, the column varchar(20), and a test asserting that 21 characters fail. The code is consistently 20.

What I had looked at were the admin and bot-creation paths only. And missing the user-facing one had a mechanical cause: the regex that finds field declarations required them to begin with private|protected|public, and that class uses Lombok, so its fields carry no modifier.

@Getter
public class UpdateMyBioRequest {
    @Size(max = 20, message = "닉네임은 20자를 초과할 수 없습니다")
    String nickname;          // no modifier

So I did not read an absence — I read a failure to read as an absence. A tool that cannot tell those two apart, wired to something that speaks with confidence, produces exactly this. It now requires a type token and no modifier, with the real-world shape pinned in a test. Seeded claims went from 4 of 7 bound to 6 of 7.

⛔ One limitation belongs next to it: the value object’s if (value.length() > 20) throw … is neither a constant nor an annotation, so it is still unreadable. That the 20 holds in three layers is something a person confirmed, not something the tool produced.

Sweep (2026-08-31). Having corrected it, I checked all eleven documented values against the code. Four agree, six diverge, one has no code site I could find — the “three” I had written was not a sweep. The history then supplies a clue the values alone cannot: four of the divergences (party name 100, party introduction 500, DJ minimum 1 minute, playlist name 100) arrived in one commit that added validation to eighteen DTOs at once. That reads as blanket defaults rather than per-field product decisions. ⛔ The one I could not place is recorded as “not found”, not “absent” — having just been wrong by confusing exactly those two.

The history answered it (2026-08-31). The owner rejected 100 and 500 as implausible and asked whether those numbers had ever changed. They had never changed. Both files date from 2024, and size validation was added exactly once, on 2026-03-09 — before that there was no validation at all. So 100 and 500 are not the residue of an older rule; they are the defaults picked on the day validation was first added. The values alone cannot tell those two apart.

Digging turned up one more thing. The introduction is validated at @Size(max = 500) while its column is varchar(255), with no entity override, ddl-auto: validate, and Flyway managing the schema. An introduction between 256 and 500 characters passes validation and then fails to store — not a documentation mismatch but a break.

⛔ And the notice limit I had recorded as no code site found exists. It reads @Size(max = PartyroomData.MAX_NOTICE_CONTENT_LENGTH), referencing a constant, which my search did not see. Writing “not found” instead of “absent” earned its keep.

Where the nickname’s 20 came from (2026-08-31). The owner asked for its history too. Followed to the end:

when	what
2023-05 to 09	column varchar(255) — no limit
2023-07-30, 08-31	two document revisions about nickname limits (the era of the documented 12 and 8/16)
2024-06-12	@Column(length = 20) first appears — in a commit titled “Add temporary users in local profile”
2026-02-15	request validation @Size(max = 20) added
2026-02-20	domain Nickname throws above 20
2026-04-19	the Flyway baseline pins varchar(20)

The 20 was never decided as a product rule. It is a column length that arrived while adding a temporary-user feature in 2024, and the two 2026 changes copied it. Neither 12 nor 8/16 has ever existed in the code.

The same 2026-02-15 commit raised the introduction column “to match frontend limit”. Within one commit, one field followed the product rule and another followed the column. Looking at the values alone, both are just numbers.

The sweep was half a sweep (2026-08-31). The owner asked whether all eighteen labels had been re-checked the same way. They had not. Only the eleven value labels were ever put against the code; the four permission labels and the three design labels were untouched — and I had still called it a sweep.

Checking the remaining seven: all four permission labels agree. Room termination is the host alone (validateHost), notices require CM or above (isBelowGrade(COMMUNITY_MANAGER)), and both role granting and bans require Mod or above (isBelowGrade(MODERATOR)). The three design labels were authored with code checks attached and still hold.

Six of the values diverge and none of the permissions do, which is not surprising on reflection: a wrong permission breaks something immediately, while a length limit quietly drifts toward whatever the column happens to be.

⛔ One more thing surfaced: “sign off the eighteen labels” is an open item I had been reporting in every summary and had never put in the open list. It lived only in conversation.

I never checked the answers (2026-08-31). It took the owner asking three times. I had only ever checked labels against the code — never whether the labels’ own expected answers match the corpus. Those answers are ones I wrote.

Checking them, of eighteen:

confirmed	12
wrong label	1
corpus disagrees with itself	1
resting on a single chunk	4

B-4 was wrong. Read row by row, the matrix separates two things:

kick (may return)     | O | O | O | X | X |   Mod allowed
ban (may not return)  | O | O | X | X | X |   Mod not allowed

I asked about the ban and recorded the kick’s answer. The correction then opens a defect: the code does not separate those two rows. One gate (isBelowGrade(MODERATOR)) covers both one-time and permanent expulsion. So a Mod can permanently ban in code while the document says only Admin and CM can.

⛔ In the previous round I wrote that all four permission labels agreed. That was a comparison run against a wrong label. When the label is wrong the comparison is wrong with it, and the direction of the error is always toward “nothing to see”.

Also: A-2 reads as both 50자 and 60자 within the same document, and four labels rest on a single chunk — not wrong, but with no control to disagree with them.

The safeguard is missing exactly where it is needed (2026-08-31). The owner asked why the capacity label still says 200. Leaving it is correct under the pre-registered rules — these labels measure whether the document is read accurately, not whether the product actually behaves that way, and divergence is explicitly a reporting matter rather than a score.

Pushing that question through the live path turned up something worse.

"how many people fit in a room?"   → answers 200 only        (owner: 50 is correct)
"minimum DJ time?"                 → answers both: 3 min in the document, 1 in the code

The difference is not whether a claim was seeded but what shape the value takes in the code. The DJ minimum is @Min(value = 1) and is readable; the capacity is if (activeCrewCount > 49) and is not. So on the one value already known to be wrong in the document, the safeguard that exists to catch exactly that is absent.

Three more sit in the same position — the 30-day auto-close (a method argument), the 3am batch (a method annotation), the nickname’s domain guard. They are invisible only because document and code currently agree.

⛔ And when the owner decided capacity was 50, that was the moment to say what happens to the three labels resting on rejected document values. I did not. Leaving them alone by the rules and saying nothing about them are not the same thing.

There was almost nothing to sign (2026-08-31). The owner asked: “you wrote the source document under each label yourself — so that must be right. And you said these only measure whether the document is read accurately.”

Counting them proved the point. Of the eighteen labels, fifteen carry source notes quoted verbatim from the corpus — a script can check those, and a person has no reason to. Only three places genuinely needed human judgement: one value where the corpus contradicts itself, one question of which document governs, and the permission matrix, where my note is my own reading of O/X cells rather than a quotation.

The matrix is the one place that is not a quotation, and three of the four defects came from it.

B-4	asked about the ban, read the kick’s row
B-3	the question is ambiguous about which side it asks about, and its evidence row covers granting one specific grade while the question asked about roles generally
A-11	the document calls it 소개; I asked about a 상태 메시지, a term the corpus does not use

The owner caught B-3’s ambiguity. Because the code applies the general rule, the answer came out the same either way — which is exactly why it could have been left alone. Then the next time code and document diverge, nobody knows what was asked.

⛔ All three surfaced before signature. No scores existed yet, so correcting labels is not tampering — that is why the rules withhold scores until signing. But what the rule protected was the order. What actually found the defects was a person asking the same question three times.

The first score after signing indicted the scorer (2026-08-31). Running all eighteen right after the owner signed, one policy label came back mentioned=fail, asserted=pass. The second-generation check is a subset of the first, so that combination cannot happen.

Two defects, both mine. The conflict branch never recomputed ok, so a conflict label could not pass the first check with any answer at all, and mentioned defaulted to False for any label without an expect list. The module had no tests.

Fixed and re-run:

	first run	after the fix
mention	14/15	15/15
assertion	12/15	13/15

The first run’s 14/15 is void. A number, once printed, gets quoted — this repository inherited its label problems that way, which is why scores are withheld until signature. That guard did not cover the minutes after signing.

Both remaining failures carried a defect of their own.

B-4 — the answer said Mod and cited a table I had never read. There are two permission matrices and their ban rows disagree: one says Admin and CM, the other adds Mod. The code allows Mod. I had “corrected” that label from Mod to CM on the strength of one table — the same mistake as the nickname, concluding from a single place. Twice now.

A-3, A-5 — they pass the first check and fail only the second, not because the answer is wrong but because it puts both the document’s value and the code’s value on the page and leads with the fact that they disagree. The second check requires the document value to sit in the lead or verdict position, a rule written when answers carried no code values. The answer got better and the score went down.

⛔ That rule does not get changed now. Changing a scoring rule after seeing the score is tampering. It gets changed by pre-registration, before the next round.

Someone else’s numbers were accumulating on a public list (2026-08-31). The owner put it plainly: “save my to-dos to memory — they aren’t work for this product.”

True, and there was a second reason. OPEN.md is khala’s open-items list, and it lives in a public repository. What I had been piling into it over several days were the team product’s screen names, field names, length limits and permission-table rows. khala found them, but not one of them was khala’s work.

Six rows moved to operator memory, outside the repository, leaving one line in their place saying they are not here and not to move them back.

Writing that line, the fingerprint check caught me. The memory file I named carried the organisation’s name, and fingerprint_scan.py stopped the commit: “partner organisation name — this becomes public.” That check exists because fingerprints crept back in a month after the first scrub. Today it fired on the commit that was cleaning up the boundary.

⛔ This repository’s defect list carries khala’s defects. For another organisation’s facts, each individual value is a fingerprint.

Two things I was about to build already existed (2026-08-31). The cutover spec was blocked on four preconditions. Two were the owner’s signature; the other two I had written down as things I needed to build — per-chunk clearance evaluation, and a new document-lifecycle reason plus an ADR to dispose of it.

Before building, I measured and looked. Neither was needed.

Clearance — comparing the two tenants’ distributions, both use only INTERNAL and RESTRICTED, with no value appearing in one and not the other. Opening the four RESTRICTED documents shows they are copies of each other. So this cutover changes nobody’s exposure. Per-chunk evaluation is needed when the vocabularies actually diverge — when a second organisation joins.

The removal mechanism — resource_status already carries soft_deleted, and hide_document (soft_delete with hold=true) and restore_document already exist. They satisfy all four requirements: invisible to search, rows survive, has an inverse, and records that a person decided. As a bonus, reconciliation refuses to revive anything with hold=true — one path by which the copy could quietly return was already closed.

⛔ I was about to build two new primitives and write an ADR, and I had not looked first. The critique pushed back that it was not the right place; looking, it was not merely the wrong place — the thing was already there.

This log already carries the same lesson in another form: most retrieval problems are already solved; search the repository’s own documents first. That time it was someone else’s paper. This time it was a function in my own repository.

The rule I wrote stopped me (2026-08-31). Five baseline runs before the cutover. The pre-registered rule: drop any label that wobbles between runs, and if fewer than twelve remain, hold the attachment.

18 labels  →  11 stable, 7 wobbling
11 < 12    →  hold

The rule was written before the scores existed, and it fired. The copy was not hidden; the gap window was never entered.

The wobble has a clear shape. Five of the seven wobble only on the second-generation check, passing the first every time. The answer sometimes leads with the document’s value and sometimes with the fact that document and code disagree, and the second check requires the value to sit in the lead or the verdict. That rule was written for answers that carried no code values. The answer improving is what shakes the instrument.

⛔ And this is where it would be easy to slip. Changing a rule because it blocked you is tampering. The grounds for changing it already exist — this drift was filed this morning, with the note that any amendment goes through pre-registration before the next round. So the order holds: register the amendment first, then re-run the baseline. Lowering twelve to eleven now would make pre-registration decorative.

The cutover ran end to end and was rolled back (2026-08-31). The rule fired twice.

step	result
baseline, five runs	14 labels stable (≥12, proceed)
hide the 122 copies	soft_deleted with hold=true, zero active chunks
five runs	exactly one of the fourteen changed — a design label dropped, which is the evidence the predicate took hold; the thirteen policy labels were untouched
attach the scope	verified live: Slack reads the source corpus with no copy
five runs	A-9, stable at five of five in the baseline, began wobbling on the second check → roll back

Rolling back was one command. Copy and configuration restored, verified live.

Three defects surfaced on the way

A model default swallowed the contract. AnswerRequest.tenant defaults to "default", so the field arrives populated whether or not the caller sent it. Read as a request, the “no tenant given, use the whole scope” branch can never fire. The lock was opened, the configuration written, the service restarted — and only the answer not changing revealed it. Every unit test constructs the request explicitly, so the field is always set in tests and always set in production, for opposite reasons.

The harness was asking a different corpus. After the hide, the design labels failed and the rule said roll back. The same question answered correctly through the live path: the harness passes one tenant string straight into search while the live path resolves a read scope. A comment four lines below records the same mistake three days earlier.

The instrument wobbled because the answers improved. The second check wants the value in the lead or the verdict; answers presenting document and code values together push the committing sentence into a section. The amendment went through pre-registration, and on implementation a control already in this repository narrowed it.

What is left

Why A-9 wobbled is not yet known. The design corpus entering the evidence may have changed how the answer is composed, or the label may always have been marginal. It does not get switched on again before that is understood.

⭐ The valuable part of this round was not the cutover but the three places it stopped. All three wore the same face: something that ran fine while measuring a path nobody travels.

The rollback was a false alarm (2026-08-31, evening). After rolling the cutover back by rule, the offending label was run eight more times in the rolled-back state — the same condition as the baseline.

first check:  8/8    the value is present every time
second check: 4/8    where it commits varies

That label was already unstable. Its five-of-five in the baseline was luck, and the cutover created no regression.

Working it out, this was inevitable. A label that passes half the time looks uniform across n runs with probability 2 × 0.5ⁿ.

n	looks uniform	expected misclassifications across 18 labels
3	25.00%	4.50
5	6.25%	1.12
10	0.20%	0.04

Five runs was a design that expects one misclassification, and produced exactly one. The critique had said to fix the run count and estimation method and state the smallest change they could detect; I raised three to five and never computed the power. Writing a number down in advance does not make it the right number.

The instability was not in the scorer but in the answer’s format:

passes:  **문서(설계) 기준: 20자**     the commitment sits in a section lead
fails:   | 문서 (화면 정책) | 20자 |   it sits only in a table row

Not counting a table row as an assertion is by design. Both are good answers; the format varies, so the label sits on the boundary. ⛔ The answer’s shape does not get bent to fit the instrument — that is backwards.

The amendment is registered: classify stability once, over ten runs, then compare five runs per point using only the labels that classification called stable. Stability is a property of the label and the system, not of the moment.

Retracting the previous entry’s root cause (2026-08-31, evening). It said the label had been unstable all along. That conclusion came from running that label alone. Across the ten-run classification, which runs the whole set, it is 10/10.

how it was run	second check
that label alone, ×8	4/8 (reproduced twice)
all fifteen, ×10	10/10

The difference reproduces. The cause is not known — the harness’s filter only subsets the query list, and the keyless bridge keeps no session between calls. The remaining candidate is that the vector leg is approximate, so its candidate set shifts with process state; unverified.

Splitting what survives from what does not:

• ⛔ “that label sits on the boundary” is retracted. It was only ever seen in subset runs.
• ✅ the power arithmetic (2 × 0.5ⁿ; at n=5, 1.12 expected misclassifications across 18 labels) stands — it has nothing to do with that label.
• ✅ classification and comparison both run the full set, so this measurement is internally consistent.

⚠ And one rule is added: a number obtained from a subset run does not get quoted for the full-run condition. Running a single label is fast and convenient when diagnosing, and the number it gives belongs to a different condition. Today I put such a number into the public record as a cause.

It took three blocks to suspect the judgment design (2026-09-01). The cutover rolled back three times at the same place, each time over a single label that passes the first check every run and wobbles only on the second. On the third, I did the arithmetic.

For a label with pass probability p: the chance it looks uniform across ten runs, times the chance it wobbles in the following five.

p	uniform over 10	wobbles in next 5	expected false alarms across 18 labels
0.50	0.2%	93.8%	0.03
0.80	10.7%	67.2%	1.30
0.90	34.9%	40.9%	2.57
0.95	59.9%	22.6%	2.44

“Uniform across n runs” is not “deterministic”. That classification screens out p≈0.5 and nothing else. A label at p=0.9 looks uniform one time in three, then wobbles in the next comparison with probability 0.41. The design expected two or three false alarms per comparison.

⛔ And the previous amendment enlarged the wrong axis. Raising five runs to ten, I wrote that expected misclassifications were now 0.04 — that figure assumes p=0.5. A label at p=0.9 keeps passing a set-equality test no matter how many runs are added. Having computed a number is not the same as having computed the right one.

The amendment is registered: judge on rates rather than set equality, and state the smallest detectable change — this harness catches 1.0 → ≤0.5 and nothing finer. The second check drops from deployment gate to improvement gauge: while the answer’s format is non-deterministic, gating deployment on it means never deploying. The regression net for absence stays with the first check, which has not wobbled once.

⛔ The answer’s shape does not get bent to fit the instrument. The instrument is made to tolerate format noise instead.

Re-measurement starts fresh on both sides. The data that motivated a rule does not get to pass that rule.

It landed on the fourth attempt (2026-09-01). After rebuilding the judgment on rates, everything was measured again from scratch.

baseline ×10   16 labels stable on the first check, 2 wobbling
hide 122 + attach              gap window ≈ 30 seconds
comparison ×10  all 16 identical, zero gauge flags  → pass

Slack now reads the source corpus directly, with no copy. The policy tenant is back to 466 chunks of its own material, and the 1,582 chunks of design documents come from where they actually live. The 122 copies are hidden rather than deleted, so the rollback is still two lines.

All three earlier blocks were instrument defects

attempt	what stopped it	the actual cause
1	one label wobbled	a request model default swallowed the scope contract, leaving the cutover silently inert
2	design labels dropped	the harness asked one corpus while the live path asks two
3	one label wobbled	the judgment design — uniform across n runs is not deterministic

The product was never the thing that was wrong. And each time the rule stopped me, I stopped — by the third, the rule firing three times at the same place was itself the signal.

One lesson worth keeping

Pre-registration protects the order of operations, not the arithmetic. I raised the run count from three to five to ten, writing a justification each time, and every justification assumed p=0.5. What actually blocked was a label near p=0.9, which keeps passing a set-equality test however many runs are added. I enlarged the wrong axis three times.

The verdict now prints its own limit with every judgment:

⚠ smallest detectable change: 1.0 → ≤0.5. Anything finer, this harness cannot see.

The limit is stated rather than left to be inferred from a precision that is not there.

Retracting “subset runs differ from full runs” as well (2026-09-02). Yesterday I recorded it as a reproducible fact and filed it as an open item. Digging in, it does not hold.

Retrieval and evidence assembly are deterministic. Running the same query with no warm-up and with eight preceding queries gives the same top-20 chunks in the same order, and the assembled prompt is byte-identical. There is nowhere for a difference to enter.

Counted by state:

	full runs	subset runs
old state	17/20	8/16
new state	10/10	10/10

The old state’s p=0.034 is a post-hoc test on a case picked because it looked odd, so it is not evidence. And two full runs under identical conditions gave 10/10 and 7/10 — the noise is larger than the thing being compared.

⇒ It was a finding built by eyeballing two batch fractions. The same class of error that stopped me three times this week. Those times a rule caught it; this time I did not apply the rule and used my eyes instead.

One rule added: test before claiming two conditions differ. And a post-hoc p-value on a case selected for looking odd is not evidence.

No signal row landed for 34 hours while 1,800 tests stayed green (2026-09-02). Attaching the read scope, I assigned the resolved list to req.tenant. That field is typed str and search_log.tenant is TEXT, so the insert failed — and record_search is built never to raise, so it failed in silence.

That design is right: writing a log must not kill a user’s request. The problem is that the failure went nowhere. Not even to the log.

How it surfaced is worth recording too. Working on an item the cutover had created — recording a cross-tenant query under a single tenant misattributes demand — I opened the schema and noticed the table was empty to begin with. Going to fix a misattribution, I found the writing had stopped.

The root cause is fixed rather than patched: nothing puts a list into a field typed for a string. Attribution keeps a single value and the scope gets its own column. Without it, Slack reading the design corpus leaves only the policy corpus on the row, and the pull signal reads as “nobody looks at the design corpus” — the cutover would have begun by poisoning the measurement meant to justify it.

Four tests now run against a real Postgres, one of them the test that would have caught this: the row actually lands, the count goes up by one, and the scope is on it. None of the previous 1,800 asked whether anything arrives.

⛔ And the same shape sits elsewhere — answer-text retention, feedback, the audit trail, the access counter. All best-effort, all able to die quietly. Filed.

Correcting the previous entry’s diagnosis (same day). It said the failure went nowhere, not even to the log. Wrong — a search.signal.persist_failed warning was written at the time. I missed it for a dull reason: the event was 34 hours old and I looked at the last two hours.

That changes the treatment. If the problem were silence, the answer would be to make failures louder. The actual problem is a warning that was written and never read, and the answer to that is a health signal on the last write time. Making the warning louder does nothing when nobody is reading.

Separately, fixing the schema in one place was not enough, and CI caught it: the runtime ensure_search_log() carries the ALTER, but CI builds its Postgres from init.sql and has tests that never call that function. The schema has to match in three places — the init file, a migration, and the runtime DDL.

⛔ And I merged with a check still red. Two were still running when my wait loop gave up, and one of them then failed. Second time this round.

I misread the health signal within minutes of building it (2026-09-02). After the search log went unnoticed for 34 hours, I added a command that shows the last write time per sink.

The first version printed only the age. Looking at it, I read search_answer_text sitting at two rows and 62 hours as a second defect. It was not — that table is only written when Slack answers something, and no Slack question had been asked. It was fine.

⇒ “nothing arriving” cannot distinguish “dead” from “quiet”. A signal that prints only an age leads its reader into exactly that misreading. So each sink now states what drives it, and a test enforces that.

search_log  1072 rows · last (1.1h ago)
    search and answer signals — driven by: every search and answer — little reason to be quiet

search_answer_text  2 rows · last (62.1h ago)
    answer text retention — driven by: only when Slack answers — silence is normal with no questions

No thresholds. A threshold becomes one more signal nobody reads — which is precisely what caused the defect this was built for.

Also: I got the dataclass field order wrong twice in the same file, putting a defaulted field ahead of undefaulted ones. The second time it went into a comment.

Two enrichments were off for two days, and every test was green (2026-09-02). Reading live logs after the cutover, I found section_fill_failed. The error text says all of it:

invalid input for query argument $2: ('default', 'design_docs') (expected str, got tuple)

Section fill bound the read scope straight into c.tenant = $n, and since 2026-08-31 the read scope is a tuple. asyncpg will not take a tuple as a TEXT argument. And the caller swallows that exception — an enrichment failure must not kill the search. The design is right. The outcome was not: section fill and pair expansion were silently off on every HTTP answer request.

⛔ A one-element scope is still a tuple. So this was never limited to the deployment that had cut over — single-tenant deployments, which never configure a scope list, went dark the same day. The blast radius of a defect nobody can see is usually wider than my first guess.

The value of this entry is why the tests were green. Section fill has eight tests against a real Postgres, and one of them asserts that the filled section reaches the LLM prompt itself. All eight pass the tenant as a string. Pair expansion’s five tests never touch the database — they check the pure function that links a design to its plan by filename. The whole wiring can die without moving one of the thirteen.

And a lie was sitting right there. The module that collects the scope predicate into one function opens by saying “a new read path calls this, and if it doesn’t, a test catches it”. No such test existed. The tests in that file exercise the function itself; none of them count its callers.

Three fixes:

1. Both queries use the scope predicate. One element yields = $n, several yield = ANY($n) — different SQL, so both shapes have to be run.
2. A test that runs the answer path once for each of three scope shapes (string, one-element tuple, two). It asserts not a return value but that no enrichment died quietly — it counts shapes, not names, so enrichments added later fall into the same net. Broken on purpose against the old code: 6 of 9 go red, and the three that stay green are exactly the string shape.
3. The false sentence is gone. That paragraph now names what actually guards the property.

⛔ A closed item had to be reopened. On 09-01 I closed one saying “pair expansion now sees specs/ and plans/ in the design corpus”. It had never run on the human surface — the live log holds pair_expansion_failed and no record of a success. The evidence for closing it was one CLI path, and that is the only path that passes the tenant as a string.

The scope was widened, and nothing recorded whether that paid off (2026-09-02). After the cutover, Slack reads the design corpus directly. But no column said what a given answer actually leaned on:

Column	What it says	What it cannot say
search_log.tenant	who it is attributed to (single value)	where the evidence came from
search_log.read_scope	what it was allowed to read	what it actually read

⇒ A widened scope whose evidence still comes from one side looks identical, in the record, to one that draws evenly. Those two columns cannot tell you whether the cutover delivered anything.

So evidence_tenants was added (migration 038): per question, the count of evidence pieces per tenant.

⭐ It counts the packet, not the hits. That is the value of this entry. The first live row reads n_snippets=10 and default:18 — filled sections, mate documents, and the correction pass enter the evidence without going through ranking. Counting hits would have answered “what did this lean on” while missing 44% of the evidence, and those three are exactly what the cutover pulls from the design corpus.

Counts, not ratios. A ratio erases its denominator — 1.00 looks the same whether it stands on one piece or twenty. Counts give you the ratio whenever you want it; the reverse does not hold.

⛔ And the first version printed 100.0% off a single row. A one-question ratio, and it looked quotable. That is twice this week for the same class of mistake (misreading 62 hours in the persistence signal; comparing two batch fractions by eye). So below ten questions it prints no percentages at all — counts only. The repo had already settled that discipline once.

No threshold. §5.3 says the first round is observation and the threshold comes from looking at the distribution, and that is what it does.

⚠ That SPEC also says, in §5.4, that the column and migration number are to be recorded in the SPEC itself. They cannot be. An approved body is frozen at the bytes it was signed on; touching it breaks the stamp — a rule this repo learned by turning master red fifteen times in a row over a single footnote. The numbers live in the migration file and the open-items list, not in the SPEC. The template is making a promise it cannot keep.

Went to fix it, and there was nothing to fix — the second time (2026-09-02). An open item read: “labels A-3 and A-5 pass the first grader and fail the second, not because the answer is wrong but because it puts the document value and the code value side by side and writes ‘the document and the code disagree’ in the conclusion slot.” It even carried a candidate fix — “an answer that discloses the conflict should pass even when the value sits inside a table.”

Starting the work, I read the artifacts before the rules. All three parts were different.

① The symptom is already gone. Across the final two arms (ten runs each), A-3 and A-5 score 10/10 and 10/10 on the second grader. The two that waver are different ones (A-1 at 9/10, A-9 at 7/10). Re-running the grader over the stored answers shows where they pass: the decisive sentence sits in a section head. The 2026-08-31 section-head amendment fixed exactly that shape — and the open item, written the same day, was never removed after the amendment resolved it.

② The candidate fix would reverse a pre-registration. “Pass when the value sits in a table” is precisely what this harness wrote down as forbidden: accept tables and you also pass the answer that prints the value in a table and then backs away in the conclusion. Separating that pair is the second grader’s whole reason to exist, and the control case is pinned in a test. Applying the fix would have turned that control red.

③ The real remainder cannot be read. Nobody knows why A-1 and A-9 flipped, because those twenty runs kept only the summary and threw the answer text away. The answer backend has no temperature and no seed, so the same answers cannot be regenerated. This repo had already burned three hours on that and written down that “the grader’s report keeps an answer-text sidecar” — and that discipline sat behind an option the scoring runs never turned on.

So what changed is instrumentation only; not one character of the scoring rules: a run that writes a results file now writes the answer sidecar by default (turning it off must be explicit), and the verdict script names the run ids where a label flipped — a path from the number back to the text.

⇒ Changing the rule before reading the answers is exactly what that open item proposed. No rule changed, so there is nothing to pre-register.

⚠ I got one thing wrong while writing this correction. My first pass aggregated all 65 verdict files and read it as “the second grader wavers on all fifteen labels.” Those files span ten arms and include runs scored by the pre-amendment grader. Split by arm, the count is two. A shared filename prefix is not a shared condition.

The file that counts open items was mirroring its own count by hand (2026-09-02). Over one day I closed four items and opened three, bumping the header by hand each time — 23 → 24 → 26. That number went straight into a report.

The user asked: “26? What are they?”

I counted. It was 21. The human-side number was 14, not 15. The cause is plain: nothing was ever subtracted on close. Only added on open.

That is precisely why the file exists. Its opening paragraph says the open items were scattered across SPECs, ADRs, memory and conversation, so nobody knew whether the number was going up or down — gather them, count them, and state the count in every report. And the count itself was hand-mirrored. This repo has already written down, twice, that a hand-mirrored list is a source of rot.

⇒ scripts/check_open_counts.py now runs in CI. Struck rows are not counted (that is the file’s own rule), and it checks both places the number appears — the summary table and the section heading — because fixing only one is exactly the shape of this mistake. It fails in both directions: too high and too low.

Verified by breaking it on purpose. And the first version of the checker died with a UnicodeEncodeError at the very moment it tried to report a failure — the console codepage was cp949. This repo’s hook already carries the same helper for the same reason. A checker that dies says nothing at all.

⚠ What surfaced this was not CI. It was one question from a person. When a number looks plausible, nobody counts it.

Asked whether the documents were stale, and found the question could not be asked (2026-09-02). Digging into why answer quality looks the way it does, the most plausible hypothesis was that the source policy documents had gone stale. So I measured.

document age: all 126 in "under 3 months"

That number says nothing. documents.updated_at is our ingest time. I nearly reported “the documents are not stale” when all the number said was we ingested them in August.

The staleness module had already written the limitation into its own docstring:

“updated_at is the ingest time, so a document re-synced unchanged looks fresh. A more accurate signal (origin_last_edited) is follow-up work.”

⇒ Every re-ingest makes every document new again. It is a signal structurally incapable of ever saying “stale”.

⭐ And the value was already arriving. The connector reads last_edited_time and carries it in frontmatter as origin_last_edited. There was simply nowhere for it to land — which is why that name appears in exactly two places in the codebase: where it is set, and the comment calling it follow-up. This repo’s recurring shape: the signal gets produced and nothing reads it.

So the column now exists (migration 039). The judgment did not change — altering the warning changes what users see, and that starts with deciding whether a pile of old documents suddenly carrying warnings is acceptable. What this does is make the question askable.

The read side shipped with it, printing both ages side by side, because printing one walks the reader into the misreading I just made:

[default] 126 active documents — these two ages are different things
                    origin edit      our ingest
  under 3 months              0             126
  unknown                   126               0

⚠ unknown means we do not know, not “new”. Without that distinction the new column would repeat the very lie it was added to remove.

Added the column, ran the ingest, and it filled nothing (2026-09-02, same day). The previous entry added storage for the document’s own edit time. After an ingest, all 126 rows were still unknown.

The value was dropped in build_csf — the function that builds the dict handed from the connector to the next stage. The field simply was not in it.

⭐ Two comments in that same function already record the two previous times this happened. One for the title (a Notion page named Index entered the corpus under a different name), one for the image count (overwritten with 0 on every re-ingest). This is the third value lost at that seam.

⛔ And my test did not catch it, because it called _save_document directly. The real path is connector → build_csf → temporary markdown → collector → pipeline, and the break was in the first link. This repo had already written that failure down — “I asserted on the producer’s dict.” There is now a test that runs the chain, verified by breaking it on purpose.

There was a second wall. A document whose body is unchanged is skipped by content-hash dedup, so a new metadata column fills only for documents that happen to change — a biased sample. Backfilling meant enabling the destructive --reconcile (which can delete documents). Opening a delete path for one metadata column is a bad trade, so --force alone now means “save the document again even if the body is identical.” It deletes nothing.

So are the documents stale — yes

[default] 126 active documents
                    origin edit      our ingest
  under 3 months             14             126
  6–12 months                 8               0
  over a year                90               0

The signal we had was saying the opposite. Restricted to the Notion corpus: of the 19 documents that carry actual prose, 9 are over a year old (newest 2025-07-20), 8 are 3–12 months (newest 2025-11-02, and they hold most of the text), and 2 are under three months. The range runs 2023-12 to 2026-08.

⇒ The code moved; the policy prose mostly has not been touched in over ten months. The document-versus-code divergences observed earlier are exactly what that produces. Fixing the documents is not this repository’s work — reporting the divergence precisely is.

I broke the live corpus (2026-09-02). Backfilling the metadata column from the previous entry, I ran ingest-notion --force. The completion output was green. The corpus came out like this:

chunks          466 → 385
machine_read     81 → 0

Every piece of text read out of a screenshot was gone. In this corpus most of the policy detail lives inside images, so live answers actually got worse — a question that had answered with a value now abstained with “cannot determine.”

One gate caused it. NEXUS_VISION defaults to off (sending document images to a provider is a decision the deployment owns), and when it is off, ingestion replaces each image slot with a blank image. I ran the re-ingest without checking that gate.

⛔ And what made that re-ingest possible was a change I had landed the same day. Backfilling otherwise required the destructive --reconcile, which I did not want, so I opened --force on its own — I opened a door to avoid one destructive path and walked into another.

It was recoverable. The extracted text was still in vision_extractions and the cache keys on the image hash, so a re-ingest with NEXUS_VISION=on restored exactly 466/81, with zero provider calls and zero spend. But that is a recovery only available to someone who knows all of that.

What was missing: a guard

ADR-0010 §4 called marker-stripping — extracted text laundered into authored text — “worse than not extracting at all.” This is worse still: not laundering but deletion. And that path had no warning, no confirmation, and no test.

There is no case where silently deleting it is correct. So it now refuses: vision off + something to lose + no declaration exits with code 2. If deletion is the intent, it must be declared with NEXUS_VISION_ALLOW_DROP=1. That is the shape of a discipline this repo already has — a test database must declare itself disposable.

Verified against the live deployment that the same command now stops, with the predicate kept in the same function the ingest path uses; a copy would eventually let the guard pass while ingestion deletes.

⚠ And while recording this in the open-items list I got the count wrong again, adding a closed item to reach 23. The check I built a few hours earlier caught it on the spot.

Counted the shapes behind “the reader misses about half”, and the bottleneck was elsewhere (2026-09-02). Before widening the code-value reader, I counted where values actually sit (899 non-test Java sources):

shape	count	read?
static final numeric constant	66	✅
named annotation argument	59	✅
positional argument @Min(1)	29	❌ → opened here
guard clause if (… > N)	18	❌ (left closed)
@Scheduled(cron=)	7	❌
argument referencing a constant	1	❌

⭐ 125 readable slots, and 10 claims. The open item said “the reader misses about half of what is out there” — counted, the unseeded side is far larger than the unreadable side. claims.yaml is written by hand, and it had ten lines. All ten are unverified; the verification loop has never run.

Positional arguments — opened, but only where Java puts them

The 1 in @Min(1) is the value element: Java lets a single-element annotation omit value=, and that rule fixes which slot it fills. So it fills only a claim asking for value. If a claim asking for @Size.max could be satisfied by @Min(1), another field’s number would quietly sit in the answer’s “the code says…” slot — worse than a wrong value, because it is wrong with confidence. A test pins that.

⭐ And opening it revealed a second layer. Of the 29 positional arguments, zero are attached to fields — they are all method and record parameters. The reader only looked at field declarations, so reading positional arguments alone would still have read none of them. Declaration terminators widened from ;= to ;=,).

Widening means a name can match in several places. That risk was already handled: when the values disagree, the resolver refuses rather than picking. The dangerous part is not the widening but the choosing, and that was left alone.

Verified live against the real tree: @Max.value → 60, @Min.value → 1 — the code-side bounds for the DJ time limit, whose document states the same value as “3 minutes or more.”

⛔ The 18 guard clauses stay closed. Reading if (activeCrewCount > 49) as 50 requires getting the comparison, the boundary, and the variable’s meaning all right. That is not value reading but code comprehension, and when it is wrong it is wrong with confidence. The capacity value the owner overruled happens to live there — but wanting it is not evidence.

An external evaluation found the largest defect: web chat was on a different path (2026-09-02). An outside evaluator (a different model) assessed Nexus and filed a result document. One of its two high-severity findings is this.

The packet_for_answer docstring in reconcile.py had already written the rule:

“The answer’s evidence packet is built by this one function. There are three answer paths — if each calls assemble_packet itself, wiring an enrichment into only one of them becomes possible, and that combination is silently wrong in production while the tests stay green. On 2026-08-29 I did exactly that, which is why they are collected here.”

No check enforced that sentence. One of the four surfaces, /search/answer/stream, called assemble_packet directly, so the correction pass, pair expansion and code values were missing from that path alone — and that is the path web chat uses (web/js/api.js).

The evaluator’s measurement, over 8 policy queries:

qid   /search/answer   /stream    diff
p04              23        20      −3
p05              21        18      −3
p07              19        13      −6   (32%)
p08              18        15      −3

⇒ For the same question, CLI, A2A and non-streaming HTTP received the enrichments; only web users did not.

The fix

Routed through packet_for_answer. As a side effect that path was also the only one still on effective_scope, so it could not read the corpora the cutover opened and left no out-of-scope audit event; both closed together.

Re-measured: all 8 queries now differ by zero, matching the non-streaming column exactly.

Why no test caught it

There was no check that counts the surfaces. Building one taught two things:

• It has to walk nested functions. The streaming handler assembles its packet inside event_stream(), so a check that reads only the outer function stays green on the broken build. Broken on purpose, it goes red on exactly that surface and no other.
• ⛔ An existing test was stubbing the bypass. test_answer_payload_contract.py replaced assemble_packet with a no-op — but the endpoint it exercises, /search/answer, never called that function. It was a dead line. Removed.

⚠ Fixing it surfaced one more: /search, the retrieval-only endpoint, is still on effective_scope. All four answer paths now receive the read scope; search alone does not. That list is read directly by people, so it needs its own verification and went onto the open-items list.

The title of a document you may not read could reach the prompt (2026-09-02). The external evaluation’s second finding (F3). nexus/CLAUDE.md states the rule: “Every SELECT carries the policy filters. No exceptions.” The four are tenant, clearance, quarantine, status.

Two places held an exception — the join to the superseding document:

LEFT JOIN documents s ON s.rid = d.superseded_by AND s.tenant = d.tenant

No clearance, no quarantine, no status on s. And that title travels through describe() into the prompt — the function itself is documented as “the one line that goes into the prompt.” The document list API returns the same field from the same join.

Live leakage was zero. But every ingredient is present:

documents with a supersede relation   121
RESTRICTED documents                   17
quarantined documents                   4
leaking combinations                    0   ← simply not overlapping yet

Latent, not safe. One ingestion away.

The split the fix required: the name versus the fact

With the filters on, the replacement’s title comes back empty. If the superseded debt disappeared along with it, a reader would consume stale evidence without knowing it is stale.

⇒ Withhold the name, keep the fact. Being retired is a fact about the document the reader is already holding; what clearance protects is the replacement’s content and identity. So superseded is now separate from the title.

And clearance is not an optional argument. Reading an empty clearance as “no filter” would turn every clearance-less caller into a bypass, so an empty value skips the lookup instead.

Why the existing tests missed it

• test_documents_api.py asserts the title does appear. Nothing asserted that it does not when out of clearance.
• test_doc_debt.py never touches the database — it builds the objects by hand, and so walks straight past this SQL.

The new test plants rows: one replacement over-clearance, one quarantined, one non-active. It asserts the title is absent from the prompt line and that the fact survives. Two controls come with it — a readable replacement is still named, and raising the clearance reveals it, which is what proves the other assertions are about clearance at all.

Disabling only the clearance clause turns exactly one of the nine red.

Topic drift — the evaluator’s hypothesis was right and its reason was wrong (2026-09-02). The external evaluation’s other high finding (F8): asked a question whose subject is absent from the corpus but whose vocabulary overlaps, the system answers confidently. Asked who signs off an approval gate, it answered — correctly — about Kubernetes certificate approval, three runs out of three.

⛔ The two axes this repo judges in code are structurally blind to it. The citations exist, the numbers exist, and it is not a guess — it answered a different question using real evidence. The only thing that caught it was a five-item abstention control set.

The evaluator left an unverified hypothesis: “the overlap raises BM25, so the and rule never fires.” Measured:

query	distance	BM25	weak
overlapping · out of corpus	0.4694	0.295	False
non-overlapping · out of corpus	0.5569	0.000	True
in corpus	0.3211	1.038	False

BM25 was already below its threshold (0.295 < 1.5). What slipped past is the vector side: distance 0.4694 cleared the 0.48 threshold by 0.011.

⭐ That is the exact spot the threshold’s author flagged in advance. From confidence.py:

⚠ The threshold is still a hypothesis. All 17 questions are mine, and the gap between the middle band’s maximum distance (0.470) and the threshold (0.48) is only 0.010 … if one real usage question fires in the middle band, move it near 0.51.

And yet it does not move — for two reasons

① The observed case is not real usage. On four retained real questions the threshold behaves correctly (out: 0.614, 0.589 → fires; in: 0.255, 0.310 → does not). The author pre-registered “before measuring, state whether the sample is authored or real usage”, and the one failure is an authored label on a different corpus.

② The trigger has already fired and cannot be read. search_log holds 8 rows in the middle band (max 0.4765). But whether those are real usage or my own probes cannot be determined — the query-retention key is deliberately non-joinable, by the same design that keeps identity from sitting beside text. The author’s trigger asks for a real usage question, and that judgment is currently unavailable.

⛔ And moving the threshold would not close this anyway. weak does not block an answer; it changes the narration contract only — the same file says so. The threshold is a partial response, and there is still no axis that judges topic drift itself.

Split the 0/3 multihop result deterministically — two are assembly, one is generation (2026-09-02). The external evaluation scored multihop 0/3 on the real corpus, and its §11 had already narrowed it: the gold chunks do reach the top ten. One question remained — does the required fact reach the prompt at all?

Checked without calling the model (zero spend, zero noise):

qid	required fact	in the prompt
m01	최대 10개	no
m02	최대 10개	no
m03	both	yes

⇒ Two assembly failures, one generation failure. And that verdict independently agrees with §9’s sufficiency=insufficient on the same two — a judge model and a string comparison picked out the same pair.

The mechanism

The missing fact sits in one section of a document that is among the hits. Section fill exists for exactly this, and it did not run.

Fill triggers on “did this document reach diversity_per_doc_cap”. A document at the cap means the diversity rule cut it, which means retrieval concentrated there. That is a sound trigger — for single-document questions only.

m01  five documents share the top ten → zero saturated
m02  the needed document sits at 4 of 5 — one short

⭐ Fill triggers on concentration, and multihop is dispersion by definition. Spread across five documents, nothing saturates; and a near miss (m02) fails by one. The second fill path — completing the section a hit landed in — cannot catch it either, because the fact lives in a different section.

This is why single-value questions score well while multihop sits at 0/3: single-value questions concentrate on one document, multihop does not.

Not fixed

The candidate settings are visible — diversity_per_doc_cap, FILL_TOP_HITS, a multihop-specific trigger. None was chosen. This repo has gone 0 for 7 adding techniques; everything that improved came from removing a defect. And widening has a price: completing the sections of all ten hits raised evidence volume by +102% when measured on 2026-08-26.

The mechanism now has a name. What comes next is a pre-registered measurement.

It said “the only place” and lived in two (2026-09-02). External evaluation F1. The first line of auth/clearance.py is “the single source of truth”, and the same file states that because the order lives in only that place on the Python side, a test can assert parity with the SQL enum. The seam map in nexus/CLAUDE.md says the same: one canonical table; a copy creates two answers.

models/resource.py held a second ordering table — and an access-control function built on it, whose docstring instructed “apply this function to every search and lookup.” The exact opposite of the seam map.

Production callers: zero. The real control is the four SQL clauses. So it was latent rather than active — but the shape of the risk is clear: the canonical table has a SQL-enum parity test and the copy does not. Add one level to the enum and the canonical goes red while the copy quietly falls behind.

base_filter_sql() went with it: psycopg-style %(tenant)s in an asyncpg codebase, already annotated as unused in claims/repository.py. An unusable function was claiming to be the common clause on every SELECT, no exceptions.

Deleted, not fixed

That is this repo’s discipline — do not repair a copy, remove it. It has written the clearance list twice before and watched the two diverge immediately.

Seven access-control tests came out of test_crm.py with it. Their subject was the copy — a pure function nobody called — while the real control is covered by tests that hit a real database.

And the claim became a check

Deleting does not stop the next person from writing it again. An AST walk now inspects dictionary literals by key set and asserts the ordering table exists in exactly one place.

⛔ The reason it is AST and not a regex is written down beside it. This repo has been bitten by source-string checks — the presence of a string does not mean the code ran. But the property here is the text itself: a second copy can diverge whether or not it runs. The AST form ignores words inside comments and strings and survives different quoting or line breaks.

Verified by restoring a copy on purpose and watching it go red.

Swept the stale claims, and gave the README a counter too (2026-09-02). The external evaluation made declaration versus code its main product and named four drifts. All are now reflected.

claim	reality	done
seam map: “four surfaces converge on assemble_packet”	only streaming bypassed it	the convergence point is now packet_for_answer, pointing at the check that counts surfaces
hybrid.py: “BM25 + Vector + Graph 3-way fusion”	graph is not in the fusion	nexus/CLAUDE.md had already been corrected; this file had not
ROADMAP: “RAG + GraphRAG”	document entity extraction was retired	separated from the OTel dual graph, which is kept
README: roughly 1,900 test functions, 49 SPECs	2,585 · 52	corrected, and a counter now runs in CI

Only the README had no guard

The evaluator’s point was exact. This repo had already mis-stated the open-items count by incrementing it by hand — the same morning, reporting 26 where the real number was 21 — and put check_open_counts.py into CI for it. The README carried the same kind of number with none of that protection.

check_readme_counts.py now verifies four figures (test functions, CI jobs, ADRs, SPECs) on every push. The counting rule lives only in the checker; the README carries values. Counting in both places would recreate the very illness.

⚠ And the hand-counted “record of thirteen defects” lost its number entirely. What counts as a log entry is ambiguous — a heading, or a bold paragraph? — and adding a counter would pin that ambiguity into code. What cannot be counted is not counted.

⭐ And the counter caught me on its own pull request. Locally it was green at 2,585; CI went red at 2,592 — git grep counts tracked files only, and the test file I had just written was not yet committed. Adding --untracked makes local and CI count the same set. The counter’s first defect was found by the counter.

---

September — the diagnoses that were wrong

August’s entries are mostly the instrument was wrong. September’s are a different shape: the instrument was right every time, and the reading of it was wrong. That distinction is the month’s whole content.

A request could not say which stage lost the answer

2026-09-05 · spans: turn capture on, and register a guard that the line stays on

Retrieval ran two legs, fused them, cut by a diversity cap, filled sections, assembled a packet and narrated — and recorded one flat row per request. So when a production query answered badly, nothing in the record separated the candidate pool never held it from fusion lost it from the diversity cap cut it from it reached the prompt and the model dropped it. The only way to tell was to reproduce the query against labelled data.

Each stage now writes a span with its inputs, outputs and ranked candidates. The unit shipped with capture switched off so the schema, the constraints and the destructive purge path could be exercised in CI while no row existed; turning it on was a separate, dated decision. The switch is one line of configuration and the code default is off, so a lost line would leave every unit test green — the flip therefore came with its own check, broken both ways before being restored.

One signal was carrying three different failures

2026-09-05 · probe: split "the model did not extract it" from "retrieval never delivered it"

The answer grader said a required fact is missing. That single signal bundled three unrelated causes: retrieval never delivered the fact, the model had it and extracted none of it, or the model had it and extracted only part. Reading such a failure told you nothing about whether to go fix retrieval or fix generation.

The material for the split was already there — the exact string the model was shown. Evidence presence is now judged with the same normaliser as answer presence, because two graders in this repository once disagreed on a single label after their normalisers drifted apart. The combining rule takes two lists of booleans and nothing else, so a third normaliser cannot travel inside it.

The first real result arrived the same day: a policy label that had both required facts in its evidence and delivered one. Before the split, that row read identically to a retrieval miss.

An instrument that only ever ran in the harness

2026-09-05 · answer: record the shape of every answer, on every surface

A deterministic, model-free check for whether an answer obeyed a formatting request had existed for weeks — in the evaluation harness only. This is the log’s most repeated shape, and it appears twice more before the month ends: the detector exists, the delivery does not.

Plugging it in exposed a limit worth stating rather than papering over. Compliance needs (requested format, answer), and live there is nothing that knows the request. Inventing a detector for it would add false positives as a new failure mode. So the live path records the answer’s shape — sentence count, table, list items, length — and judges nothing. Canned strings are not measured; the abstention and generation-failure notices write the same keys as null, because a constant’s sentence count is not a measurement. The keys are never dropped, which is what lets the next reader tell not measured from lost.

I named the bottleneck wrong three times in one day

2026-09-05 · probe: attribute the must_contain labels too, and retract what I said about them

The audit that organised this work predicted the misalignments would converge on one missing instrument. They did — and then the convergence point moved, three times, and each move was recorded as fact before it was checked.

recorded as the bottleneck	what refuted it
no stage spans	correct — adding them opened the next item
no labels requiring more than one fact	only one label set had been looked at. Counting all of them: 41 such labels
the corpus does not hold the answer	the answer was in another tenant
the harness was aimed at the wrong corpus	fixing the tenant passed three of the four labels immediately

Two of those are the same mistake. The verdict upstream says the required fact was not in the evidence, and it was true every time. It does not distinguish a corpus gap from a retrieval miss from a misaimed run — that takes other evidence, and it was read without any being gathered.

Standing rule since: a verdict names a state, not a cause. Before writing a cause down, name what else could produce the same verdict, and go and look.

Two tests looked right and protected nothing

2026-09-05

Twice in one day a test was written, reviewed, passed — and then passed again with the defect it existed to catch deliberately introduced.

The first asserted that a refusal-stripping rule is not applied to evidence, but chose an example where both required facts sat outside the refusal segment, so stripping changed no verdict. The second asserted a distinction between unreachable by search and unreachable by search but reachable by vectors — a distinction that had halved a measured figure — while the logic making it lived inside a database function no pure test could reach.

Neither would have been found by reading. Both were found by breaking the code on purpose and watching whether the suite went red, which this repository already required and which is cheap enough that there is no excuse for skipping it.

A pre-registration stopped a report of noise as an improvement

2026-09-05 · measure: answer-facts on the scope production reads, three runs

A signed label set had been measured against one tenant while live readers see two, so every recorded number came from a corpus nobody queries. Widening it to match required changing what a signed set measures, so the expectation was written down first, in its own commit: widening adds competing evidence, so the number can hold or fall; a fall is not a regression but the first honest reading; if it rises, be suspicious.

The first judgement held at 15/15. The second judgement went from 13/15 on the narrow corpus to 15/15 on the wide one — a clean-looking gain. Three runs on the same corpus then produced 15, 14, 15.

The difference was inside the run-to-run spread. Without that sentence written before the run, it would have been reported as a win for widening the corpus. The output of the round was therefore a property, not a number: the second judgement varies by at least 1 in 15 under identical conditions, and the first does not vary at all — so one is quotable from a single run and the other is not.

A page cannot drift on code nobody told it about

2026-09-06 · guard: anchor the code behind four claims the public page does not make yet

Checking the public pages against the code they describe found no false claim — the page never states pool sizes, so a change to one falsified nothing, and a fix to a leaking document title made its security claim more true rather than less.

What it found was omission. Four capabilities shipped and appear on neither the English nor the Korean page: fetching a correcting document so the corrected one stops winning, fetching a design and its implementation plan together, putting the code’s value beside the document’s, and reading more than one corpus. Three of them are configuration switches whose code default is off, each with its measured effect written beside it.

⛔ Correction (2026-09-07). The sentence above said three of them were the exact switches a deployment check pins as on. Two are. The deployment check pins three switches, but one of those is section fill, which the page already describes and which is not among the four; the third missing capability is a switch nobody pins. It was caught while writing the page text these anchors were registered for — and the tempting repair was to add the unpinned switch to the check, which would have made the published sentence true by changing the world to fit it. Filed as an open item instead, to be decided on its own merits.

They were never forced into view because the code behind them was not among that page’s anchors. Registered now — and only on the pages that describe them, since anchoring code a page does not mention teaches people to ignore the signal.

⚠ One limit is now recorded next to the number it qualifies: the drift count does not know why a document was edited. A terminology sweep resets it to zero. The Korean page was showing 7 against the English page’s 17 while carrying identical content, because its last edit changed a single word.

The hypothesis was mine, and the measurement I wrote to test it refuted it

2026-09-06 · measure: what a rechunk actually costs

Outside practice derives a chunk’s identity from the source record plus a hash of that chunk’s own content, so re-chunking keeps the identity of anything that did not change. This system derives it from position — document, section path, index — and the index counts monotonically across the whole document. From that I wrote down the consequence: insert a sentence near the top and every identity below it shifts, orphaning whatever referenced them.

The measurement says zero. Across six documents and three edit shapes, not one identity moved. A small edit does not move a chunk boundary, so the (section, index) pair it is named by is unchanged; only one chunk’s body differs.

What does move them is a change in the number of chunks. Inserting a paragraph long enough to add one shifts almost everything after it — 21 of 22, 24 of 36, 14 of 15 — and in every one of those cases exactly one chunk’s body had actually changed. So when the cost is paid it is paid almost entirely on chunks whose content is identical: recomputed vectors, and orphaned references from spans, labels and feedback.

The remedy was not adopted. Changing an identity scheme is expensive to reverse, and the frequency of the edit that triggers it was unknown — so the next unit recorded chunk counts before and after each re-ingest, which makes the frequency answerable from that day forward and never backwards.

A stamp had been accumulating for three weeks and nothing read it

2026-09-06 · feat: read the provenance stamp nothing was reading

Twice this system hit the same defect: a derived value that a repair queue looking for NULL cannot see, because invalidation missed it and the value was never emptied. Coverage counts it as filled, so no number looks wrong. Asked whether a third instance existed, the sweep found none — of six derived artifacts, four already close the class with a hash stamp, and three further hypotheses were ruled out, one of them by measuring zero live.

What the sweep found instead was the log’s most repeated shape again. The table recording which model wrote each vector also records when it was written. That column had been filled since mid-August and no code read it.

Reading it splits the corpus in a way that costs one query: a vector written at or after its row’s last update cannot be stale. That rules out 1,736 of 2,045 rows and leaves 309 to check — where the only previous way to see a stale-but-present vector was recomputing every embedding.

The number it produces is deliberately not a verdict. A row’s timestamp also moves on a re-ingest whose text did not change, and no invalidation fires then, so the figure is an upper bound on what to check, not a count of what is wrong. Rows with no stamp at all are reported as a third state, because hiding them would make unknown and fine look the same.

The control group was pinned to a date, so it silently stopped existing

2026-09-06

The harness that detects a stale vector by recomputing it carries its own rule at the top: read the control group first, and if the control does not come out clean, the rest of the numbers are unusable. The control was defined as rows edited on a particular date. No such rows remain, so the control line is not printed at all — and its absence is silent.

That matters because the run came back 0 stale out of 466, and every cached comparison was exactly 1.000000. A perfect score is equally consistent with nothing is stale and this instrument distinguishes nothing.

So a control was supplied by hand: two chunks in a scratch tenant, one of them given the other’s vector. The harness flagged one of two, at 0.65 against a 0.9999 threshold. Only then was the zero worth reporting.

The harness was not changed in the same unit — it was the instrument in use, and altering an instrument mid-measurement is precisely what this repository forbids. The fix is recorded instead: define the control by the timestamp comparison above, which cannot rot into an empty set.

A test failed once, and the first explanation was the familiar one

2026-09-06

A full suite run went red on two tests. One was a real defect and the check was right to catch it: a hand-written database stand-in, which is effectively a whitelist of what one function may call, did not implement a method that had just been added — exactly the visibility that stand-in exists to provide.

The other was reported here as my new tests contaminated another module’s fixtures, because that had happened the day before. It does not hold. The test passes alone, passes beside the new module, and passed in a full green suite afterwards; the new module also sorts after it, so it cannot disturb what ran earlier. Two candidate mechanisms were ruled out by reading the code.

It is filed as unexplained rather than fixed. The path it exercises is an exact scan with a deterministic tie-break over deterministic vectors, so identical rows must give an identical answer — which means the rows differed, and nothing yet says why. One green run is not evidence of absence, and this repository has been fooled once by a green from a stale baseline.

What September adds to the log

August’s lesson was that the instrument was wrong more often than the system. For most of September the opposite held and it was more useful: the verdict was accurate, and the reading of it was wrong. A verdict names a state. Every wrong conclusion this month came from treating that state as if it named a cause — a missing fact, a narrow corpus, a misaimed run and a contaminated fixture all produce the same signal, and the difference between them is other evidence nobody had gone and gathered.

⚠ That framing did not survive the last day of the section. On the 6th an instrument was wrong again, and in the worst way available to one: its control group had been pinned to a date, the rows matching that date were gone, and the check therefore reported nothing rather than reporting that it could not run. A green from an instrument that cannot fail is not a result. So the month ends with both lessons live, and the second is the older one restated: an instrument that cannot say “I could not measure this” will eventually say “fine” instead.

The defences that actually worked were not better instruments. They were a pre-registration written before the run, a rule that counts open items rather than describing them, the habit of breaking a check on purpose to see whether it goes red, and — twice this month — a person asking a question the record could not answer.

⚠ And the same period shows the shape of what is not here. Against a published list of ten RAG failure modes, this system’s diagnostic questions cover six. The four they miss — retrieval timing, context position bias, retrieval-generation model mismatch, recursive retrieval loops — are missing for one reason: this system has none of those components. Not having met a failure is not the same as knowing it cannot happen.