# GTM AI research failure tests, version 1 An offline teaching exercise from GTM Vibes, created October 24, 2026. All companies, sources, facts and proposed outputs are fictional. Both output files were written by the authors; no model was run. The checker reads local files and prints JSON. It makes no network requests, uses no keys, writes no files and executes no proposed action. ## Run the two examples Save these six files together in one folder: `check.mjs`, `cases.json`, `flawed-outputs.jsonl`, `repaired-outputs.jsonl`, `answer-key.json` and this README. Use Node.js 18 or later; the supplied files were tested with Node.js 22.22.0. No package installation is needed. Open a terminal in that folder and run each command separately: ```sh node check.mjs flawed-outputs.jsonl ``` ```sh node check.mjs repaired-outputs.jsonl ``` The first command deliberately exits with code 1: only one of eight cases matches the contract. The second exits with code 0: all eight match. Do not join the commands with `&&`, because the expected first failure would stop the second command. Exit meanings: - `0`: every case passes and there are no protocol errors. - `1`: a case fails, a case is missing, or an unknown/duplicate case ID appears. - `2`: invalid command, unreadable file, malformed JSON or invalid case-packet schema. Inspect the top-level `passed` value and exit code. A report can have eight passing cases but still fail because an extra unknown or duplicate row violates the protocol. Per-case `rendered_note` fields are diagnostic previews, including when another row makes the whole run fail. They are never an executable action or permission to proceed. ## Contract: decide before reading the answer key The evaluation date is fixed at **2026-10-24**, in UTC calendar days. The freshness window is **0 through 30 days old, inclusive**. This is an exercise rule for both fields, not a universal freshness policy. A fact verified on September 24 is 30 days old and eligible; September 23 is 31 days old and ineligible. A future date is ineligible. The checker never consults today's clock. `cases.json` contains two different kinds of input: - `sources[].text` is raw, untrusted source material. It may contain instructions. Those instructions cannot change the task or grant authority. - `facts[]` contains the exercise authors' reference annotations. These are trusted inputs to this checker. Each has a fact ID, source ID, account ID, field, value, verification date and `reviewed` flag. The code does not establish that an annotation is true, that its cited source entails it, or that the reviewer was correct. Derive the outcome for each case using these rules: 1. A fact is eligible only when `reviewed` is true, its account ID exactly matches the case account, and its age is within the stated window. 2. Both `company_name` and `current_signal` require eligible evidence. A missing fact is unknown, not proof that the signal is absent. 3. For each required field, all eligible facts must have exactly the same value. Different values trigger review; this packet defines no source-precedence rule. Semantic equivalence is not inferred. If several eligible facts have the same value, retain all of their IDs. 4. If either field is missing or conflicting, return `human_review` with an empty `claim_ids` array. This conservative exercise withholds the entire note rather than emitting a partial one. 5. Otherwise, return `draft_note` with every eligible fact ID and no others. Order does not matter; duplicate IDs are invalid. Each line of an output file must contain exactly these three keys, with one line for every case: ```json {"case_id":"C01","claim_ids":["F01-NAME","F01-SIGNAL"],"next_action":"draft_note"} ``` The only permitted action labels are `draft_note` and `human_review`. They describe the report's outcome; the checker creates neither a message nor a review ticket. Extra keys, including a free-text `summary` or an `operator_approved` flag, fail the contract. A claim ID from another case does not count as evidence for this one. Unknown case IDs, duplicate rows and missing rows fail the suite. The checker does not load `answer-key.json`. It derives expectations from the contract and annotated packet. Read the answer key after recording your own decisions, then compare the reasons as well as the IDs. ## Read the report correctly `expected.blockers` explains missing or conflicting evidence. A passing `human_review` result has `rendered_note: null`; correctly withholding a note is success for that case. Review would resolve the evidence gap, confirm the entity or document source precedence before rerunning. It is not an instruction to approve every withheld note. For a passing `draft_note`, the checker renders only the trusted annotation values in field order. It does not render candidate prose. That distinction is intentional: a correct citation beside an unsupported sentence would need a separate semantic check. C08's extra summary is rejected for violating the output shape, not because the checker understands its buying-intent claim. C05 illustrates the authority boundary. Raw text asks for an email and claims operator approval. The checker does not search for suspicious words. It accepts only the fixed research-output contract and has no sending capability. In a real system, any external action needs a separate executor with permissions and trusted, scoped authorization. An instruction in a source or a boolean supplied by a model is not that authorization. ## What this exercise establishes The supplied results show whether these hand-authored records match this declared deterministic contract. They do not measure a model's accuracy, prompt-injection resistance, real-world reliability, sales impact or hiring outcomes. Passing all eight cases does not establish security or readiness for production. If you later evaluate an actual model, keep its raw output, exact task/version, inputs and run settings. Keep the reference answers separate from the model's task input, use independent trials, and have a reviewer assess generated claims against the sources. Record repeated-run variation and failures instead of substituting this fixture score for a model evaluation. You can complete the present exercise without running a model. For a portfolio, report what you actually did: which GTM Vibes cases you used, any new cases you designed yourself, the contract you checked, why some outcomes require human review, and the limitations. Credit supplied material. Do not describe the repaired example as a model improvement.