Learning without ground truth

The situation where confident wrongness costs most: you cannot evaluate the answer, because not knowing is why you asked.

Every technique so far has assumed you can grade the output. You know your codebase, so you notice when a claim about it is wrong. Learning inverts that: you are asking precisely because you cannot evaluate the answer.

This is the highest-risk use of an agent and also one of the most valuable, so the discipline is about building external checks rather than about trusting less.

The three failure modes

1. Confidently outdated. The training cutoff means anything that changed recently is described in terms of its previous state, fluently. A framework's major version, a deprecated API, a library that was replaced — all get explained as current.

2. Plausible synthesis. The answer blends two real things into one that does not exist: a method from version 2 with a parameter from version 4, a pattern from one ecosystem described as idiomatic in another. Individually recognisable, collectively fictional.

3. Missing the shape of the disagreement. On genuinely contested questions — which state library, whether to use an ORM — you get one confident answer rather than the map of positions and trade-offs that you actually needed.

The rule

Never learn from an agent alone. Learn from an agent plus a primary source, where the agent's job is to help you read the primary source faster.

That reframing changes everything about how you prompt. Not "explain X" but "here is the documentation for X; explain the part I am asking about, quoting it."

# risky: answer comes from training data
"How do I do server-side rendering with streaming in this framework?"

# grounded: answer comes from the source in front of it
"Here is node_modules//dist/index.d.ts and the docs page I fetched
 [paste].
 Explain how streaming SSR works in THIS version. Quote the relevant
 declarations. If something I am asking about is not present here,
 say so rather than filling it in."

Ground everything you can

The installed package is the single best source, because it is the version you actually have. Documentation on the web may describe a different one.

The verification habit

For anything you will act on, one of three checks, in order of strength:

1. Run it. The strongest. A five-line script that exercises the claim settles it in a minute.
2. Find it in the source or the types. If the method exists, it is in the .d.ts.
3. Find it in a primary document — the spec, the release notes, the changelog.

"It sounded right" is not on the list, and the fact that it sounds right is precisely the problem: fluency is uncorrelated with accuracy and your own sense of plausibility is calibrated on writing, not on truth.

What agents are genuinely excellent at here

Once grounded, they are an outstanding learning tool, and better than documentation in one specific way: you can ask follow-up questions. Reading a spec answers the questions the author anticipated; asking about the spec answers yours. Use that, and keep the source in the window.

Answered with:
  client.get(url, { timeout: 5000 })

Confident, idiomatic, matches most HTTP clients, and is the v2 API.
In v4 the option is { signal: AbortSignal.timeout(5000) } and `timeout`
is silently ignored - no error, no warning, no timeout.

Would I have noticed? Almost certainly not. It looks exactly right,
TypeScript accepted it because the options type has an index signature,
and the failure mode is "requests occasionally hang forever" - which I
would have debugged for an hour, weeks later.

Same question, with node_modules//dist/index.d.ts pasted:

  "The RequestOptions interface in this version declares:
     signal?: AbortSignal
     retry?: RetryOptions
     headers?: HeadersInit
   There is no `timeout` field. Per-request timeouts are expressed with
   AbortSignal.timeout(ms) passed as `signal`. I have quoted the
   declaration; there is nothing named timeout in this file."

const t0 = Date.now()
try { await client.get('https://httpbin.org/delay/10', { timeout: 500 }) }
catch { }
console.log(Date.now() - t0)   // 10004  -> timeout ignored, confirmed

try { await client.get('https://httpbin.org/delay/10',
                       { signal: AbortSignal.timeout(500) }) }
catch { }
console.log(Date.now() - t0)   // 507    -> works

A course by Pieter Zandbergen