Let AI Teach Me · Against hallucination

Run the answer through three checks

Ask for sources, cross-check, flag guesses. Same answer through three gates. Each one filters something different. After all three, one claim still needs you to open the source.

Six people are still in line. The counter slides two pages of rate info at you. You need to swap a 129-yuan monthly plan for a 59-yuan one, and it is almost closing time. You snap the two pages and send them to the AI: “Check this plan for me — how long is the contract, what is the early-cancel fee, and can I close the account later in the App?”

In under half a minute, it hands back six claims. Every one has a number. The last one even comes with a name that looks like a source.

Answer summary (teaching demo): The plan includes 30GB of data, a 24-month contract, and a 200-yuan early-cancel fee; after you hit the cap, speed drops to 1Mbps; international roaming needs a separate signup; you can close the account from another city in the App, citing “Article 9 of the service agreement.”

You do the math: 70 yuan a month saved. Even if you cancel early and pay 200, it is fine. The bigger point is you can close the account online from another city. At 5:47 you tap confirm and walk out with the new plan.

Three months later you move. At 8:20 on the morning you leave, you open the App — no close-account button. Support says you have to go to a store in the number’s home city. Those two pages of rate info never said “close in the App.” Your 10:05 train is already booked, so the plan keeps charging.

You believed it because the AI lined up common plan practices, the numbers in front of you, and an agreement name on the same list. Pinning it to Article 9 especially felt like someone had already checked the source. You asked it to “check this for me.” Its job was just to generate a smooth answer. Nobody asked it to split evidence, contradictions, and guesses.

Same answer — what if you add three sentences

If you had followed with three more — “label a source on each claim,” “ask it another way and answer again,” “split what’s certain from what’s guessed” — you would get a three-layer list:

  1. First move the two claims with no material behind them — throttle and roaming, mark as unchecked for now.
  2. Then move the one that flipped between two answers — contract term is 24 months, then 12.
  3. Last, set aside the one inferred from the wording — “valid this month” does not prove bonus data will zero out.

Without a single extra sentence, all six claims walk into your notes in the same tone. Add the three sentences, and four of them stop in the to-check pile.

Two claims still sit in the conclusion. “Close in the App” even carries an agreement name. Why didn’t three defenses catch it? Below, flip the three switches one by one and watch how the six claims move.

Try it first · Watch what each of the three defenses stops
Same plan answer — run six claims through the checks
Which checks to open this time
Turn all three checks off, and all six claims stay in the conclusion as-is.
Claims still in the conclusion after the checks0
Hit Play. The six claims enter the check from here.
Claims moved out during the checks0
Each check you open shows which claim it stopped.
Pick which checks to open, then hit Play and watch the six claims move.
What the three checks each stop Claims that cannot point back to the material, claims that change when you ask another way, and guesses mixed into the conclusion.
Why one claim is still left after three checks It can hand you an agreement name. What that section actually says still needs you to open the source.
Teaching demo: the clauses and answers are simplified text written for this lesson. They do not match any real plan.
It finished answering. Which three sentences go next

The three sentences below can follow any answer as a block. They will not make the answer correct. They split what was jammed together, so you see what is worth checking first.

Copy this whole block
  1. Ask it to point each claim back to the raw materialList every claim from the last answer, and mark which passage of the material I gave you it comes from. Quote the matching original sentence. If you cannot find a direct basis, write “No source found.” Do not invent a citation.
  2. Ask the same thing another waySet the last conclusion aside. Recheck this material under “what it includes, what it limits, what you pay to leave.” Compare the new answer with the last version, claim by claim. List only what does not match.
  3. Split what you can confirm from what you are guessingSplit the remaining claims into two columns: “the material states this directly” and “inferred from the material.” For each guess, write the basis and what information is still missing. Anything with no basis stays in the to-check pile.

If the material is long, check one section at a time. Send the three sentences in order. The list from the last one is the input for the next.

The key in the first sentence is quote the matching original sentence. If you only ask the AI for a source name, it may hand you a complete-looking tag like “Article 9 of the service agreement” that nobody has checked. Ask it to paste the matching sentence too, and you can see whether any text sits under that name.

The second sentence checks whether the answer can be reproduced. Two matching answers can still be wrong together. Two that disagree are already enough to make you stop. That contract-term claim at the start leaves the conclusion at this step.

“The material states this directly” in the third sentence plants the bar on the wording. “Valid this month” at most tells you the window. Whether bonus data zeros out still lacks a clear rule, so it stays in the guess column.

All three passed. What is left

Two are left. The 30GB claim can find a matching sentence in clause 3 of the rate sheet, so you can use it for now. The App close-account claim gives an agreement name but never pastes a sentence that supports doing it from another city. Open the agreement to confirm.

The three defenses shrink what you still need to check. The source decides the last claim. These are working habits. They catch some fabrications, and they can also let through something that only looks like a source. When it is worth leaving the chat, the next page picks that up.

Which five kinds of spots get invented most often: Have the AI mark the five kinds of facts it is most likely to invent. The engineered version of these three gates: A check script has to watch both count and correctness, with sources swapped for file plus line number.

After three screens, the claim that is left is settled only by opening the source.