When AI agents should answer — and when they should not

AI agents often become confidently wrong when a user describes something they know exists but can't name — a half-remembered book, film, song, or event. The fragment may carry enough signal to locate the answer, but not enough for a model to answer safely from memory.

This is the class of query Search Fragments is built for — there is a live demo on the homepage.

What a fragmented query looks like

Here is a real example from our test set:

“There’s a piece of music in a David Lynch scene where almost nothing happens. A woman sits at a table. The music is almost unbearable. I can’t remember the film.”

A keyword filter finds nothing useful — there is no clear title, composer, or scene description to match against. A plain LLM recall is worse: the model has seen enough Lynch to generate a plausible-sounding answer, but not enough to know which film and scene this actually is. The result is fluent, specific, and wrong. The fragment has real signal — it just needs evidence-grounded resolution, not a confident guess.

Why agents bluff

The core problem is that generation optimises for the plausible next token, not for whether evidence supports the specific claim being made. Confidence tracks how well an answer fits the language — how fluently it continues the query, how naturally it resolves the pattern — not how well the underlying facts support it. Nothing in the generation step checks that support: a fluent, specific, wrong answer is produced by exactly the same machinery as a fluent, specific, right one. This is how hallucination happens silently: the agent produces something that passes surface inspection and fails only when someone checks the underlying facts.

Fragmented queries make this worse. The query contains real signal — a director, a mood, a setting — which activates plausible-but-unverified associations. The model recognises the pattern without being able to verify the answer, and still commits to a response. The more recognisable the fragment, the more confidently wrong the hallucination tends to be.

The well-balanced answer

The fix is confidence calibrated to evidence. Rather than forcing a single answer at every confidence level, a resolution should produce one of three honest shapes:

  • Resolved A named answer, with confidence justified by the evidence found.
  • Shortlist Ranked web sources to read and confirm by eye — results returned, but no confident semantic identification made.
  • No resolution An explicit no-resolution — not resolvable from the clues given.

Built to decline rather than guess.

This matters because a confident-wrong answer is the worst kind of failure for an agent: it looks correct, passes downstream checks, and propagates into whatever the agent produces. An honest decline is far less harmful — it can be routed to a human, retried with a different tool, or surfaced to the user as a gap. Silence about uncertainty is what causes real damage.

What a baseline agent invented

On 50 hard, under-documented fragments, a baseline agent produced three specific fabrications — a real film falsely attributed to a real director who had no connection to it, a specific named sketch that was never performed, and a specific researcher and location for a study that has no single canonical paper. Search Fragments did not assert an answer to any of them.

SQL agent hallucination: when text-to-SQL agents invent the schema

The same failure shows up in SQL and text-to-SQL agents. A user asks a question in plain language; the agent writes a query. If the agent has not actually checked the database, it fills the gaps with whatever looks most plausible: a column that sounds right, a status value that “should” exist, a join that fits the question rather than the data. The SQL is fluent and often runs. The number it returns is wrong, and nothing in the output says so.

A worked example. The question:

“How many customers churned last quarter?”
-- Agent output (looks fine, runs, and is wrong)
SELECT COUNT(*) AS churned_customers
FROM customers
WHERE status = 'churned'
  AND churn_date >= '2026-07-01';

In this database there is no churn_date column, and the status column only ever holds active or trial. The query fails, the agent “fixes” it by dropping the date filter, and the new query runs and returns 0. The agent then reports “no customers churned last quarter” with full confidence. That is a hallucination with the same shape as an invented film title: plausible, specific, unsupported.

A grounded agent inspects the schema first and only uses identifiers that exist:

-- Grounded in the real schema
-- customers(id, created_at, plan)
-- subscriptions(customer_id, started_at, cancelled_at)
SELECT COUNT(DISTINCT s.customer_id) AS churned_customers
FROM subscriptions s
WHERE s.cancelled_at >= DATE '2026-07-01'
  AND s.cancelled_at <  DATE '2026-10-01'
  AND NOT EXISTS (
    SELECT 1 FROM subscriptions s2
    WHERE s2.customer_id = s.customer_id
      AND s2.started_at > s.cancelled_at
  );

Practical guards that cut SQL agent hallucination:

  • Read the real schema (for example information_schema.columns) before writing the query, and reject any table or column the schema does not contain.
  • Check categorical values against the data (SELECT DISTINCT status …) instead of assuming them.
  • Validate before running: a dry run or EXPLAIN catches invented identifiers cheaply.
  • Treat a suspicious result (zero rows, a round number, a sudden jump) as a prompt to re-check, not as an answer.
  • Show the query and state its assumptions, and let the agent say “this schema cannot answer that” rather than forcing a number.

Search Fragments is not a SQL tool, and it does not touch your database. It applies the same principle to a different gap: when the missing piece is a half-remembered name, title, person or event, or a specific claim that needs checking, it returns a named answer with sources, a ranked shortlist, or an honest no instead of a guess. Agents that already guard their SQL this way can hand those fragments to Search Fragments for the same treatment.

Try Search Fragments in Claude or Cursor. It is a free, non-commercial prototype. No account needed.

Try it

Try a fragmented query in Search Fragments — the demo runs a real resolution on whatever you put in, right now. Or add it in Claude or Cursor.

Connect

Search Fragments is a remote MCP server. Three tools: resolve_fragment, verify_claim, and submit_resolution_feedback. Free, no account needed.

{
  "mcpServers": {
    "search-fragments": {
      "url": "https://searchfragments.com/api/mcp"
    }
  }
}

Full documentation →