Fragment comparison — SF vs. Tavily

“On fragments where it produced a candidate, Search Fragments contributed a candidate the incumbent result set did not contain 44% of the time.”

Method

What was captured. The SF side used resolve_fragment via MCP against the production endpoint. 12 fragments (R15–B03) had their SF side captured 2026-08-23 via the preview deployment; the incumbent side for those rows was also captured 2026-08-23 except R15, which was re-queried 2026-08-28 after an encoding error. The remaining 13 fragments (B04–X10) had both sides captured 2026-08-28 via production.

Incumbent. Tavily Search API, verbatim fragment as query, top 5 result titles. Tavily results vary by user, region, and date. A reader rerunning this may see a different incumbent set and reach a different delta.

These are constructed fragments, not a sample of real usage. Every fragment was written to be solvable — a specific real thing exists that matches the description. This shows which outcome shapes fire and whether SF contributes a candidate the incumbent lacks; it does not show the real distribution of query types or the real share of queries SF can resolve. Actual usage contains unsolvable queries, ambiguous queries, and queries where the fragment is too thin to resolve. The numbers here are an upper bound on resolve rate, not a baseline.

Denominator. 25 fragments run. 7 produced no candidate (ranked_stalls with no title, or no_resolution). Those 7 are not hidden — they are reported in the table and the zeros section below. The denominator for the 44% rate is 18: the fragments where SF produced a named candidate.

We do not score correctness.A candidate counts as “contributed” when it contains at least one content word (≥5 chars) absent from all five incumbent titles. Whether the candidate is right is decided by eye.

R01 — The Most Mysterious Song, and why it is not a loss

R01 is the fragment about a song heard on German radio in the 1980s — the query that became known as The Most Mysterious Song on the Internet. SF returned no_resolution. Tavily surfaced the identification directly (titles 2 and 5 name the viral search and the song).

On the hit-rate metric, that scores as zero for SF. It is not a loss. The correct answer was deeply contested for decades and was only recently resolved by a large collaborative effort. SF declined rather than guessing at low confidence. Tavily surfaced crowd-sourced identification that SF correctly did not claim as its own resolution.

That is the product doing what it says — built to decline rather than guess — on a query where guessing would have looked good. A reader who thinks R01 weakens the case should read the five Tavily titles: none of them answer the fragment. They point to the answer. SF pointed to nothing and said so.

Outcome distribution

ranked_stalls + no_resolution combined: 8 / 25 (32%). On constructed, solvable fragments, the engine declines or hedges nearly a third of the time. That is the correct behaviour when it cannot reach confident identification.

Marginal contribution rate

Delta method: is_new=True when SF title contains ≥1 word of ≥5 chars absent from all five Tavily titles (case-insensitive). Conservative — a stricter test requiring the specific work/model name would move some is_new=False rows to True (see B03 note in the detail below).

All 25 rows

Zeros are rendered with the same weight as wins. No row is hidden or collapsed.

Per-fragment detail

Fragment text, SF result, top-5 Tavily titles with URLs, delta reasoning. All rows — zeros and wins rendered the same.

Rerun any row

To reproduce a row: run the fragment text against the Tavily Search API (verbatim, top 5), then run it on searchfragments.com, and compare the candidate sets. Tavily results vary by date and region, so you may see a different set and reach a different delta.

A general web search (Google, Bing, etc.) will give a different result set — Tavily is an LLM-oriented search API with different ranking. We have not measured the delta against Google or any other engine. The 44% figure applies to Tavily only.

← Documentation · Search Fragments