FATSEARCH benchmarks
Are we better?
| Job | Better? | The numbers (FATSEARCH vs the best rival) |
|---|---|---|
| Reading pages inside an agent (Claude Code) | Mostly yesYes on money and tokens, about the same right answers, slower on blocked pages | right answers 50 vs 51 · $15.06 vs $53.48 per 1,000 tasks · 5,026 vs 38,129 tokens per task · slowest runs 43.8 s vs 21.5 sFATSEARCH vs Firecrawl MCP, 51 tasks |
| Hard pages (social, shops, news) | Mostly yesYes: more answers (35 vs 24), first answers faster, slower on walled pages | answer found 35 vs 24 · answered within 1 s 12 vs 0 · on the 24 pages both answered, median 23.7 s vs 4.9 sFATSEARCH 0.2.0 on 2026-10-21 vs Firecrawl on 2026-10-16, 38 pages |
| Web search quality | Results coming | no numbers yet |
Money = Claude Sonnet at list price + the tool's own fee. Hard pages and one page: model input only, Claude Sonnet 5.5 at $2/M input tokens (price, checked ); each reader's fee is listed with its set. We make FATSEARCH. The numbers below are our own runs, dated. Where another reader is the better pick, we say so.
FATSEARCH 0.2.0: 35 of 38 hard pages answered, 12 within the first second
Answer in the text: FATSEARCH 0.2.0 found 35 of 38 on 2026-10-21. The table below is the 2026-10-16 run: FATSEARCH 0.1.0 found 35, next best Firecrawl 24. A refusal never counts, however fast.
Pages with the answer back, by seconds after the request
38 hard pages · FATSEARCH 0.2.0 · FATSEARCH 0.1.0 and rivals
- FATSEARCH 0.2.0 · 2026-10-21 35 of 38
- FATSEARCH 0.1.0 · 2026-10-16 35 of 38
- Firecrawl · 2026-10-16 24 of 38
- Tavily · 2026-10-16 19 of 38
- Jina Reader (no key) · 2026-10-16 8 of 38
- Jina Reader (no key) · 2026-10-20 17 of 38
Seconds after the request, log scale →
38 pages: 2 pages from one government site were dropped: on 2026-10-20 it blocked every reader we ran, and no hosted reader ever read it.
A line rises only when a reader hands back a page with the answer in it. A refusal, a wall or an empty page never counts, however fast it comes back. Hover or tab to a time for the counts.
Rivals were measured on 2026-10-16 and FATSEARCH 0.2.0 on 2026-10-21: different days, and pages and providers change in between.
FATSEARCH 0.1.0 (2026-10-16, dashed): 35 answered, 12 within 1 s, 27 within 20 s. 0.2.0 is slower: fewer answers within 1 s or 20 s.
0.2.0 is slower than 0.1.3 on this run (19 vs 28 pages within 20 s; 13 pages slower, 1 faster, one new miss on TikTok). On most of the slower pages the paid call that won came back later. An A/B run of the 0.1.3 binary ten minutes after 0.2.0 (same script, keys and pages) was as slow: 33/38 answered, 12/20/21/33 within 1/10/20/60 s. Bright Data's winning call took a median 23.8 s then, against 13.9 s the day before. So this is the providers' hour, not a 0.2.0 change (measure/2026-10-20-hard-pages-0.2.0, sub-folder ab-0.1.3).
On the 24 pages both answered, Firecrawl’s median was 4.9 s, FATSEARCH 0.2.0 23.7 s. FATSEARCH tries the free request first and moves to a paid provider only when a wall blocks it: it was faster on the 10 pages its free request read and slower on the 14 behind a wall. Measured on 0.2.0 with the cache off.
On the 18 pages both answered, Tavily’s median was 1.0 s, FATSEARCH 0.2.0 23.7 s. On the 8 pages both answered, Jina Reader’s median was 3.5 s, FATSEARCH 0.2.0 12.1 s. On the 17 pages both answered, Jina Reader’s median was 5.8 s on its 2026-10-20 re-run, FATSEARCH 0.2.0 23.4 s.
Show as a table
| Answered within | FATSEARCH 0.2.02026-10-21 | FATSEARCH 0.1.02026-10-16 | Firecrawl2026-10-16 | Tavily2026-10-16 | Jina Reader(no key) · 2026-10-16 | Jina Reader(no key) · 2026-10-20 |
|---|---|---|---|---|---|---|
| 0.5 s | 4/38 | 3/38 | 0/38 | 4/38 | 1/38 | 0/38 |
| 1 s | 12/38 | 12/38 | 0/38 | 9/38 | 1/38 | 0/38 |
| 2 s | 17/38 | 16/38 | 1/38 | 10/38 | 3/38 | 0/38 |
| 3 s | 17/38 | 16/38 | 4/38 | 11/38 | 3/38 | 4/38 |
| 5 s | 17/38 | 16/38 | 13/38 | 14/38 | 5/38 | 7/38 |
| 10 s | 19/38 | 17/38 | 22/38 | 17/38 | 6/38 | 16/38 |
| 15 s | 19/38 | 20/38 | 23/38 | 17/38 | 6/38 | 16/38 |
| 20 s | 19/38 | 27/38 | 24/38 | 18/38 | 8/38 | 17/38 |
| 30 s | 24/38 | 30/38 | 24/38 | 18/38 | 8/38 | 17/38 |
| 45 s | 33/38 | 34/38 | 24/38 | 19/38 | 8/38 | 17/38 |
| 60 s | 35/38 | 35/38 | 24/38 | 19/38 | 8/38 | 17/38 |
| 90 s | 35/38 | 35/38 | 24/38 | 19/38 | 8/38 | 17/38 |
measure/2026-10-20-hard-pages-0.2.0 (per-page times) · FATSEARCH 0.2.0 · FATSEARCH 0.1.0 and the rivals: the run in Benchmarks doc § Other readersAnswers vs money
Model bill per 1,000 questions → (model input only)
Not on this chart (no cost measured): Tavily, Jina Reader (no key).
On top of the model bill, what each reader charges (its own words):
- Firecrawl: Free: 1,000 credits a month (one scraped page = one credit). Hobby: $16 a month billed yearly ($19 monthly) for 5,000 credits. Standard: $83 a month billed yearly for 100,000. source · checked
- Tavily: Free: 1,000 API credits a month. Pay as you go: $0.008 a credit. A basic extract costs 1 credit per 5 successful URLs (advanced: 2). source · source · checked
- Jina Reader (no key): Free without a key at 20 requests a minute. With a key it is billed by output tokens; a new key comes with 10M free tokens. source · checked
The same 38 hard pages (2026-10-16) through three hosted readers and Claude's built-in tools, one run each.
"Built-in" is Claude's own web search and web fetch tools.
In the agent runs, FATSEARCH answers "pending" after 5 seconds while a slow page is still loading. The model often gave up there, which cost it more points than any page it failed to read.
With FATSEARCH the tool runs the command line; with Firecrawl it returns Firecrawl's saved copy, cut to 1,000 tokens.
Agent rows: the 2026-10-16 run, FATSEARCH 0.1.0 (measure/2026-10-16-hard-pages-speed).
Show as a table, with where each reader missed
| Reader | Answer in the text | Model bill per 1,000 questions | Tokens per question |
|---|---|---|---|
| FATSEARCH | 35/38 (agent answered right 26/38) | $5.20 | 2.6k |
| Firecrawl | 24/38 (agent answered right 24/38) | $5.60 | 2.8k |
| Tavily | 19/38 | — | — |
| Jina Reader (no key) | 8/38 | — | — |
| Built-in fetch | 17/38 (page read · agent answered right 17/38) | $16.60 | 8.3k |
- Firecrawl declined 9 pages outright as sites it does not support (social networks, a news site, a recipe site, an airline).
- Tavily tried and came back empty on 16.
- Jina's keyless tier refused 21 domains for anonymous use on 2026-10-16 and only 3 on 2026-10-20, so its number is about access, not extraction.
- Four answers have gone stale since the questions were written; FATSEARCH 0.1.0 read all 34 that have not.
FATSEARCH: about the same right answers, 7.6× fewer tokens, 3.6× cheaper
The same 51 web tasks in Claude Code, one run each. A typical task takes about as long (FATSEARCH 5.3 s, Firecrawl 6.7 s); the slowest runs take FATSEARCH 43.8 s, Firecrawl 21.5 s.
Firecrawl MCP and FATSEARCH on the same 51 tasks
Each row on its own scale ·
Claude Code 2.1.280, claude-sonnet-5, headless, built-in web tools off; Firecrawl's hosted MCP with an API key vs the FATSEARCH MCP with an Oxylabs key. Right = the agent's final answer matched the known one. Every right answer was also in what the tool returned. We ran FATSEARCH twice more on the same tasks to check stability: 99/102 right.
Slowest runs = the slowest tenth (p90). FATSEARCH is slow on blocked pages: it waits for the paid render, and a known retry bug, which we are fixing, makes the agent wait again (5 runs over a minute, the worst 191 s).
Firecrawl's server cache already held most pages; FATSEARCH started empty, so Firecrawl's times are a best case.
Tokens = Claude Code's own count per task: input + cache + output. The Firecrawl MCP adds 10,977 tokens to every model call (16 tools and its instructions vs FATSEARCH's 1 tool).
$ = Claude at list price, as Claude Code reports it, + the tool: Firecrawl credits at the Hobby plan's $16 per 5,000 credits; FATSEARCH = Oxylabs' price as fats prints it.
Show as a table
| Reader | Right answers | Cost per 1,000 tasks | Tokens per task | Median / slowest tenth |
|---|---|---|---|---|
| FATSEARCH | 50/51 | $15.06 | 5,026 | 5.3 s / 43.8 s |
| Firecrawl MCP | 51/51 | $53.48 | 38,129 | 6.7 s / 21.5 s |
measure/2026-10-20-claude-code-firecrawl-vs-fats/results.json · 51 questions · When the agent has to search first
Measuring, results soon.
One page: FATSEARCH sends 1,384 tokens, the others 2× to 85× more
One question about Wikipedia “Rust (programming language)”, every reader we measured.
Tokens each reader hands the agent for the same question
One page · default settings · one bar per reader, grouped by when it was measured
Hosted readers ·
Open source, run on your machine ·
FATSEARCH reads an outline, one filter and one section; the others hand over the page. Tokens: Anthropic’s own counter. “Answer” = all of the answer strings are in what the reader returned. The bill is model input only; reader fees are not in it (FATSEARCH read this page over plain HTTP for $0). Each row is the tool’s default call; its other call is behind the checkbox. Time is not compared here (one fetch per reader, different networks).
Show as a table
| Reader | Call | Tokens | Answer | Bill per 1,000 reads | Measured |
|---|---|---|---|---|---|
| FATSEARCH | outline, then one filter, then one section (plain HTTP, $0) | 1,384 | found | $2.77 | |
| Firecrawl | v2 /scrape, markdown, default settings | 112,517 | found | $225 | |
| Bright Data | Web Unlocker, markdown | 103,625 | found | $207 | |
| Oxylabs | Web Scraper API /v1/scrape, markdown, no JS | 102,609 | found | $205 | |
| Tavily | /extract, basic depth | 97,504 | found | $195 | |
| Jina Reader | r.jina.ai, no key | 94,415 | found | $189 | |
| SearXNG + mcp-searxng | web_url_read, default (whole page as Markdown) | 117,088 | found | $234 | |
| SearXNG + mcp-searxng (other call) | Its headings first, then two sections picked from them (readHeadings, section) | 3,037 | found | $6.07 | |
| Firecrawl self-hosted | v2.11.162 from the self-host guide, /v2/scrape, markdown, default settings | 107,332 | found | $215 | |
| Firecrawl self-hosted (other call) | Through firecrawl-mcp (firecrawl_scrape), the tool text an agent gets | 113,609 | found | $227 | |
| Crawl4AI | Docker server 0.9.5, MCP md tool, default filter fit | 74,896 | found | $150 | |
| Crawl4AI (other call) | With the bm25 filter and our question as the query | 36,307 | found | $72.61 | |
| open-webSearch | fetchWebContent, default maxChars of 30,000 (cut, and it says truncated) | 11,609 | found | $23.22 | |
| open-webSearch (other call) | With maxChars 200000 (its maximum), the whole page | 34,881 | found | $69.76 | |
| mcp-server-fetch | fetch tool, default max_length of 5,000 characters: one call, the top of the page, cut with a hint to read on from start_index 5000 | 2,373 | not in text | $4.75 | |
| mcp-server-fetch (other call) | With max_length 999999 (its maximum), the whole page in one call | 76,147 | found | $152 |
measure/2026-10-17-wikipedia-rust, measure/2026-10-19-wikipedia-rust-oss · each reader in detailOver a day: Oxylabs 12–16, Bright Data 8–14 of 19 pages served
The same 19 hard pages, run 9 times in one day, one provider key at a time.
Each run of the day, one provider key at a time
19 hard pages · 9 runs · , EEST
- Oxylabs key only 12–16 of 19
- Bright Data key only 8–14 of 19
Runs on . Hover or tab to a run for its times.
Provider weather. The same build, run a minute apart, has moved by up to 3 pages.
p50 = median time to the first answer over all 19 pages, served or not.
Three earlier runs on the same 19 pages fell in the same bands: 2026-10-14 18:15–18:37 (Oxylabs 16 and 15 in two passes, Bright Data 14), 2026-10-14 21:19–21:40 (Oxylabs only, 16) and 2026-10-15 01:59–02:40 (Oxylabs 14, Bright Data 10). Both keys together served 14 of the 20 in the one run made with both (2026-10-14). Before the any-one-key ladder, one key got 10 and 12 of 20.
Spend for one 19-page pass: $0.017–0.028 billed per key mode at list prices, plus up to $0.013 in calls fats cut, printed as unverified.
Show as a table
| Run (EEST) | Oxylabs key only | p50 | Bright Data key only | p50 |
|---|---|---|---|---|
| 03:33–03:53 | 13/19 | 46 s | 14/19 | 21 s |
| 04:37–04:53 | 12/19 | 36.3 s | 8/19 | 17.1 s |
| 06:50–07:08 | 14/19 | 22.2 s | 13/19 | 17.5 s |
| 07:47–08:05 | 13/19 | 30.6 s | 11/19 | 12.9 s |
| 09:15–09:40 | 16/19 | 30.5 s | 8/19 | 23.6 s |
| 11:10–11:26 | 15/19 | 30.3 s | 12/19 | 28.4 s |
| 12:13–12:40 | 14/19 | 30.7 s | 12/19 | 20.9 s |
| 14:05–14:35 | 14/19 | 42.7 s | 10/19 | 25.5 s |
| 17:26–17:45 | 16/19 | 27.2 s | 9/19 | 37.7 s |
Easy pages: 7 pages, the slowest 732 ms
All seven answered from the free plain request, $0.
! no_host)101 msSeven pages fetched in parallel in one session, both keys set, 2026-10-15 17:26–17:45: example.com, a Wikipedia article, the Hacker News front page, a docs.rs crate page, an arXiv PDF, gov.uk, and a host that does not exist. All seven answered from the free plain request, $0.
A 17.4 MB binary that starts in 4.07–4.13 ms
- Release binary, symbols stripped17.4 MB17,405,936 bytes · macOS arm64 · 0.2.0
- Start-up, p504.07–4.13 msp90 4.30–4.57 ms · /usr/bin/true: 1.02–1.08 ms
- A cached read5.1 msFATSEARCH 0.1.0
- Hard wall for one run8.09 stimeout:8, both keys
- Refusing a 53 MB page10 msdefault 16 MiB cap · 27 MB RSS
- Real tokens per estimated token1.01×36 of the 42 reads within ±10%
Benchmarks doc § Binary and startup · one Apple Silicon Mac · startup measured
Where FATSEARCH loses
Pages it still doesn’t get
- Instagram posts (shells or "Post isn't available" pages)
- The YouTube results page on Bright Data (the results sit in a script block, see the outline's islands)
- Renders cut at 60 s (TikTok, the Joomla forum)
- Bright Data relaying a site's 502
- Reddit's captcha when it lands on both the plain request and the provider
Slower on walled pages
On the 24 pages both answered, Firecrawl’s median was 4.9 s, FATSEARCH 0.2.0 23.7 s. FATSEARCH tries the free request first and moves to a paid provider only when a wall blocks it: it was faster on the 10 pages its free request read and slower on the 14 behind a wall. Measured on 0.2.0 with the cache off. Rivals were measured on 2026-10-16 and FATSEARCH 0.2.0 on 2026-10-21: different days, and pages and providers change in between.
An agent gives up on “pending”
In the agent runs, FATSEARCH answers "pending" after 5 seconds while a slow page is still loading. The model often gave up there, which cost it more points than any page it failed to read.
Benchmarks doc § Other readers · measure/2026-10-20-hard-pages-0.2.0