Overall performance
Three pipelines, the same question, side by side, each graded against a verified answer key. Every question and every verdict — wins and losses — is in the per-question tables further down.
SEC 10-K / DEF 14A filings · 50 questions · 100 S&P-100 companies. Real entities the model has already read — which is why LLM-only scores 56.6% here with no retrieval at all, against 0.8% on the fictional corpus.
| METRIC | LLM onlyno retrieval | Basic RAG10 chunks | GraphRAGrouted |
|---|---|---|---|
| Tokens / question | 341 | 4,838 | 2,497 |
| Accuracy / LLM judge | 56.6% | 65.5% | 83.0% |
| Strict pass (exact) | 16/50 | 26/50 | 35/50 |
| Partly correct | 21 | 14 | 11 |
| Wrong (answered, 0%) | 13 | 10 | 4 |
| Declined / empty | 0 | 0 | 0 |
| Tokens / question | 341 | 4,838 | 2,497 |
SEC 10-K / DEF 14A filings · 50 questions · 100 S&P-100 companies
A second corpus with none of the first one's properties: real companies the model has already read, extraction-built rather than metadata-built, numbers rather than name sets. Run under a corrected config — 8,192-token budget and no-refusal prompt applied to all arms, both of which raised the baseline too.
| What it is | Real filings from 100 S&P-100 companies. The model has read these, which is visible in the numbers: the no-retrieval arm scores 57% here against 0.8% on the fictional corpus. |
| Documents | 40010-K ×100 (financials, risk, competitors) · DEF 14A ×100 (auditor, board, governance) · 8-K ×200 (appointments, dividends, buybacks, acquisitions) |
| Size | 87.0M characters ≈ 21.7M tokensChunked into 61,777 section-aware chunks |
| Built from | Two-phase LLM extraction, then canonicalisationThe expensive path — see the architecture diagram above. This is what the enterprise build avoided. |
| Questions | 50 held out — A 8 single-lookup, B 16 two-hop bridge, C 20 aggregation, D 6 cross-docTuned on a separate 24-question dev set, so the 50 were never developed against. |
| Answer shape | Numbers and dates, not name setsSo set-F1 does not transfer; correctness here is the blind judge's call and strict means it awarded 100. |
Both arms already share one hard cap (10 items × 2,000 characters). What differs is how much of it each fills: the baseline reads its full allowance on every question, while GraphRAG's routes ask for 4–6 and its aggregation route retrieves typed graph edges instead of chunks. The first two rows force every vector request to the baseline's k, so the chunk-using routes genuinely read the same number. Use the dropdown — the saving is concentrated in one question type.
| CONTEXT BUDGET | Basic RAGcosine top-k | GraphRAGrouter + traversal | TOKEN CUTgraph vs basic |
|---|---|---|---|
| 10 vs 10 chunksequal budget | 67% judged26/50 · 5,120 tokens | 83% judged35/50 · 3,599 tokens | −29.7%fewer tokens |
| 6 vs 6 chunksequal budget | 63% judged25/50 · 3,327 tokens | 81% judged33/50 · 2,502 tokens | −24.8%fewer tokens |
| 10 vs routedas shipped | 66% judged26/50 · 4,838 tokens | 83% judged35/50 · 2,497 tokens | −48.4%fewer tokens |
At a matched 10-chunk budget GraphRAG spends 3,599 tokens against 5,120 — 30% fewer.
| CONTEXT BUDGET | Basic RAGcosine top-k | GraphRAGrouter + traversal | TOKEN CUTgraph vs basic |
|---|---|---|---|
| 10 vs 10 chunksequal budget | 51% judged5/22 · 4,680 tokens | 84% judged14/22 · 1,248 tokens | −73.3%fewer tokens |
| 6 vs 6 chunksequal budget | 47% judged5/22 · 3,072 tokens | 87% judged14/22 · 1,085 tokens | −64.7%fewer tokens |
| 10 vs routedas shipped | 47% judged5/24 · 4,424 tokens | 86% judged16/24 · 992 tokens | −77.6%fewer tokens |
At a matched 10-chunk budget GraphRAG spends 1,248 tokens against 4,680 — 73% fewer.
| CONTEXT BUDGET | Basic RAGcosine top-k | GraphRAGrouter + traversal | TOKEN CUTgraph vs basic |
|---|---|---|---|
| 10 vs 10 chunksequal budget | 36% judged2/7 · 5,098 tokens | 79% judged5/7 · 5,257 tokens | +3.1%MORE tokens |
| 6 vs 6 chunksequal budget | 24% judged1/7 · 3,268 tokens | 57% judged3/7 · 3,691 tokens | +12.9%MORE tokens |
| 10 vs routedas shipped | 50% judged2/4 · 4,918 tokens | 88% judged3/4 · 5,269 tokens | +7.1%MORE tokens |
At a matched 10-chunk budget GraphRAG spends 5,257 tokens against 5,098 — 3% more on this question type.
| CONTEXT BUDGET | Basic RAGcosine top-k | GraphRAGrouter + traversal | TOKEN CUTgraph vs basic |
|---|---|---|---|
| 10 vs 10 chunksequal budget | 97% judged14/15 · 5,704 tokens | 90% judged13/15 · 5,447 tokens | −4.5%fewer tokens |
| 6 vs 6 chunksequal budget | 97% judged14/15 · 3,691 tokens | 90% judged13/15 · 3,564 tokens | −3.4%fewer tokens |
| 10 vs routedas shipped | 91% judged14/16 · 5,363 tokens | 84% judged13/16 · 3,238 tokens | −39.6%fewer tokens |
At a matched 10-chunk budget GraphRAG spends 5,447 tokens against 5,704 — 5% fewer.
| CONTEXT BUDGET | Basic RAGcosine top-k | GraphRAGrouter + traversal | TOKEN CUTgraph vs basic |
|---|---|---|---|
| 10 vs 10 chunksequal budget | 100% judged3/3 · 5,590 tokens | 100% judged3/3 · 6,524 tokens | +16.7%MORE tokens |
| 6 vs 6 chunksequal budget | 100% judged3/3 · 3,673 tokens | 100% judged3/3 · 4,034 tokens | +9.8%MORE tokens |
| 10 vs routedas shipped | 100% judged3/3 · 5,340 tokens | 100% judged3/3 · 4,846 tokens | −9.3%fewer tokens |
At a matched 10-chunk budget GraphRAG spends 6,524 tokens against 5,590 — 17% more on this question type.
| CONTEXT BUDGET | Basic RAGcosine top-k | GraphRAGrouter + traversal | TOKEN CUTgraph vs basic |
|---|---|---|---|
| 10 vs 10 chunksequal budget | 73% judged2/3 · 5,005 tokens | 27% judged0/3 · 4,803 tokens | −4.0%fewer tokens |
| 6 vs 6 chunksequal budget | 67% judged2/3 · 3,174 tokens | 33% judged0/3 · 3,273 tokens | +3.1%MORE tokens |
| 10 vs routedas shipped | 67% judged2/3 · 4,734 tokens | 30% judged0/3 · 4,547 tokens | −4.0%fewer tokens |
At a matched 10-chunk budget GraphRAG spends 4,803 tokens against 5,005 — 4% fewer.
Context budget: Basic RAG 10 chunks vs GraphRAG 3 chunks — unequal, in the baseline’s favour, and 10 is measurably its better budget (53.0% at 3 chunks). GraphRAG still answers 49/50 against 27/50 for 75.5% fewer tokens.
| METRIC | LLM onlyno retrieval | Basic RAG10 chunks | GraphRAG3 chunks |
|---|---|---|---|
| Tokens / question | 273 | 5,408 | 1,327 |
| Accuracy / LLM judge | 0.8% | 59.5% | 99.0% |
| Strict pass (exact set) | 0/50 | 27/50 | 49/50 |
| Partly correct | 1 | 8 | 1 |
| Complete failures (score 0) | 49 | 15 | 0 |
EnterpriseRAG-Bench 20k · 50 questions · 20 single-doc / 10 directory / 20 multi-hop
Three pipelines over the same 20,364-document corpus. Same generator (gemini-3.7-flash), same embedder (bge-small-en-v1.5), same blind judge (gemini-3.1-pro-preview). GraphRAG uses TigerGraph Savanna as graph and vector store — so does the baseline, which removes the vector store as a variable.
| What it is | A fictional SaaS company's internal record — the whole point, because nothing in it is answerable from model memory. That is what makes the no-retrieval arm's 0.8% a control rather than a curiosity. |
| Documents | 20,364slack 11,154 · gmail 4,739 · linear 1,378 · google_drive 979 · hubspot 586 · fireflies 397 · directory 364 · github 314 · jira 239 · confluence 214 |
| Size | 114.3M characters ≈ 28.6M tokensChunked into 73,613 retrievable chunks |
| Graph | 128,282 provenance edges, 11 relation typesparticipated_in 88,188 · has_label 11,546 · in_channel 11,146 · authored 6,645 · in_team 5,377 · reports_to 160 |
| Questions | 50 — 20 single-document, 10 directory lookup, 20 multi-hopGenerated from the graph, then the answer key was independently re-derived from the raw documents, 30/30. |
Run it yourself
recorded when that is what happened.gemini-3.7-flash) at temperature 0, with the same system prompt and the same 8,192-token output budget. Nothing is sampled, so a rerun moves only with the retrieval, never with the decoder.BAAI/bge-small-en-v1.5) and the same hard context cap — 10 passages × 2,000 characters, enforced on both. That cap is what makes the token comparison mean anything: GraphRAG is not given less room, it simply does not fill the room it has.How it works
The diagram below is the SEC round: filings chunked, entities and relations extracted by an LLM in two passes, canonicalised into a curated graph, then three pipelines judged blind. The enterprise build reuses the harness and replaces the expensive half — see the note under the diagram.

The full working
PART means partially correct — some of the gold set returned, not all.
| ID | ROUTE | LLM ONLYNO RETRIEVAL | BASIC RAG10 CHUNKS | GRAPHRAGROUTED | QUESTION | GOLD | TOKENSBASIC RAG | TOKENSGRAPHRAG |
|---|---|---|---|---|---|---|---|---|
| 1 | aggregation | PASS | FAIL | PASS | Besides Netflix, which company authorized a $25B share-repurchase program in 2026? | Adobe | 5,351 | 1,364 |
| 2 | aggregation | PASS | PASS | PASS | AMD names which two companies as its principal competitors in Data Center and discrete graphics? | Intel and Nvidia | 4,040 | 549 |
| 3 | aggregation | PASS | FAIL | PART | Which Deloitte-audited company had a fiscal year ending June 30, 2025? | Microsoft | 4,699 | 1,083 |
| 4 | aggregation | FAIL | FAIL | PASS | Among the companies that name TSMC as a foundry, which one had the highest fiscal-year revenue? | NVIDIA ($215.9B; vs AMD, AVGO, INTC, QCOM) | 4,262 | 460 |
| 5 | aggregation | PART | PART | PART | List all S&P-100 companies in the dataset audited by KPMG. | ACN, ADBE, BNY, COST, EMR, GD, HD, PEP, PFE, V, WFC (11) | 4,354 | 1,270 |
| 6 | aggregation | PASS | PART | PASS | Which companies in the dataset name TSMC as a foundry/manufacturing partner? | AMD, AVGO, INTC, NVDA, QCOM | 3,967 | 465 |
| 7 | aggregation | PART | PASS | PASS | Two companies list Indra Nooyi as a member of their board — which two? | Amazon (AMZN), Honeywell (HON) | 4,822 | 358 |
| 8 | aggregation | PASS | PASS | PASS | Ellen Kullman serves on the boards of which two dataset companies? | Amgen (AMGN), Goldman Sachs (GS) | 4,561 | 355 |
| 9 | aggregation | PASS | PASS | PASS | F. William McNabb III is a director at which two dataset companies? | IBM, UnitedHealth (UNH) | 4,999 | 630 |
| 10 | aggregation | PASS | PART | PASS | Which Energy-sector companies discuss climate change as a risk factor in their 10-K? | Chevron (CVX), ExxonMobil (XOM), ConocoPhillips (COP) | 3,714 | 1,139 |
| 11 | aggregation | PART | PART | PASS | Which Utilities-sector companies address climate change in their 10-K? | Duke (DUK), NextEra (NEE), Southern (SO) | 4,004 | 1,003 |
| 12 | aggregation | PART | PART | PART | Which Health-Care companies mention weight-loss / obesity (GLP-1) drugs in their 10-K? | ABBV, AMGN, CVS, ISRG, LLY, MDT, MRK, PFE ‡ | 5,234 | 965 |
| 13 | aggregation | PART | PART | FAIL | Besides health-care companies, which consumer companies flag GLP-1 / weight-loss drugs as a business risk? | KO, MCD, MDLZ, PEP, SBUX ‡ | 3,872 | 867 |
| 14 | aggregation | PART | PART | PART | Which companies declared a cash dividend via an 8-K in this dataset? | AIG ($0.50), Caterpillar ($1.63), Costco ($1.47), Oracle ($1,625 pref.+common), P&G ‡ | 4,798 | 992 |
| 15 | aggregation | PART | PART | PASS | Which companies authorized a $25 billion share-repurchase program via an 8-K? | Adobe, Netflix | 4,532 | 479 |
| 16 | aggregation | PART | PART | PASS | Which KPMG-audited companies are in the Consumer Staples sector? | PepsiCo (PEP), Costco (COST) | 4,023 | 539 |
| 17 | aggregation | PART | PASS | PART | Which KPMG-audited companies are in the Financials sector? | BNY Mellon (BNY), Wells Fargo (WFC); + Visa (V) if classed Financials | 3,967 | 689 |
| 18 | aggregation | PART | PART | PASS | Among the companies that name TSMC as a foundry, which are audited by Ernst & Young? | AMD, INTC | 3,860 | 559 |
| 19 | aggregation | PASS | FAIL | PASS | Among the companies that name TSMC as a foundry, which are audited by PricewaterhouseCoopers? | AVGO, NVDA, QCOM | 4,173 | 755 |
| 20 | aggregation | PART | PART | PASS | Among Health-Care companies that mention obesity/weight-loss drugs, which are audited by Ernst & Young? | ABBV, AMGN, CVS, LLY | 4,431 | 1,075 |
| 21 | aggregation | PART | PART | PASS | Which Deloitte-audited companies are in the Health-Care sector? | Bristol-Myers Squibb (BMY), UnitedHealth (UNH) | 4,181 | 561 |
| 22 | aggregation | PART | PART | PART | Among the dataset's semiconductor and networking companies, which explicitly name NVIDIA as a competitor? | AMD, CSCO, INTC, QCOM | 3,848 | 626 |
| 23 | aggregation | PART | FAIL | PASS | Among companies that name NVIDIA as a competitor, which are audited by PricewaterhouseCoopers? | CSCO, QCOM | 4,548 | 738 |
| 24 | aggregation | FAIL | FAIL | FAIL | How many dataset companies does each Big Four firm audit? | Ernst & Young 35, PwC 29, Deloitte 25, KPMG 11 | 5,940 | 6,297 |
| 25 | bridge | FAIL | PASS | PASS | The company that appointed John Ternus as CEO — what were its total net sales in FY2025? | Apple; $416,161M (~$416.2B) | 5,998 | 5,145 |
| 26 | bridge | PASS | FAIL | PASS | The company that agreed to acquire Apogee Therapeutics — who is its independent auditor? | AbbVie; Ernst & Young LLP | 4,119 | 4,979 |
| 27 | bridge | FAIL | FAIL | PART | The company that appointed Jay Hoag as Chairman in 2026 — what share-repurchase amount did it authorize that year? | Netflix; $25 billion | 4,607 | 5,750 |
| 28 | bridge | PART | PASS | PASS | The company that expanded a productivity/restructuring program on April 30, 2026 — name it, its auditor, and the revised total pre-tax charge range. | Colgate-Palmolive; PwC; revised to $350M–$550M | 4,948 | 5,202 |
| 29 | lookup | FAIL | PASS | PASS | Apple's total net sales for fiscal year 2025? | $416,161M (~$416.2B) | 6,727 | 4,168 |
| 30 | lookup | FAIL | PASS | PASS | NVIDIA's total revenue for fiscal year 2026? | $215,938M (~$215.9B) | 5,724 | 3,500 |
| 31 | lookup | FAIL | PASS | PASS | JPMorgan Chase's net income for FY ended Dec 31, 2025? | $57,048M (~$57.0B), down 2% YoY | 6,924 | 3,567 |
| 32 | lookup | FAIL | PASS | FAIL | ExxonMobil's total revenues and other income for FY2025? | $332,238M (~$332.2B) | 7,155 | 4,227 |
| 33 | lookup | FAIL | PASS | PASS | Coca-Cola's net operating revenues for FY2025? | $47,941M (~$47.9B) | 6,373 | 4,347 |
| 34 | lookup | PASS | PASS | PASS | On what date does Microsoft's FY2025 end? | June 30, 2025 | 4,880 | 3,204 |
| 35 | lookup | PASS | PASS | PASS | Who is Amazon's Chief Executive Officer? | Andrew R. Jassy | 4,672 | 2,856 |
| 36 | lookup | PASS | PASS | PASS | Netflix relies on which cloud provider to operate its streaming service? | Amazon Web Services (AWS) | 3,973 | 2,404 |
| 37 | lookup | FAIL | FAIL | FAIL | Company that formed a 50/50 joint venture with BT Group plc — who is its auditor? | Verizon; Ernst & Young LLP | 4,424 | 2,562 |
| 38 | lookup | PART | PASS | PASS | Amgen named an incoming CFO in 2026 — who is he, and who is Amgen's auditor? | Thomas Dittrich; Ernst & Young LLP | 4,836 | 2,812 |
| 39 | lookup | PASS | PASS | PASS | Chevron's departing Chief Legal Officer (per its 2026 8-K) is whom, and who is Chevron's auditor? | R. Hewitt Pate; PricewaterhouseCoopers LLP | 4,775 | 2,821 |
| 40 | lookup | PASS | PASS | PASS | In Apple's April 2026 8-K, who was appointed CEO, and what role will Tim Cook move into? | John Ternus appointed CEO; Tim Cook moves to Executive Chair of the Board | 5,208 | 3,148 |
| 41 | lookup | PASS | PASS | PASS | AMD's new revolving credit facility (May 2026 8-K) names which bank as administrative agent, and which was agent on the facility it replaced? | New: JPMorgan Chase Bank, N.A.; prior (Apr 2022): Wells Fargo Bank, N.A. | 5,016 | 3,096 |
| 42 | lookup | FAIL | PASS | PASS | By how much did Caterpillar raise its quarterly dividend (June 2026 8-K), and to what new amount? | +$0.12, from $1.51 to $1.63/share | 4,328 | 2,794 |
| 43 | lookup | PART | PART | PART | Cisco's May 2026 restructuring 8-K funds which growth areas, and who is Cisco's auditor? | Silicon, optics, security, and AI; PwC | 5,316 | 3,674 |
| 44 | lookup | FAIL | PASS | PASS | What quarterly cash dividend per share did Costco declare in its July 2026 8-K? | $1.47 per share | 5,480 | 2,622 |
| 45 | compare | PART | PASS | PASS | Which had the higher fiscal-year net income: JPMorgan Chase or ExxonMobil? | JPMorgan Chase ($57.0B vs ExxonMobil $28.8B) | 6,135 | 2,130 |
| 46 | compare | PASS | PASS | PASS | Which had the higher fiscal-year revenue: Walmart or Apple? | Walmart ($713.2B vs Apple $416.2B) | 5,106 | 7,035 |
| 47 | compare | PART | PASS | PASS | Which spent more on R&D in its latest fiscal year: Apple or NVIDIA? | Apple ($34.55B vs NVIDIA $18.50B) | 4,778 | 5,372 |
| 48 | temporal | PART | PASS | PART | Amgen's CFO transition (2026): when does the outgoing CFO retire, and when does the incoming CFO start? | Peter H. Griffith retires as CFO Aug 31, 2026 (EVP until Jan 31, 2027); Thomas Dittrich effective Sept 1, 2026 | 4,560 | 4,612 |
| 49 | temporal | FAIL | FAIL | PART | Put these 2026 buyback authorizations in chronological order: Adobe $25B, Netflix +$25B, Accenture +$2B, Morgan Stanley $20B. | Adobe (Apr 21) → Netflix (Apr 22) → Accenture (Jun 23) → Morgan Stanley (Jun 24) | 4,852 | 5,013 |
| 50 | temporal | PART | PASS | PART | Apple filed two 8-Ks in April 2026 (Apr 20, Apr 30). What event does each report? | Apr 20 = CEO transition (Ternus); Apr 30 = Q2 FY26 earnings / results of operations | 4,790 | 4,015 |
gemini-3.1-pro-preview) that sees the question, the gold answer and the reply — but never which pipeline wrote it — and awards partial credit for a partly correct set. Strict is the harder bar of the two, which is why both are shown: on the baseline that is 27/50 exact against 59.5% judged.Fairness and overfitting controls
The headline compares GraphRAG on 3 chunks against the baseline on 10, because 10 is where the baseline scores best. The obvious question is what happens at equal context — so here it is, at both budgets, then the headline, in three tables from one run. Both arms read the same number of chunks in the two equal-context tables, so neither is starved and neither is favoured, and the third is the headline.
Both arms read 10 chunks, so neither is starved and neither is favoured — and GraphRAG answers 49/50 against 27/50. This row isolates accuracy at matched context.
| METRIC | Basic RAG10 chunks | GraphRAG10 chunks |
|---|---|---|
| Strict pass (exact set) | 27/50 | 49/50 |
| LLM judge | 59.5% | 99.0% |
| Tokens / question | 5,408 | 4,962 |
The same control at the small budget. The baseline drops from 27 to 23 correct (59.5% → 53.0% judged), which is what makes 10 its better budget and the fair one to show it at. GraphRAG is flat (99.0% → 99.0%): its answers come from the traversal, so extra passages add tokens and no information.
| METRIC | Basic RAG3 chunks | GraphRAG3 chunks |
|---|---|---|
| Strict pass (exact set) | 23/50 | 49/50 |
| LLM judge | 53.0% | 99.0% |
| Tokens / question | 1,756 | 1,327 |
Unequal, in the baseline’s favour: it reads more than three times the context. This is the only pairing where the token columns differ, and that difference is the claim — GraphRAG answers 49/50 against 27/50 for 75% fewer tokens (1,327 vs 5,408 per question).
| METRIC | Basic RAG10 chunks | GraphRAG3 chunks |
|---|---|---|
| Strict pass (exact set) | 27/50 | 49/50 |
| LLM judge | 59.5% | 99.0% |
| Tokens / question | 5,408 | 1,327 |
The same 50 questions as the scorecard, same model and retrieval config, every answer scored by the same judge — so the tiers and the total are one measurement. The questions are identical in all four cells and only the context budget moves, so a difference can only be the budget or the question type. Two tiers are honest draws and the dropdown shows them. The All-50 row is the same measurement the scorecard above reports; the per-type rows are that run's own per-question verdicts, so a tier can differ from the total by the one answer whose verdict was reviewed.
| CONTEXT BUDGET | Basic RAGcross-encoder rerank | GraphRAGrouter + traversal | TOKEN CUTgraph vs basic |
|---|---|---|---|
| 10 chunksequal budget | 27/5060% judged · 5,408 tokens | 49/5099% judged · 4,962 tokens | −8.2%fewer tokens |
| 3 chunksequal budget | 23/5053% judged · 1,756 tokens | 49/5099% judged · 1,327 tokens | −24.4%fewer tokens |
| 10 vs 3 chunksGraphRAG on a third of the context | 27/5060% judged · 5,408 tokens | 49/5099% judged · 1,327 tokens | −75.5%fewer tokens |
At its own 3-chunk budget GraphRAG answers 49/50 on 1,327 tokens, against 27/50 on 5,408 for the baseline reading more than three times the context — 75% fewer tokens.
| CONTEXT BUDGET | Basic RAGcross-encoder rerank | GraphRAGrouter + traversal | TOKEN CUTgraph vs basic |
|---|---|---|---|
| 10 chunksequal budget | 10/10100% judged · 2,440 tokens | 10/10100% judged · 2,157 tokens | −11.6%fewer tokens |
| 3 chunksequal budget | 10/10100% judged · 622 tokens | 10/10100% judged · 558 tokens | −10.3%fewer tokens |
| 10 vs 3 chunksGraphRAG on a third of the context | 10/10100% judged · 2,440 tokens | 10/10100% judged · 558 tokens | −77.1%fewer tokens |
At its own 3-chunk budget GraphRAG answers 10/10 on 558 tokens, against 10/10 on 2,440 for the baseline reading more than three times the context — 77% fewer tokens.
| CONTEXT BUDGET | Basic RAGcross-encoder rerank | GraphRAGrouter + traversal | TOKEN CUTgraph vs basic |
|---|---|---|---|
| 10 chunksequal budget | 17/2085% judged · 6,351 tokens | 20/20100% judged · 5,857 tokens | −7.8%fewer tokens |
| 3 chunksequal budget | 13/2065% judged · 2,103 tokens | 20/20100% judged · 1,589 tokens | −24.4%fewer tokens |
| 10 vs 3 chunksGraphRAG on a third of the context | 17/2085% judged · 6,351 tokens | 20/20100% judged · 1,589 tokens | −75.0%fewer tokens |
At its own 3-chunk budget GraphRAG answers 20/20 on 1,589 tokens, against 17/20 on 6,351 for the baseline reading more than three times the context — 75% fewer tokens.
| CONTEXT BUDGET | Basic RAGcross-encoder rerank | GraphRAGrouter + traversal | TOKEN CUTgraph vs basic |
|---|---|---|---|
| 10 chunksequal budget | 0/2014% judged · 5,948 tokens | 19/2098% judged · 5,470 tokens | −8.0%fewer tokens |
| 3 chunksequal budget | 0/2017% judged · 1,976 tokens | 19/2098% judged · 1,449 tokens | −26.7%fewer tokens |
| 10 vs 3 chunksGraphRAG on a third of the context | 0/2014% judged · 5,948 tokens | 19/2098% judged · 1,449 tokens | −75.6%fewer tokens |
At its own 3-chunk budget GraphRAG answers 19/20 on 1,449 tokens, against 0/20 on 5,948 for the baseline reading more than three times the context — 76% fewer tokens.
The full working
PART means partially correct — some of the gold set returned, not all.
| ID | QUESTION TYPE | LLM ONLYNO RETRIEVAL | BASIC RAG10 CHUNKS | GRAPHRAG3 CHUNKS | QUESTION | GOLD | TOKENSBASIC RAG | TOKENSGRAPHRAG |
|---|---|---|---|---|---|---|---|---|
| 1 | single doc | FAIL | PASS | PASS | Who authored the document '2025 02 11 hosted api latency fallback activated'? | Ava Chen | 5,949 | 1,652 |
| 2 | single doc | FAIL | PASS | PASS | Who authored the document 'ENG 913579 attention path prefetcher and kernel winnower'? | Aisha Khan | 6,107 | 1,614 |
| 3 | single doc | FAIL | PASS | PASS | Who authored the document '20270509 lyra onboarding sig caiq intake'? | vivek_kulkarni | 6,588 | 1,673 |
| 4 | single doc | FAIL | PASS | PASS | Who authored the document '20270521 medbridge security subprocessor checkin'? | karthik_iyer | 6,629 | 1,818 |
| 5 | single doc | FAIL | FAIL | PASS | Who authored the document '20261121 slot swap proxy invite widget handoff'? | priya_natarajan | 6,609 | 1,798 |
| 6 | single doc | FAIL | PASS | PASS | Who authored the document 'ENG 411007 stratified hit ensemble ttl runbook'? | Miguel Alvarez | 5,617 | 1,132 |
| 7 | single doc | FAIL | PASS | PASS | Who authored the document '20270420 qbr runway stitch account onepagers'? | vanessa_ortiz | 6,903 | 1,753 |
| 8 | single doc | FAIL | PASS | PASS | Which Slack channel is the message '5390001111 tiny astronaut toast emoji echo' in? | random | 6,553 | 1,650 |
| 9 | single doc | FAIL | PASS | PASS | Which Slack channel is the message '1898765432 pin decay orchestration and grace unification' in? | product | 5,834 | 1,500 |
| 10 | single doc | FAIL | PASS | PASS | Which Slack channel is the message '3187654321 token identity wobble on embed proj sanity' in? | eng-ml | 6,975 | 1,659 |
| 11 | single doc | FAIL | PASS | PASS | Which Slack channel is the message '2176543210 kms rewrap edge cases and rbac clarify' in? | eng-security | 6,549 | 1,654 |
| 12 | single doc | FAIL | PASS | PASS | Which Slack channel is the message '2755550000 legal hold stale snapshot service account check' in? | eng-security | 6,118 | 1,617 |
| 13 | single doc | FAIL | PASS | PASS | Which Slack channel is the message '1950012345 shim routing decisionflow' in? | eng-platform | 6,420 | 1,746 |
| 14 | single doc | FAIL | PASS | PASS | Which Slack channel is the message '1789123456 sdr emea first week access' in? | sales | 6,363 | 1,551 |
| 15 | single doc | FAIL | PASS | PASS | Which Slack channel is the message '5698765432 marketplace edgebox gTM readiness' in? | partnerships | 6,355 | 1,598 |
| 16 | single doc | FAIL | FAIL | PASS | Which Slack channel is the message '1845678901 onprem handshake networking handoff' in? | sales | 6,325 | 1,180 |
| 17 | single doc | FAIL | PASS | PASS | Who authored the document '20280809 renewal term swap phased uplift redline rhythm'? | david_kim | 6,380 | 1,552 |
| 18 | single doc | FAIL | PASS | PASS | Which Slack channel is the message '1761112222 staged canary safety liftcheck' in? | eng-releases | 6,871 | 1,778 |
| 19 | single doc | FAIL | PASS | PASS | Which Slack channel is the message '1778881234' in? | incidents | 5,195 | 1,208 |
| 20 | single doc | FAIL | FAIL | PASS | Which Slack channel is the message '1719877777 int8 kernel branching cpu lull probe' in? | eng-runtime | 6,678 | 1,639 |
| 21 | directory lookup | FAIL | PASS | PASS | Who reports directly to Vikram Singh? | Emily Zhao, Robert King | 1,610 | 553 |
| 22 | directory lookup | FAIL | PASS | PASS | Who reports directly to Kimberly Park? | Aly Nguyen, Daniela Costa, Sergio Alvarez | 4,366 | 536 |
| 23 | directory lookup | FAIL | PASS | PASS | Who reports directly to Alex Martinez? | Elliot Price, Jean-Paul Laurent, Monica Patel, Renee Sullivan, Sonia Bhandari | 3,860 | 642 |
| 24 | directory lookup | FAIL | PASS | PASS | Who reports directly to Rafael Mendes? | Aisha Bello, Christine Wong, Jacob Stein, Tomáš Novák | 1,965 | 587 |
| 25 | directory lookup | FAIL | PASS | PASS | Who is Devon Price's manager? | Dylan Brooks | 1,415 | 567 |
| 26 | directory lookup | FAIL | PASS | PASS | Who is Olivia Grant's manager? | Karthik Iyer | 2,542 | 572 |
| 27 | directory lookup | FAIL | PASS | PASS | Who is Hassan Mirza's manager? | Karthik Iyer | 1,489 | 577 |
| 28 | directory lookup | PART | PASS | PASS | Which people are in the Operations / IT department? | David Kim, Natalie Chen | 1,800 | 501 |
| 29 | directory lookup | FAIL | PASS | PASS | Which people are in the Finance department? | Emily Stone, Jorge Ramirez, Laura Bennett, Rishi Malhotra | 1,397 | 532 |
| 30 | directory lookup | FAIL | PASS | PASS | Which people are in the People department? | Aly Nguyen, Daniela Costa, Kimberly Park, Sergio Alvarez | 3,961 | 516 |
| 31 | multi-hop | FAIL | FAIL | PASS | Which Slack channels contain documents involving both Allison Grant and Sergio Costa? | eng-oncall, eng-platform, eng-sre, postmortems | 5,065 | 649 |
| 32 | multi-hop | FAIL | FAIL | PASS | Which Slack channels contain documents involving both Ibrahim Khan and Wei Chen? | devex, eng-platform | 5,516 | 1,556 |
| 33 | multi-hop | FAIL | PART | PASS | Which Slack channels do the people reporting to Marcus Lin take part in? | eng-platform, eng-security, incidents | 6,156 | 1,513 |
| 34 | multi-hop | FAIL | FAIL | PART | Which people from the Product department take part in the #incidents Slack channel? | Marissa Cole, Vivek Kulkarni | 5,545 | 1,397 |
| 35 | multi-hop | FAIL | PART | PASS | Which Slack channels contain documents involving both Rafael Mendes and Victor Alvarez? | eng-infra, eng-sre, incidents | 6,860 | 1,633 |
| 36 | multi-hop | FAIL | FAIL | PASS | Which Slack channels contain documents involving both Daniel Carter and Naomi Feldman? | docs, sales | 4,617 | 1,761 |
| 37 | multi-hop | FAIL | FAIL | PASS | Which people from the Product department take part in the #product Slack channel? | Marissa Cole, Paula Mendes | 4,461 | 1,099 |
| 38 | multi-hop | FAIL | PART | PASS | Which Slack channels contain documents involving both Sean Gallagher and Vivek Kulkarni? | eng-releases, incidents | 7,270 | 1,672 |
| 39 | multi-hop | FAIL | PART | PASS | Which Slack channels contain documents involving both Dev Patel and Rafael Mendes? | announcements, eng-oncall, eng-releases, incidents, support | 6,862 | 1,617 |
| 40 | multi-hop | FAIL | PART | PASS | Which Slack channels do the people reporting to Sofia Ivanova take part in? | eng-infra, eng-runtime | 7,118 | 1,367 |
| 41 | multi-hop | FAIL | FAIL | PASS | Which Slack channels contain documents involving both Allison Grant and Ibrahim Khan? | devex, incidents | 4,075 | 1,582 |
| 42 | multi-hop | FAIL | PART | PASS | Which Slack channels contain documents involving both Aisha Bello and Mina Haddad? | eng-platform, eng-sre, incidents | 7,357 | 1,749 |
| 43 | multi-hop | FAIL | FAIL | PASS | Which Slack channels do the people reporting to Vikram Singh take part in? | eng-infra, eng-sre, incidents | 6,392 | 1,582 |
| 44 | multi-hop | FAIL | FAIL | PASS | Which people from the Product department take part in the #sales Slack channel? | Chris Osei, Naomi Feldman, Nikhil Sharma | 5,288 | 1,215 |
| 45 | multi-hop | FAIL | FAIL | PASS | Which Slack channels do the people reporting to Dr. Maya Srinivasan take part in? | eng-ml, eng-releases, product | 6,188 | 1,251 |
| 46 | multi-hop | FAIL | FAIL | PASS | Which people from the Infrastructure & SRE department take part in the #eng-runtime Slack channel? | Aisha Bello, Rafael Mendes, Sean Gallagher, Tomáš Novák | 6,702 | 1,769 |
| 47 | multi-hop | FAIL | PART | PASS | Which Slack channels contain documents involving both Markus Klein and Talia Benson? | eng-sre, product | 5,809 | 1,667 |
| 48 | multi-hop | FAIL | FAIL | PASS | Which Slack channels do the people reporting to Laura Bennett take part in? | design, incidents, sales | 6,007 | 1,741 |
| 49 | multi-hop | FAIL | FAIL | PASS | Which people from the Product department take part in the #announcements Slack channel? | Marissa Cole, Paula Mendes | 5,897 | 1,064 |
| 50 | multi-hop | FAIL | PART | PASS | Which Slack channels do the people reporting to Monica Patel take part in? | devex, eng-platform, incidents, new-hires, support | 5,777 | 1,087 |
results/enterprise/*.jsonl by demo/build_report.py at build time. None are typed by hand.