Overall performance

Ask a question. Three pipelines, side by side.

Three pipelines, the same question, side by side, each graded against a verified answer key. Every question and every verdict — wins and losses — is in the per-question tables further down.

Scorecard

SEC 10-K / DEF 14A filings · 50 questions · 100 S&P-100 companies. Real entities the model has already read — which is why LLM-only scores 56.6% here with no retrieval at all, against 0.8% on the fictional corpus.

METRICLLM onlyno retrievalBasic RAG10 chunksGraphRAGrouted
Tokens / question3414,8382,497
Accuracy / LLM judge56.6%65.5%83.0%
Strict pass (exact)16/5026/5035/50
Partly correct211411
Wrong (answered, 0%)13104
Declined / empty000
Tokens / question3414,8382,497

SEC 10-K / DEF 14A filings · 50 questions · 100 S&P-100 companies

Generalization to real filings.

A second corpus with none of the first one's properties: real companies the model has already read, extraction-built rather than metadata-built, numbers rather than name sets. Run under a corrected config — 8,192-token budget and no-refusal prompt applied to all arms, both of which raised the baseline too.

Data overview

The corpus

What it isReal filings from 100 S&P-100 companies. The model has read these, which is visible in the numbers: the no-retrieval arm scores 57% here against 0.8% on the fictional corpus.
Documents40010-K ×100 (financials, risk, competitors) · DEF 14A ×100 (auditor, board, governance) · 8-K ×200 (appointments, dividends, buybacks, acquisitions)
Size87.0M characters ≈ 21.7M tokensChunked into 61,777 section-aware chunks
Built fromTwo-phase LLM extraction, then canonicalisationThe expensive path — see the architecture diagram above. This is what the enterprise build avoided.
Questions50 held out — A 8 single-lookup, B 16 two-hop bridge, C 20 aggregation, D 6 cross-docTuned on a separate 24-question dev set, so the 50 were never developed against.
Answer shapeNumbers and dates, not name setsSo set-F1 does not transfer; correctness here is the blind judge's call and strict means it awarded 100.

Equal budget — the same question set at matched chunk counts

Both arms already share one hard cap (10 items × 2,000 characters). What differs is how much of it each fills: the baseline reads its full allowance on every question, while GraphRAG's routes ask for 4–6 and its aggregation route retrieves typed graph edges instead of chunks. The first two rows force every vector request to the baseline's k, so the chunk-using routes genuinely read the same number. Use the dropdown — the saving is concentrated in one question type.

CONTEXT BUDGETBasic RAGcosine top-kGraphRAGrouter + traversalTOKEN CUTgraph vs basic
10 vs 10 chunksequal budget67% judged26/50 · 5,120 tokens83% judged35/50 · 3,599 tokens−29.7%fewer tokens
6 vs 6 chunksequal budget63% judged25/50 · 3,327 tokens81% judged33/50 · 2,502 tokens−24.8%fewer tokens
10 vs routedas shipped66% judged26/50 · 4,838 tokens83% judged35/50 · 2,497 tokens−48.4%fewer tokens

At a matched 10-chunk budget GraphRAG spends 3,599 tokens against 5,120 — 30% fewer.

The saving is aggregation, and it survives every equalisation
1,248 tokens against 4,680 on the 22 aggregation questions — 73% fewer. Forcing chunks does not touch it: this route answers from typed edges and sends one provenance sentence each, so there is no chunk budget to equalise. It is the same shape a top-k retriever structurally cannot do — gather every company matching a property before the answer exists.
On the other routes GraphRAG is level or dearer, and the grid shows it
Lookup, bridge, compare and temporal are not token wins at equal chunks. At 10 v 10 lookup is within 5% of the baseline and compare is 17% worse; at 6 v 6 bridge is 13% worse. Those routes fetch chunks like the baseline does and then add a graph hop on top, so there is nothing to compress. The set-wide figure is an average over one route that wins heavily and four that do not.
Why the set-wide number falls as the budget falls
48% as shipped, 30% at 10 v 10, 25% at 6 v 6. Cutting the budget makes the baseline cheaper while aggregation hardly moves, so the gap narrows even though the ratio on aggregation holds. Part of the shipped figure is GraphRAG reading fewer chunks; the part that survives matched budgets is the compression.

Run it yourself

Query console — query specific

Query console

Every question below has a verified answer key, so the console grades each reply rather than leaving you to judge it. With a backend configured it runs all three pipelines live; without one it replays the recorded run — the same measured answer either way, and the card says recorded when that is what happened.
Fairness controls — what is held identical across the three pipelines
One generator, one setting. All three arms call the same model (gemini-3.7-flash) at temperature 0, with the same system prompt and the same 8,192-token output budget. Nothing is sampled, so a rerun moves only with the retrieval, never with the decoder.

One retriever, one budget. Basic RAG and GraphRAG share the embedder (BAAI/bge-small-en-v1.5) and the same hard context cap — 10 passages × 2,000 characters, enforced on both. That cap is what makes the token comparison mean anything: GraphRAG is not given less room, it simply does not fill the room it has.

And the asymmetry, stated rather than buried. On the enterprise corpus the baseline is the reranked one — a cross-encoder over a 50-candidate pool, which is a stronger baseline than plain top-k and is deliberately the one shown. On the SEC corpus neither arm reranks: the reranked variant was ablation-tested against plain cosine and lost on strict pass, graded quality and BERTScore at once, so it was dropped. Either way the baseline is shown at its best measured configuration, not at a convenient one.
The baseline retrieves through TigerGraph too
The baseline retrieves through TigerGraph too. On the enterprise corpus both arms use the same vector store, the same embedder and the same corpus, so the only difference between them is whether the graph is consulted. And the baseline is the reranked one — a cross-encoder over a 50-candidate pool, which took it from 0.30 to 0.51 set-F1 at the same token cost. Beating a weak baseline would prove nothing.
Two scoring changes, applied to all three arms — and every arm gained
Two scoring changes, applied to all three arms. First, the rubric was corrected: it had been penalising answers for omitting derived detail — gold written as "$57,048M (~$57.0B), down 2% YoY" against an answer of "$57,048 million", which is that fact exactly. The rubric now treats the same number in another unit as identical, a ticker as a company name, and citations as ignorable. It still does not forgive a missing fact. Every arm gained and the weakest gained most — LLM-only +3.6, Basic RAG +2.1, GraphRAG +0.9 — so the gap over the baseline narrowed, which is the evidence the change was not bent in our favour. A third, more lenient rubric was built and rejected: it scored 100 to answers reciting real figures from the wrong fiscal year.
Contamination is visible here and absent on the enterprise set
Contamination is visible here and absent on the enterprise set. With no retrieval at all, LLM-only scores 57% on real filings and 0.8% on the fictional corpus. The model has read these filings. That is why the enterprise result is the one to trust — and why the leak gate matters.

How it works

Two builds, one comparison harness.

The diagram below is the SEC round: filings chunked, entities and relations extracted by an LLM in two passes, canonicalised into a curated graph, then three pipelines judged blind. The enterprise build reuses the harness and replaces the expensive half — see the note under the diagram.

End-to-end architecture: 400 SEC filings, two-phase Gemini 3.7 Flash-Lite extraction and canonicalisation into a curated knowledge graph, TigerGraph Savanna as graph and vector store, the three pipelines with the GraphRAG router and its four routes, and the blind judging layer.
Round 3 / SEC pipeline.

The full working

Every question, every verdict.

PART means partially correct — some of the gold set returned, not all.

IDROUTELLM ONLYNO RETRIEVALBASIC RAG10 CHUNKSGRAPHRAGROUTEDQUESTIONGOLDTOKENSBASIC RAGTOKENSGRAPHRAG
1aggregationPASSFAILPASSBesides Netflix, which company authorized a $25B share-repurchase program in 2026?Adobe5,3511,364
2aggregationPASSPASSPASSAMD names which two companies as its principal competitors in Data Center and discrete graphics?Intel and Nvidia4,040549
3aggregationPASSFAILPARTWhich Deloitte-audited company had a fiscal year ending June 30, 2025?Microsoft4,6991,083
4aggregationFAILFAILPASSAmong the companies that name TSMC as a foundry, which one had the highest fiscal-year revenue?NVIDIA ($215.9B; vs AMD, AVGO, INTC, QCOM)4,262460
5aggregationPARTPARTPARTList all S&P-100 companies in the dataset audited by KPMG.ACN, ADBE, BNY, COST, EMR, GD, HD, PEP, PFE, V, WFC (11)4,3541,270
6aggregationPASSPARTPASSWhich companies in the dataset name TSMC as a foundry/manufacturing partner?AMD, AVGO, INTC, NVDA, QCOM3,967465
7aggregationPARTPASSPASSTwo companies list Indra Nooyi as a member of their board — which two?Amazon (AMZN), Honeywell (HON)4,822358
8aggregationPASSPASSPASSEllen Kullman serves on the boards of which two dataset companies?Amgen (AMGN), Goldman Sachs (GS)4,561355
9aggregationPASSPASSPASSF. William McNabb III is a director at which two dataset companies?IBM, UnitedHealth (UNH)4,999630
10aggregationPASSPARTPASSWhich Energy-sector companies discuss climate change as a risk factor in their 10-K?Chevron (CVX), ExxonMobil (XOM), ConocoPhillips (COP)3,7141,139
11aggregationPARTPARTPASSWhich Utilities-sector companies address climate change in their 10-K?Duke (DUK), NextEra (NEE), Southern (SO)4,0041,003
12aggregationPARTPARTPARTWhich Health-Care companies mention weight-loss / obesity (GLP-1) drugs in their 10-K?ABBV, AMGN, CVS, ISRG, LLY, MDT, MRK, PFE ‡5,234965
13aggregationPARTPARTFAILBesides health-care companies, which consumer companies flag GLP-1 / weight-loss drugs as a business risk?KO, MCD, MDLZ, PEP, SBUX ‡3,872867
14aggregationPARTPARTPARTWhich companies declared a cash dividend via an 8-K in this dataset?AIG ($0.50), Caterpillar ($1.63), Costco ($1.47), Oracle ($1,625 pref.+common), P&G ‡4,798992
15aggregationPARTPARTPASSWhich companies authorized a $25 billion share-repurchase program via an 8-K?Adobe, Netflix4,532479
16aggregationPARTPARTPASSWhich KPMG-audited companies are in the Consumer Staples sector?PepsiCo (PEP), Costco (COST)4,023539
17aggregationPARTPASSPARTWhich KPMG-audited companies are in the Financials sector?BNY Mellon (BNY), Wells Fargo (WFC); + Visa (V) if classed Financials3,967689
18aggregationPARTPARTPASSAmong the companies that name TSMC as a foundry, which are audited by Ernst & Young?AMD, INTC3,860559
19aggregationPASSFAILPASSAmong the companies that name TSMC as a foundry, which are audited by PricewaterhouseCoopers?AVGO, NVDA, QCOM4,173755
20aggregationPARTPARTPASSAmong Health-Care companies that mention obesity/weight-loss drugs, which are audited by Ernst & Young?ABBV, AMGN, CVS, LLY4,4311,075
21aggregationPARTPARTPASSWhich Deloitte-audited companies are in the Health-Care sector?Bristol-Myers Squibb (BMY), UnitedHealth (UNH)4,181561
22aggregationPARTPARTPARTAmong the dataset's semiconductor and networking companies, which explicitly name NVIDIA as a competitor?AMD, CSCO, INTC, QCOM3,848626
23aggregationPARTFAILPASSAmong companies that name NVIDIA as a competitor, which are audited by PricewaterhouseCoopers?CSCO, QCOM4,548738
24aggregationFAILFAILFAILHow many dataset companies does each Big Four firm audit?Ernst & Young 35, PwC 29, Deloitte 25, KPMG 115,9406,297
25bridgeFAILPASSPASSThe company that appointed John Ternus as CEO — what were its total net sales in FY2025?Apple; $416,161M (~$416.2B)5,9985,145
26bridgePASSFAILPASSThe company that agreed to acquire Apogee Therapeutics — who is its independent auditor?AbbVie; Ernst & Young LLP4,1194,979
27bridgeFAILFAILPARTThe company that appointed Jay Hoag as Chairman in 2026 — what share-repurchase amount did it authorize that year?Netflix; $25 billion4,6075,750
28bridgePARTPASSPASSThe company that expanded a productivity/restructuring program on April 30, 2026 — name it, its auditor, and the revised total pre-tax charge range.Colgate-Palmolive; PwC; revised to $350M–$550M4,9485,202
29lookupFAILPASSPASSApple's total net sales for fiscal year 2025?$416,161M (~$416.2B)6,7274,168
30lookupFAILPASSPASSNVIDIA's total revenue for fiscal year 2026?$215,938M (~$215.9B)5,7243,500
31lookupFAILPASSPASSJPMorgan Chase's net income for FY ended Dec 31, 2025?$57,048M (~$57.0B), down 2% YoY6,9243,567
32lookupFAILPASSFAILExxonMobil's total revenues and other income for FY2025?$332,238M (~$332.2B)7,1554,227
33lookupFAILPASSPASSCoca-Cola's net operating revenues for FY2025?$47,941M (~$47.9B)6,3734,347
34lookupPASSPASSPASSOn what date does Microsoft's FY2025 end?June 30, 20254,8803,204
35lookupPASSPASSPASSWho is Amazon's Chief Executive Officer?Andrew R. Jassy4,6722,856
36lookupPASSPASSPASSNetflix relies on which cloud provider to operate its streaming service?Amazon Web Services (AWS)3,9732,404
37lookupFAILFAILFAILCompany that formed a 50/50 joint venture with BT Group plc — who is its auditor?Verizon; Ernst & Young LLP4,4242,562
38lookupPARTPASSPASSAmgen named an incoming CFO in 2026 — who is he, and who is Amgen's auditor?Thomas Dittrich; Ernst & Young LLP4,8362,812
39lookupPASSPASSPASSChevron's departing Chief Legal Officer (per its 2026 8-K) is whom, and who is Chevron's auditor?R. Hewitt Pate; PricewaterhouseCoopers LLP4,7752,821
40lookupPASSPASSPASSIn Apple's April 2026 8-K, who was appointed CEO, and what role will Tim Cook move into?John Ternus appointed CEO; Tim Cook moves to Executive Chair of the Board5,2083,148
41lookupPASSPASSPASSAMD's new revolving credit facility (May 2026 8-K) names which bank as administrative agent, and which was agent on the facility it replaced?New: JPMorgan Chase Bank, N.A.; prior (Apr 2022): Wells Fargo Bank, N.A.5,0163,096
42lookupFAILPASSPASSBy how much did Caterpillar raise its quarterly dividend (June 2026 8-K), and to what new amount?+$0.12, from $1.51 to $1.63/share4,3282,794
43lookupPARTPARTPARTCisco's May 2026 restructuring 8-K funds which growth areas, and who is Cisco's auditor?Silicon, optics, security, and AI; PwC5,3163,674
44lookupFAILPASSPASSWhat quarterly cash dividend per share did Costco declare in its July 2026 8-K?$1.47 per share5,4802,622
45comparePARTPASSPASSWhich had the higher fiscal-year net income: JPMorgan Chase or ExxonMobil?JPMorgan Chase ($57.0B vs ExxonMobil $28.8B)6,1352,130
46comparePASSPASSPASSWhich had the higher fiscal-year revenue: Walmart or Apple?Walmart ($713.2B vs Apple $416.2B)5,1067,035
47comparePARTPASSPASSWhich spent more on R&D in its latest fiscal year: Apple or NVIDIA?Apple ($34.55B vs NVIDIA $18.50B)4,7785,372
48temporalPARTPASSPARTAmgen's CFO transition (2026): when does the outgoing CFO retire, and when does the incoming CFO start?Peter H. Griffith retires as CFO Aug 31, 2026 (EVP until Jan 31, 2027); Thomas Dittrich effective Sept 1, 20264,5604,612
49temporalFAILFAILPARTPut these 2026 buyback authorizations in chronological order: Adobe $25B, Netflix +$25B, Accenture +$2B, Morgan Stanley $20B.Adobe (Apr 21) → Netflix (Apr 22) → Accenture (Jun 23) → Morgan Stanley (Jun 24)4,8525,013
50temporalPARTPASSPARTApple filed two 8-Ks in April 2026 (Apr 20, Apr 30). What event does each report?Apr 20 = CEO transition (Ternus); Apr 30 = Q2 FY26 earnings / results of operations4,7904,015
‡ marks a gold answer the dataset authors flagged as non-exhaustive — the listed companies are correct but the set may be incomplete, so an answer naming an unlisted-but-valid company is not necessarily wrong. Three questions carry it (EQ32, EQ33, EQ34) and all three are scored like the rest, which means those three understate every arm.
Every figure on this page is computed from results/enterprise/*.jsonl by demo/build_report.py at build time. None are typed by hand.