QueryData
Google's QueryData API turns the question into SQL, guided by a tuned context set of query templates, facets and value searches, and scoped to the tables each agent needs.
Ostrander Pump & Valve Co. does not exist. Its people, customers, suppliers, products, data and documents are all synthetic, created for demos and prototyping. Every image on this site is AI-generated.
Ostrander Pump & Valve is a 52-year-old pump and valve maker from Rockford, Illinois, that never existed. I built it, along with five generations of its business systems, to find out how different ways of building AI data agents hold up against the kind of database real companies actually have.
The company's recent years are simulated day by day, through October 2026: orders, shipments, invoices, payroll, machine telemetry, warranty claims and the rest, on top of thinner history going back further. The result is one PostgreSQL database with the scars of every system that ever wrote to it. That database is the test bed.
Rockford, Ill. · Synthetic data estate
Est. 1974 · Fictional
Why build a fake company
Most demos of AI agents that answer questions from a database use a dozen clean, well-named tables. Real company databases grow by accretion: a system from the 1980s, a port that kept its six-character names, a web app bolted on later, an acquisition that never got fully merged. Questions that sound simple, like "what did we ship to this customer last year?", cross three of them.
Real company data can't be published or shared, so I made a company instead, with a history that explains why its data looks the way it does. Because I generated it, I also know the right answer to every question, which makes agents' answers gradable.
CYYMMDD integersEvery trap is catalogued, and the true answers are kept in a separate answer key that agents can't see.
What is being tested
The benchmark asks when each approach is the right one: a managed natural-language query service, hand-built tools, or an agent that writes its own SQL from catalog metadata. Each runs as six business agents (customer service, production supervisor, purchasing, finance, operations and quality, and sales) on operational and analytical tasks, scored for accuracy, latency and cost.
Google's QueryData API turns the question into SQL, guided by a tuned context set of query templates, facets and value searches, and scoped to the tables each agent needs.
Hand-designed, parameterized SQL tools served by MCP Toolbox. The agent picks a tool and fills in its arguments; it never writes SQL.
The same tools, with a guarded execute_sql escape hatch for questions the tools don't cover.
No custom tools. The agent searches Knowledge Catalog for table descriptions, a business glossary and document passages, then writes its own SQL.
All four share the same model, the same simulated users, graders and scenarios. Each approach is built with full knowledge of the data, so the test measures how well it works in competent hands, not how easy the data is to figure out.
The company's paperwork
OPV has department manuals, a systems history from IT, memos about the Kessler acquisition, an annual report, a product catalog and a quarterly employee newsletter, The Impeller. Figures in them come from the simulated data, and the people named in the newsletters are rows in the employee table. Some passages are out of date on purpose, as real manuals are.
For invited testers
Lookup screens for orders, the parts catalog, the shop floor, support cases, CRM and finance, a shared drive with the company's documents, and OPVChat, which puts the four agents side by side on the same question.
Sign-in is by invitation only, with a Google account that has been added to the test group.