SearchProbe
SearchProbe turns the claim that neural search ignores logical operators into a measurement. Every returned page is fetched and judged against each constraint in the query, and results come out as rank-aware scorecards with bootstrap confidence intervals — plus metamorphic twin tests that need no ground truth at all.
Python · Typer · Claude (judge) · Exa · Tavily · Brave · SerpAPI · SciPy · sentence-transformers · Streamlit · Modal · SQLite · Wikidata SPARQL
v3.0.0 · Python 3.11
- 321
- typed test cases (250 base + 71 minimal-pair twins)
- 48–53%
- open-retriever pairwise accuracy (chance is 50%)
- 331
- tests, no network or API keys needed
The problem
Queries like “European countries that are not members of the EU” hinge on an operator. Embedding-based search tends to return the near-miss — France — because it mentions everything in the query.
Judging whole result lists as good or bad hides that failure. So SearchProbe judges each returned document against each typed constraint: exclusion, numeric range, date window, language, set membership.
Pipeline
Cases follow a versioned schema: 186 generated from Wikidata with answer and negative sets from SPARQL, 105 structured language and date cases, and 30 hand-written seeds. The runner sends every case to every system, with a cost tracker that reserves budget before dispatch.
Each URL is fetched once and reduced to main text, date and language. Mechanical constraints are decided deterministically; the rest go to an LLM judge. Judgments are cached by case, content hash, judge and prompt version, so scorecards recompute offline.
┌──────────┐ ┌──────────────┐ ┌────────────┐
│ cases │──▶│ runner │──▶│ providers │
│ 321 JSONL│ │ + cost guard │ │ exa tavily │
└──────────┘ └──────────────┘ │ brave serp │
└─────┬──────┘
▼
┌──────────────┐ ┌─────────────┐ ┌─────────────┐
│ metrics │◀──│ judging │◀──│ evidence │
│ violation@k │ │ checkers ─▶ │ │ fetch once │
│ nDCG MRR CI │ │ llm judge │ │ text/date │
└──────────────┘ └─────────────┘ └─────────────┘┌──────────┐ ┌──────────────┐ ┌────────────┐
│ cases │──▶│ runner │──▶│ providers │
│ 321 JSONL│ │ + cost guard │ │ exa tavily │
└──────────┘ └──────────────┘ │ brave serp │
└─────┬──────┘
▼
┌──────────────┐ ┌─────────────┐ ┌─────────────┐
│ metrics │◀──│ judging │◀──│ evidence │
│ violation@k │ │ checkers ─▶ │ │ fetch once │
│ nDCG MRR CI │ │ llm judge │ │ text/date │
└──────────────┘ └─────────────┘ └─────────────┘Statistics
Scorecards report violation@k, all-constraints-satisfied@k, pooled nDCG@10 and MRR, each with case-paired bootstrap CIs and Benjamini–Hochberg correction across comparisons.
The Operator Blindness Index measures how much results move between a query and its twin — negation removed, a number ×10, a year shifted — normalized by rank-biased overlap against the engine's own re-query noise.
What it found
On 203 Wikidata triplets, none of the seven open scorers tested — BM25, MiniLM, BGE-small, E5-base, Qwen3-Embedding-0.6B and two cross-encoder rerankers — beat chance overall. On “member of X but not Y”, every model scored 16–34%, below chance.
An earlier headline — 70% negation accuracy for the BGE reranker — didn't replicate after fixing Wikidata membership errors. The correction is written up in the repo rather than quietly dropped.