Mandip Adhikari
← Index
ProjectMar 2026 — Oct 2026

SearchProbe

SearchProbe turns the claim that neural search ignores logical operators into a measurement. Every returned page is fetched and judged against each constraint in the query, and results come out as rank-aware scorecards with bootstrap confidence intervals — plus metamorphic twin tests that need no ground truth at all.

Python · Typer · Claude (judge) · Exa · Tavily · Brave · SerpAPI · SciPy · sentence-transformers · Streamlit · Modal · SQLite · Wikidata SPARQL

v3.0.0 · Python 3.11

321
typed test cases (250 base + 71 minimal-pair twins)
48–53%
open-retriever pairwise accuracy (chance is 50%)
331
tests, no network or API keys needed

The problem

Queries like “European countries that are not members of the EU” hinge on an operator. Embedding-based search tends to return the near-miss — France — because it mentions everything in the query.

Judging whole result lists as good or bad hides that failure. So SearchProbe judges each returned document against each typed constraint: exclusion, numeric range, date window, language, set membership.

Pipeline

Cases follow a versioned schema: 186 generated from Wikidata with answer and negative sets from SPARQL, 105 structured language and date cases, and 30 hand-written seeds. The runner sends every case to every system, with a cost tracker that reserves budget before dispatch.

Each URL is fetched once and reduced to main text, date and language. Mechanical constraints are decided deterministically; the rest go to an LLM judge. Judgments are cached by case, content hash, judge and prompt version, so scorecards recompute offline.

┌──────────┐   ┌──────────────┐   ┌────────────┐
│ cases    │──▶│ runner       │──▶│ providers  │
│ 321 JSONL│   │ + cost guard │   │ exa tavily │
└──────────┘   └──────────────┘   │ brave serp │
                                  └─────┬──────┘
                                        ▼
┌──────────────┐   ┌─────────────┐   ┌─────────────┐
│ metrics      │◀──│ judging     │◀──│ evidence    │
│ violation@k  │   │ checkers ─▶ │   │ fetch once  │
│ nDCG MRR  CI │   │ llm judge   │   │ text/date   │
└──────────────┘   └─────────────┘   └─────────────┘

Statistics

Scorecards report violation@k, all-constraints-satisfied@k, pooled nDCG@10 and MRR, each with case-paired bootstrap CIs and Benjamini–Hochberg correction across comparisons.

The Operator Blindness Index measures how much results move between a query and its twin — negation removed, a number ×10, a year shifted — normalized by rank-biased overlap against the engine's own re-query noise.

What it found

On 203 Wikidata triplets, none of the seven open scorers tested — BM25, MiniLM, BGE-small, E5-base, Qwen3-Embedding-0.6B and two cross-encoder rerankers — beat chance overall. On “member of X but not Y”, every model scored 16–34%, below chance.

An earlier headline — 70% negation accuracy for the BGE reranker — didn't replicate after fixing Wikidata membership errors. The correction is written up in the repo rather than quietly dropped.