How we fixed search failures with negative-aware alternatives expansion

TL;DR: When someone searches for “python framework not django”, our system used to strip the excluded term but then accidentally promote it again in retry queries. The fix was to make both the intent engine and gateway truly negation-aware — and to stop hard-coding filters in favor of structural content analysis. Calibration data and commit-level diffs below.

---

The failure mode: “not X” still returned X pages

IntentForge v2 has two layers that shape a query before it reaches search backends:

1. Intent engine — classifies intent and expands the query.
2. Gateway — strips negative constraints, builds alternative-listing retries, and reranks.

For negation-heavy queries, the interaction between these layers created a feedback loop:

• The gateway would strip “not django” and retry with “django alternatives”.
• The intent engine would re-add “django” through topic-word extraction.
• The retry fan-out to SearXNG would return Django documentation instead of comparison pages.

The result: a user explicitly asking for alternatives still saw the excluded product at the top.

---

The fix: two commits, three coordinated changes

The repair landed in two tightly scoped commits that touched the same two components:

• e848276 — “negative-aware alternatives expansion for bypass search failures”
• 3396e4f — “comprehensive negation handling for alternative-seeking queries”

1. Strip negated content terms from the retry query

Before, the alternatives query was built from a stripped_query that only removed negation triggers:

"python framework not django" → "python framework django"

After e848276, the intent engine now filters out both the trigger and the negated content term:

let neg_terms: HashSet<&str> = structured_constraints.negative.iter()
    .flat_map(|n| n.split_whitespace())
    .collect();
let clean_words: Vec<&str> = s_q.split_whitespace()
    .filter(|w| !neg_terms.contains(w))
    .collect();

And in 3396e4f, simple_negation_strip became trigger + next-word aware:

"browser not chrome not edge" → "browser"

2. Generate cleaner alternative-seeking expansions

The intent engine now inserts a negation-stripped core query into the expansion set and caps duplicate topic words (orm orm → orm):

let has_alt_concept = expansions.iter().any(|e| {
    let e_low = e.to_lowercase();
    e_low.contains("alternative") || e_low.contains("instead of")
        || e_low.contains("replacement") || e_low.contains("competitor")
});
expansions.push(format!("{} alternatives", alt_text));

This removes the earlier arbitrary cap of 2 expansions so alternative-listing pages survive alongside intent-specific variants.

3. Relax hard penalties for legitimate alt-listing pages

Once the query is correct, the retry URLs are cleaner. But if alt-listing pages still mention excluded terms in their titles, the reranker shouldn’t crush them. The gateway now:

• Skips the hard title penalty when is_alternative_listing_page is strong (alt > 0.6).
• Applies a mild penalty for moderate alt pages (0.50) instead of 0.20.
• Preserves the hard penalty only for genuinely off-topic negative hits.

This is a scoring discipline fix, not a ranking hack.

---

From hardcoded lists to structural analysis

A second thread in the same workstream replaced hardcoded domain/keyword filtering lists with structural URL / content analysis (a0def8e). That matters because:

• Allow-lists age poorly.
• Keyword lists miss new content shapes.
• Structural signals — path patterns, section headings, term density — generalize better.

The new topic-coherence gate applies to both local and web results. Distinctive query terms are extracted, and if a result contains neither positive nor negative constraint terms, it is progressively demoted based on source tier. That closes an entire class of off-topic leakage without maintaining a blacklist.

---

Calibration snapshot

We run a 200-query calibration benchmark (calibration_benchmark_200.csv) covering navigational, informational, how-to, transactional, comparison, and fresh intents. Two patterns matter most for this fix:

Navigational misclassifications with technical fallback

Query	Expected	Predicted	Correct	Confidence	Time
docker hub	navigational	technical	❌	0.75	2007 ms
aws console	navigational	technical	❌	0.75	3027 ms
vscode marketplace	navigational	transactional	❌	0.9616	5 ms
github login	navigational	navigational	✅	0.85	3024 ms

High-confidence informational queries

Query	Expected	Predicted	Correct	Confidence	Time
how does photosynthesis work	informational	informational	✅	0.9374	6 ms
what is machine learning	informational	informational	✅	0.8	4 ms
define api	informational	informational	✅	0.8	6 ms

The remaining failure mode is not negation-specific, but the same post-processing layer will eventually absorb it. What this data proves: the negation fixes above do not regress clean informational coverage.

---

Infrastructure context

The gateway runs as a Rust service on top of a Traefik front-end. SearXNG instances are split between a VPN-backed node and a Tor node. 0d75d06 fixed early timeout behavior so the negation-aware retries don’t double the tail latency under load.

---

Why this approach scales

Negation handling in search is usually solved with per-case filters or hard-coded exceptions. What we shipped instead:

• Constraint-based stripping at both engine and gateway boundaries.
• Alternatives expansion that tracks intent semantic, not just term presence.
• Graduated penalties for alt-listing pages instead of binary removal.
• Topic coherence gating backed by structural content analysis.

That combination keeps the system honest about what the user explicitly excluded while still rewarding legitimate comparison content.

---

Related: IntentForge: How We Built a Privacy-First Search Engine on Tor (2026-04-22) — the architecture overview that preceded these fixes.