IntentForge v2

Privacy-first, intent-driven search engine with zero-trust architecture.

IntentForge is a high-performance discovery platform that combines BM25 keyword matching with ONNX-powered semantic search to deliver intent-aligned results. Built with privacy at its core — no tracking, no data harvesting, no corporate dependencies.

Current Capabilities

🔍 Core Search

• Hybrid Search: Combines BM25 keyword matching with semantic vector search (ONNX embeddings)
• Intent Classification: Detects Navigational, Informational, Transactional, and Exploratory queries with adaptive semantic ratios
• Multi-tier Caching: L1 in-memory cache + Redis for sub-millisecond repeated queries
• Meta-search Aggregation: Simultaneously queries 9+ providers (Brave, Google, Wikipedia, Arxiv, Reddit, GitHub, Hacker News, Medium, DuckDuckGo)

🖼️ Image Search

• Zero-bandwidth indexing: Extracts context from HTML metadata (alt, title, captions, surrounding text) without downloading images
• Perceptual hashing: dHash (64-bit) for deduplication
• Visual fingerprints: ThumbHash for compact (~20-30 byte) thumbnail representation
• Quality gates: Filters tracking pixels, SVGs, and 1x1 placeholders

📹 Video Search

• Multi-source discovery: YouTube Unified, Piped, Invidious, Dailymotion, Vimeo, Internet Archive
• Intent-first scoring: Ranks by intent match, relevance, and quality signals
• Privacy-friendly frontends: Prefers Piped/Invidious over direct YouTube API

🛡️ Privacy & Security

• Anti-detection: TLS fingerprinting, randomized timing, viewport spoofing to bypass AI-generated content detection
• Tor integration: Optional route through Tor for anonymity
• No tracking: No cookies, no user profiling, no data retention
• Cloudflare bypass: Automated CAPTCHA solving for accessibility

📚 Content Extraction

• Trafilatura integration: Boilerplate removal, readable text extraction
• Multi-format support: Articles, documentation, technical content
• RSS firehose: 80+ curated sources for continuous content discovery

🚀 Performance

• Cross-encoder reranking: ms-marco-MiniLM-L6-v2 for precision reordering
• Adaptive semantic ratio: Query-type-aware blend of keyword vs semantic search
• Self-improvement: Automatic gap analysis triggers background crawling for weak results
• Common Crawl integration: Massive URL discovery from web archives

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                        IntentForge Core (Rust)                  │
├──────────────┬──────────────┬───────────────┬───────────────────┤
│ Intent      │   Ranking    │  Discovery    │  Anti-Detection   │
│ Classifier  │   Engine     │  (Firehose)   │  + Cloudflare     │
├──────────────┴──────────────┴───────────────┴───────────────────┤
│                    Meilisearch Index (Hybrid)                   │
├─────────────────────────────────────────────────────────────────┤
│                 Query Layer (Python + FastAPI)                 │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────────────┐ │
│  │  ONNX Embed  │  │ Cross-Encoder│  │  Multi-tier Cache     │ │
│  │  (all-MiniLM)│  │  Reranker    │  │  L1 + Redis          │ │
│  └──────────────┘  └──────────────┘  └──────────────────────┘ │
├─────────────────────────────────────────────────────────────────┤
│                      Meta-Search API (Rust)                     │
│  Brave · Google · Wikipedia · Arxiv · Reddit · GitHub · HN     │
├─────────────────────────────────────────────────────────────────┤
│                    Video Services (Node.js)                     │
│           YouTube Unified · Piped · Invidious                  │
└─────────────────────────────────────────────────────────────────┘

API Endpoints

Stack

Privacy Principles

1. Zero data retention — No search logs, no analytics, no cookies
2. No corporate dependencies — All sources are open or privacy-respecting
3. Encrypted transport — HTTPS everywhere, Tor optional
4. No AI content detection fingerprinting — Built-in bypass
5. Open source — Full transparency on all components

Getting Started

# Full release build
cargo build --release

# Run API server (port 9100)
cargo run --release

# Docker full stack
docker-compose -f docker-compose.dev.yml up -d --build

Project Status

IntentForge v2 is operational and actively developed. The core search, image indexing, and video discovery systems are functional. Ongoing work focuses on latency optimization, expanded source coverage, and enhanced personalization.

See docs/ROADMAP.md for planned improvements.