nafi@portfolio:~$
bash — ~/projects/spyder
$cd ~/projects/spyder && cat summary.txt
Keyless, LLM-free research pipeline that searches five engines in parallel, cross-references sources, and produces structured cited reports. TF-IDF ranking implemented from scratch. BibTeX/APA/MLA export, PDF generation, zero-dependency stdlib web UI with history and permalinks.

SPYDER

Backend Developer · #d914aa2 research
Python · TF-IDF · DuckDuckGo/ArXiv/PubMed/Semantic Scholar APIs github.com/sleepyhead000/spyder
~/projects/spyder $

cat README.md

README.md
$cat overview.txt
Spyder is a zero-API-key research agent that queries multiple scholarly and web sources simultaneously, deduplicates and clusters findings, ranks them via TF-IDF (implemented from scratch — no sklearn), detects thematic clusters and statistical contradictions across sources, and exports structured reports with proper citations. Entirely local, no LLM calls, no external keys required.
$cat sources.txt
Five retrieval engines queried in parallel via asyncio: DuckDuckGo HTML scrape (web), ArXiv API (preprints), PubMed/Entrez (biomedical), Semantic Scholar API (CS/ML papers), Crossref (DOI metadata). Each source has a dedicated adapter implementing a common SearchProvider interface. Rate limits respected per provider; results merged by normalized title+author fingerprint.
$cat ranking.txt
TF-IDF from scratch: builds document-term matrix from abstracts/snippets, computes inverse document frequency across the merged corpus, scores each finding by cosine similarity to the query vector. Thematic clustering via k-means on TF-IDF vectors (k=auto via silhouette). Contradiction detection: pairwise cosine distance > threshold + opposing sentiment lexicon matches.
$cat exports.txt
Report formats: BibTeX (for LaTeX), APA 7th, MLA 9th, RIS, CSL-JSON. PDF generation via reportlab (pure Python). Web UI: single-file stdlib HTTP server (http.server) serving Jinja2 templates — history persisted in SQLite, permalinks via content hash. Zero npm, zero build step, runs with `python -m spyder.web`.
~/projects/spyder $

ls -l features/

retrieval
5 engines parallel · asyncio · adapter pattern · rate-limited · dedup by fingerprint
ranking
TF-IDF from scratch · cosine similarity · no sklearn · sparse matrix impl
clustering
k-means on TF-IDF · auto-k via silhouette · thematic labels via top terms
contradictions
Pairwise distance + sentiment lexicon · flagged in report
exports
BibTeX / APA / MLA / RIS / CSL-JSON / PDF (reportlab)
web-ui
stdlib http.server · Jinja2 · SQLite history · permalinks · zero build
~/projects/spyder $

cat stack.yaml

language
Python 3.11+
async
asyncio · aiohttp (HTTP client)
math
NumPy (TF-IDF matrix) · stdlib only for core logic
pdf
reportlab
templates
Jinja2
db
SQLite (history, cache)
cli
argparse (stdlib) · rich (optional pretty output)
~/projects/spyder $

git log --oneline

#1a2b3c4feat/providers

Multi-engine search provider abstraction

Python · aiohttp · asyncio
  • SearchProvider ABC: search(query, max_results) → List[Finding].
  • Concrete: DuckDuckGoHTML, ArXivAPI, PubMedEntrez, SemanticScholar, Crossref.
  • Unified error handling: retry with backoff, per-provider timeout, graceful degradation.
#2b3c4d5feat/tfidf

TF-IDF implementation from scratch

Python · NumPy
  • Tokenizer: regex word boundary + stopword list (NLTK stopwords bundled).
  • Sparse CSR matrix build: vocab → doc-term counts → IDF → TF-IDF normalization.
  • Query vector: same tokenizer + vocab projection; cosine via dot product / norms.
  • Benchmarked: ~50ms for 200 docs / 5000 vocab on laptop CPU.
#3c4d5e6feat/clustering

Thematic clustering & contradiction detection

NumPy · scikit-learn (k-means only)
  • k-means on TF-IDF rows; k selected by max silhouette score (range 2-8).
  • Cluster labels: top-3 TF-IDF terms per centroid; human-readable.
  • Contradiction: pairwise cosine distance > 0.7 AND sentiment polarity flip (VADER lexicon).
#4d5e6f7feat/exports

Multi-format citation export & PDF reports

reportlab · Jinja2 · CSL-JSON
  • Citation formatter: BibTeX, APA 7, MLA 9, RIS, CSL-JSON from unified metadata dict.
  • PDF: cover page, table of contents, clustered findings with citations, contradiction appendix.
  • Template-driven — customizable via user-provided Jinja2 overrides.
#5e6f7a8feat/webui

Zero-dependency stdlib web UI

http.server · Jinja2 · SQLite
  • Single command: `python -m spyder.web` → serves on localhost:8080.
  • Routes: / (search), /history, /report/{hash}, /static/ (embedded CSS/JS).
  • History: SQLite FTS5 virtual table for full-text search over past queries.