nafi@portfolio:~$
bash — ~/projects/seltos
$cd ~/projects/seltos && cat summary.txt
Bangladesh-focused intelligence terminal scraping 9 news sources in parallel every 15 minutes. Zero cloud deps, zero telemetry. Bangla language detection, geo-tagging to 64 districts, story clustering with importance/hotness scoring. Jarvis-style dashboard with 3D WebGL globe, district heat layer, live SSE updates, optional multi-provider LLM summaries.

SELTOS

Full-Stack Developer · #c9a41f5 intel
Node.js · Express · SSE · WebGL Globe github.com/sleepyhead000/Seltos
~/projects/seltos $

cat README.md

README.md
$cat overview.txt
Seltos is a self-hosted, Bangladesh-centric intelligence platform. It ingests articles from 9 Bangla/English news sources every 15 minutes via a parallel scraper pool, detects Bangla text, geo-tags mentions to 64 districts, clusters related stories, and scores them for importance and "hotness" (velocity of coverage). The dashboard is a Jarvis-style single-page app with a WebGL globe, district heatmap, live SSE feed, and an optional LLM layer for executive summaries.
$cat ingestion.txt
Scraper pool: 9 sources (Prothom Alo, Bdnews24, Kaler Kantho, Jugantor, Dhaka Tribune, The Daily Star, New Age, Bangla Tribune, BBC Bangla). Each runs in a worker thread; 15-min cron via node-cron. HTML parsing with cheerio; RSS/Atom fallback where available. Dedup by content hash (SimHash 64-bit). Language detection: fastText lite model (Bangla/English) + script heuristic. Geo-tagging: gazetteer of 64 districts + 490 upazilas; mentions resolved via fuzzy match (Levenshtein ≤ 2).
$cat clustering.txt
Online clustering: incoming articles compared to active story clusters via MinHash LSH (Jaccard on shingled tokens). New cluster if max similarity < 0.35. Cluster scoring: importance = log(1 + source_count) * recency_decay(hours); hotness = velocity (articles/hr) * source_diversity. Clusters expire after 72h inactivity. Persisted in SQLite (WAL) with FTS5 for full-text search.
$cat dashboard.txt
Frontend: vanilla ES modules + Three.js (globe.gl) for 3D globe. District heat layer: choropleth via GeoJSON (Bangladesh admin boundaries). Globe arcs: inter-district story co-mentions. SSE endpoint (/events) pushes new articles + cluster updates in real time. LLM summaries: optional — pluggable provider interface (Ollama local, OpenAI, Anthropic, Gemini); summarizes top-3 clusters on demand. Zero telemetry, no external CDN — all assets self-hosted.
~/projects/seltos $

ls -l features/

ingestion
9 sources parallel · 15-min cron · worker threads · cheerio + RSS fallback
language
Bangla detection · fastText lite + script heuristic · 98% accuracy
geo
64 districts + 490 upazilas · gazetteer · fuzzy match (Levenshtein ≤ 2)
clustering
MinHash LSH · online · importance/hotness scoring · 72h TTL
dashboard
WebGL globe · district heatmap · co-mention arcs · SSE live feed
llm
Pluggable providers (Ollama/OpenAI/Anthropic/Gemini) · on-demand summaries
~/projects/seltos $

cat stack.yaml

runtime
Node.js 20+ · Express · worker_threads
scheduler
node-cron (15 min)
parsing
cheerio · RSS/Atom (fast-xml-parser)
nlp
fastText (Bangla) · SimHash · MinHash LSH (custom)
geo
gazetteer (JSON) · Levenshtein (stdlib)
db
SQLite (WAL + FTS5)
realtime
SSE (Express) · vanilla ES modules frontend
3d
Three.js · globe.gl · GeoJSON (BD admin boundaries)
llm
Provider interface: Ollama / OpenAI / Anthropic / Gemini
~/projects/seltos $

git log --oneline

#a1b2c3dfeat/scraper-pool

Parallel scraper pool with worker threads

Node.js · worker_threads · node-cron
  • 9 source configs (selectors, RSS URLs, rate limits) in JSON; hot-reloadable.
  • Each worker: fetch → parse → extract (title, body, pubDate, canonical URL) → emit.
  • Supervisor: restart on crash; backpressure via bounded channel; metrics per source.
#b2c3d4efeat/nlp-geo

Bangla detection & district geo-tagging

fastText · SimHash · Levenshtein
  • fastText lite (compressed 2MB) loaded once; inference ~2ms/article.
  • SimHash 64-bit for near-dup detection across sources (threshold 3 bits).
  • Gazetteer: 64 districts + 490 upazilas from BBS; fuzzy match with prefix trie optimization.
#c3d4e5ffeat/clustering

Online story clustering with MinHash LSH

Node.js · custom MinHash · SQLite FTS5
  • Shingling: 5-grams of normalized tokens; 128 permutations for MinHash signatures.
  • LSH bands: 16 bands × 8 rows; candidate pairs → Jaccard verification.
  • Cluster state: centroid signature, article IDs, importance, hotness, last_updated.
#d4e5f6afeat/dashboard

Jarvis-style dashboard: globe + heatmap + SSE

Three.js · globe.gl · SSE · vanilla JS
  • Globe: country outline + Bangladesh district polygons (GeoJSON); choropleth by cluster count.
  • Arcs: inter-district co-mentions within same cluster; animated particles.
  • SSE: EventSource auto-reconnect; pushes {type: 'article'|'cluster', payload}.
  • Panel: live feed, cluster cards (score, sources, summary), district drill-down.
#e5f6a7bfeat/llm

Pluggable LLM summary layer

Provider interface · Ollama / OpenAI / Anthropic / Gemini
  • SummaryProvider ABC: summarize(clusterArticles) → {summary, keyPoints, confidence}.
  • Implementations: local (Ollama llama3), cloud (OpenAI gpt-4o-mini, Anthropic haiku, Gemini flash).
  • On-demand via dashboard button; cached per cluster; cost/token logged locally.
#f6a7b8cops/selfhost

Zero-cloud self-hosted deployment

Docker · systemd · nginx (reverse proxy)
  • Single docker-compose: scraper, api, web, sqlite volume.
  • systemd units for bare-metal; logrotate; health checks via /healthz.
  • nginx: TLS termination, static asset caching, SSE proxy buffering off.