Building a smart, context-aware archive of Dutch politicians
Approach, hardware, data sources, and — deliberately — the failures and near-misses, written down so the next build skips them.
This is not a highlight reel. The point of writing it down is the failures: several cost real hours, one briefly destroyed a live, public dataset. If you're building something similar, the fastest way to skip our mistakes is to read the incident log first.
A searchable record, not a database
A pipeline that collects everything a public figure has said — official government transcripts, historical records, video appearances — parses it into a searchable, timestamped archive, and serves it through semantic (meaning-based, not keyword) search with an optional AI summary. Every result links back to a playable moment in the original source, so a claim can always be checked against the primary record.
Two live instances exist so far (Geert Wilders, Dilan Yeşilgöz), architected from the start to share data across politicians rather than duplicate it. A sibling project, abo-ali-search, applies the same architecture to a single public figure and predates the multi-politician work — its diarization pattern and infrastructure conventions were reused directly, several times, described below.
What we pulled, and from where
| Source | What | Method | Gotcha |
|---|---|---|---|
| TK OData | Official corrected parliamentary transcripts, 2013–present | Open API, unfiltered bulk download | $top silently caps pagination — omit it |
| Handelingen | Pre-2013 parliamentary record | SRU search, per-politician | 429 without a request delay |
| YouTube | Party channel content | yt-dlp, incremental archive | Rate-limited within minutes, no throttle |
| Debat Direct | Plenary session video, ~2010–present | Windowed HLS, stream-copy | Committee rooms: no coverage, ever |
| Abo Ali's output | Historical VOD + current live broadcast | yt-dlp then raw HLS capture | Current output isn't VOD at all |
The YouTube channel content is not curated — it's the entire channel, unfiltered: campaign ads, holiday greetings, and clips of other politicians from the same party end up in the pool, sorted out only by diarization at index time.
Owned iron, no cloud rental
Everything runs over a private Tailscale mesh — no cloud GPU rental, no per-token API costs.
+ 1 K2200 (unused, too old)
shared with live production workers
unified memory
Tailscale reverse proxy
A 4×P100 node sits on the same network, never brought into service — its management interface is locked out, and fixing that was never worth it against hardware already in use. Nothing here is exotic; the V100 is a 2017 GPU. What made the timelines short was reuse, not horsepower.
ASR, diarization, search
ASR — WhisperX (large-v3), GPU-batched. Diarization — pyannote.audio via its Python API directly, not WhisperX's CLI flag (see log 5.1). Convention: the diarized voice with the most total airtime in a recording is labeled as that person; every other voice stays an anonymous SPEAKER_NN — correct for single-uploader content without needing voice enrollment.
Search — BGE-M3 embeddings, FastAPI + SQLite + a flat .npy vector array — no vector database; the corpus is small enough that this is simply faster, with zero moving parts. RAG — a local llama.cpp instance, auto-discovered, with graceful degrade to search-only.
Distribution — HLS video/audio stream-copied (no re-encode), sharded index % N across every available host. 8-way sharding across 4 hosts took an estimated multi-week video backlog to about five days.
What actually went wrong
Roughly chronological. Each cost somewhere between an annoying hour and half a day — except one, marked critical, which cost a live dataset for about twenty minutes before full recovery.
OData pagination silently truncated
$top caps the total result count across every page and suppresses @odata.nextLink in doing so — counts looked plausible until they didn't match reality. Omit it; follow server-side paging instead.
YouTube rate-limit, zero warning
A bulk channel pull without a throttle got the session rate-limited within minutes — no useful error, just zero further downloads. Fixed with -t sleep between requests; a large channel now takes hours instead of minutes, which beats zero.
ffmpeg version drift across shards
ffmpeg ≥7/8 rejects a CDN's .m4v HLS segments under a stricter default extension allowlist — one host misbehaved, the others didn't, until the version difference was spotted. Detect and pass -allowed_extensions ALL -extension_picky 0 when supported.
A silent 24-hour hang
Two multi-hour debates caused ffmpeg to hang for 24+ hours on a half-closed CDN connection it never noticed. Because shards process dates sequentially, the hang silently blocked every later item behind it — a shard that looked "still running" had been dead for most of a day. Fixed with a read timeout plus a hard subprocess timeout.
Orphaned children, one live data race
Killing an orchestrator process doesn't kill its ffmpeg child — reparented to init, it keeps running. A restarted shard was found writing to the same output file as an orphan from the previous run. Fixed by reaping any reparented ffmpeg, on every host, before starting or restarting a shard.
Eleven hours lost to a hung SSH launch
ssh host 'nohup thing &' can hang the SSH session on inherited file descriptors even though the remote command started fine — nothing crashed, the orchestrator just sat there. Fix: launch inside an explicit subshell so the remote shell exits immediately, ssh host '(nohup thing &); exit 0'.
Stopped the wrong container — a live one
Cleaning up a crashed test container (already self-removed), a cleanup command filtered by image rather than exact name — and stopped the live production instance sharing that image, three days into uptime. Caught within seconds, restarted immediately.
Fixed, confirmed, still broken
A separate app's AI-answer feature failed with connection-refused. The first fix looked confirmed by an HTTP 200 — but that 200 carried {"error": "..."} in the body, missed because a truncated response glance landed on the wrong field. The real cause: a hardcoded environment override pointing at a port nothing had listened on since an earlier migration, silently shadowing working auto-discovery.
Config on disk, config in memory — disagreed
A reverse-proxy file on disk pointed at the wrong backend entirely, while the running process still had the old, correct value loaded from before a never-reloaded edit. It worked by accident; the next routine reload would have broken it silently. Found by diffing the file against nginx's actual in-memory config.
New GPU work failed, old work didn't
An unattended package upgrade bumped the userspace NVIDIA driver without reloading the kernel module — already-running GPU contexts survived, nothing new could be created. Fixed with a full reboot after confirming every production service would auto-recover.
A rebuild wiped a live dataset to one entry
An index-rebuild script does a full rebuild from source transcripts every time, by design. Testing it revealed — too late — that the source transcripts for 2,519 of 2,523 archived videos no longer existed anywhere on the serving host; only the compiled index and raw video had survived. The rebuild found one new test file and the live index dropped from 2,523 videos to 1, instantly and publicly, with nothing in the surrounding tooling expecting a rebuild to be destructive.
No backup existed on the serving host, in git, or in any snapshot. A deliberately widened search across every other host on the network turned up a complete, undocumented copy on a third machine, found by chance because the search didn't stop at the obvious places. Restored within the hour, verified with a real content query this time.
Mistook a live stream for an archive
New content from Abo Ali's current output was assumed to be VOD, matching the historical YouTube pattern. It's a continuous 24/7 live stream — confirmed when the HLS manifest's internal naming read literally live40_..., and a fluctuating live bitrate broke a first extraction attempt that had hardcoded one manifest's bandwidth value. Recording a live stream needs a scheduled capture window, not an incremental sync — and the correct window came directly from asking, since nothing in any log documented it.
Right data, wrong name on screen
Two live instances share one frontend, told apart only by a backend config value. The backend was correctly person-aware from day one — but the static page had the first politician's name hardcoded into the title, hero text, and body copy, so the second instance served correct results under the wrong name and photo. Initial verification checked only the API response, never the rendered page.
A "bug" that was a real, permanent gap
A distributed run reported all shards complete, but seven expected files were missing with no error logged anywhere. The instinct was a sharding bug. The actual cause, confirmed against the source agenda API directly: those sessions were held in committee rooms, which the video source doesn't cover — the script correctly produced no output at all for a day matching nothing, indistinguishable from a silent bug unless you check the source directly.
Reuse, not horsepower
The distinguishing factor across this whole build was reuse of prior, hard-won work over reinventing it:
A near-identical diarization implementation already existed in the sibling project, proven at scale months earlier. The first attempt ignored it and used a different, broken-on-this-hardware library path — reverting to the proven pattern fixed it immediately, zero further debugging.
The shared-transcript-pool architecture meant the second politician's onboarding reused the majority of already-downloaded data outright — measured directly: 62% of relevant debate transcripts and 30% of relevant debate video needed zero new work. Every download step is idempotent and resumable by design; a crash never loses partial progress. Docker images and model caches validated on one host were copied wholesale to a second rather than re-derived, avoiding a repeat of dependency-version debugging in a new environment.
The pattern underneath all of it, wins and incidents alike: check whether the problem in front of you has already been solved somewhere in the same family of projects before treating it as new.
And: never let a destructive operation run without a backup existing first, as a property of the tool, not a step someone has to remember.