I stopped indexing the internet and let the database build itself

MyTops started as a fix for the half-hour “what should we watch” negotiation. It turned into a proof of concept for a database that gets populated by the questions people ask it.

29 August 2026
// part of MyTops — Your favourites, everywhere

Every couple of evenings the same thing happens. We sit down, we want to watch something, and half an hour later we’re still deciding.

It isn’t a lack of ideas. The ideas are constant — they just never arrive at the moment they’re needed. Someone mentions a film mid-conversation, a trailer catches me while I’m doing something else entirely, a friend recommends a series in a message I never go back to. By the time we’re actually on the sofa, all of it is gone.

Then I noticed the same loop runs for restaurants. And for games. Different topic, identical failure: ideas at the wrong time, nothing at the right one.

There are apps for this. That’s the problem — there’s an app per topic. My list of films lives one place, my list of restaurants another, games somewhere else, and none of them know about the others. And every one of them eventually breaks the same way: I go to save something and it isn’t in their catalogue, so it can’t be saved at all. A collection that refuses items is not really a collection.


The idea came before I wrote anything. The first commit landed in December 2025, and I’ve been picking at it since, in whatever time a pet project gets.

My first instinct was the wrong one. If the failure is missing content, index everything. Pull the catalogues, stitch six vertical providers together, build the union.

That’s impossible, and — the part that took me longer to accept — it’s also not advisable. There’s no dataset you can license that covers anime and restaurants and board games with equal quality. Six integrations means six failure modes, six sets of terms, six things to keep in sync. And after all of it, you still have no answer for the one item none of them carry, which is the exact case that made me start.

So I inverted it. Don’t index the internet. Index what people actually ask for.


The result is embarrassingly simple to describe. Every query hits Postgres first. If the item is there — which, once a topic has any history, it usually is — it’s a plain database read and no model is involved at all.

If it isn’t there, the pipeline runs. Search and fetch through Tavily, extraction by the Claude API into that topic’s schema, validation against that schema, insert with a confidence score attached.

Two decisions in there matter more than the rest.

Per-topic schemas, not a generic item table. A film is a director and a runtime and a year. A restaurant is a cuisine and a location and a price band. Flatten those into one shape and you get a catalogue where nothing is searchable in the way its topic actually wants to be searched. So each topic defines its own metadata and its own validation, and extraction targets a real shape rather than a bag of fields.

No human in the write path. I considered review-before-publish and dropped it fast. The whole promise is that the item you asked for exists now; queuing it for approval reintroduces exactly the failure I was fixing. So entries land immediately, carrying a confidence score, and users can flag anything wrong for admin review afterwards. Optimistic happy path. It’s a bet that a slightly imperfect entry beats no entry, and so far that bet has held.

The nice property falls out on its own: a miss for one person is a hit for everyone after. Each item costs something exactly once, and the catalogue ends up written by its own traffic.


Now the parts I got wrong, or haven’t got to yet.

A miss takes seconds. Search, fetch, extract, validate — that’s real wall-clock time and there’s no hiding it. When I use the app, I know what’s happening behind that pause, so I wait. My friends know too. A stranger would assume it’s broken and close the tab, and they’d be right to. Explaining the delay in the UI helps, but the honest version is that this is still the weakest part of the product.

I never built the cheap wins. There’s no bulk import of the obvious stuff — the several thousand items any user is statistically likely to search first. And there’s no batch strategy on a hit: found a film, so pull that director’s filmography while you’re already there. Both are straightforward. Neither is done. Which means every single query today, popular or obscure, is a fresh model call.

The app is open — anyone can sign up and use it, free — and I’m paying those tokens myself. So there are usage limits in place, sized to protect me from a runaway bill rather than to keep anyone out. That works, but it’s the blunt instrument. The limits are currently doing the job that better strategy should be doing: a common query and an obscure one cost me exactly the same today, and capping volume doesn’t fix that, it just contains it.

The design is plain on purpose. I’m a backend engineer. I don’t know how to build UI or UX, and I’ve stopped pretending the gap will close by reading about it. So the interface is deliberately straightforward: shadcn/ui, sensible defaults, nothing I can’t reason about. It isn’t beautiful. But plain and working is the only thing I can ship honestly, and a design I half-understand would be worse than one I chose.


It’s live, it works, and the thesis I actually cared about held up: you can grow a usable cross-topic catalogue from literally zero, with no licensing deal and no crawl, just by answering the questions people bring you and keeping the answers.

The rest is cost engineering, and cost engineering is a solvable problem.

MyTops is live and free, and the whole thing is open source. If you want to talk about building something like it, get in touch.