Up to now the IMDb scraper and the movie stream aggregator have been separate things. The scraper ran on its own and dumped metadata somewhere, and the aggregator was a separate prototype that assumed that data already existed. I'm merging them into one app instead of two things I run side by side.
Why keeping them apart didn't make sense
The aggregator doesn't really work without fresh IMDb metadata behind it, and the scraper was writing to a format that the aggregator's API then had to reshape anyway. On top of that, two codebases meant two places to update whenever the schema changed, and I kept forgetting to update both.
It made sense to build them separately at first. I wanted to get the scraper working on its own before wiring anything else to it. Now that both are further along, keeping them as separate projects is just extra friction for no real benefit.
How it fits together now
stream-aggregator/
scraper/ (Go, IMDb metadata)
api/ (Express, availability + search)
web/ (Next.js frontend)The scraper still runs as its own process on a schedule, but it's part of the same app now, writing directly into the schema the API expects instead of a generic dump I had to transform separately. Everything lives in one repo and shares one schema, so I only have to update it in one place.
What changes for the frontend
Practically, this means search results in the app can now show real IMDb ratings, cast, and year alongside streaming availability, instead of the placeholder data I was using while the two pieces were separate. Search on the aggregator now actually returns something worth looking at.
Next
Getting the scheduled scraper runs to also update existing entries, not just add new ones, so ratings and cast data stay current instead of going stale after the first fetch.