The movie stream aggregator needs a reliable source of movie and show metadata, things like titles, release years, cast, and ratings, to match against whatever the streaming availability scrapers pull back. IMDb is the obvious source since it's the most complete database out there, so that's what I'm building against. It's also my first scraper written in Go instead of Node.
What it needs to pull
- Title, year, and type (movie vs. series vs. episode)
- Rating and vote count
- Cast and crew, at least top-billed
- Genre tags, for filtering later on
The annoying part: IDs vs. names
IMDb identifies everything by a stable ID (tt0111161 for The
Shawshank Redemption, for example), which is great once you have it but useless
as a starting point when all you have is a title string from another source.
Most of the actual work so far has been building a decent search-and-match step:
searching IMDb by title, then confirming the match using year and type before
trusting the result.
Rate limiting, carefully
This is a bigger dataset than the football scraper touches, and I'm hitting it more often, so I'm being more careful with request pacing and caching results locally rather than re-fetching the same title twice. Go's goroutines make it easy to go too fast by accident, so most of the early work has been on the throttling side rather than the parsing side.
Status
Search-and-match and the core metadata fields are working. Cast and crew parsing is next, then I'll wire the output into the same normalized schema the aggregator project uses so it can sit next to the streaming availability data.