The ESPN scraper is Node, and it works fine. But once the movie stream aggregator
needed multiple scrapers running on a schedule, hitting several sources
concurrently, I ran into the usual Node scraping annoyances. Juggling concurrency
with Promise.all and semaphores, memory creeping up on long runs, and
error handling that gets messy once you're retrying failed requests across
several sources at once.
What Go actually gives me here
Goroutines make concurrent scraping across multiple sources simple to write and reason about, which is the main draw. It also compiles down to a static binary, so there's no dependency tree to worry about when I deploy a scraper to run on a schedule. On top of that, memory use stays noticeably lower on long-running or repeated scraping jobs.
None of this is a knock on Node. It's still what I reach for on the API and frontend side. This is specifically about the scraping workload, where a lot of short-lived concurrent HTTP requests plus HTML parsing is the kind of thing Go handles well.
What the switch looks like in practice
// one goroutine per source, results collected on a channel
for _, src := range sources {
go func(s Source) {
data, err := scrape(s)
results <- ScrapeResult{s.Name, data, err}
}(src)
}That's roughly the shape of it. Nothing exotic, just goroutines and channels doing what they're good at. Compared to coordinating the same thing in Node, it's less code and easier to follow six months later.
Where this leaves things
The IMDb scraper is the first one I'm writing in Go from scratch. The ESPN one stays in Node for now since it works and isn't running concurrently against anything else. No reason to rewrite something that isn't broken.