mercredi 1 février 2017

Testing Go Web Scraper with Go-VCR

I'm newer to the Go language and it's resources, but have been looking around for quite some time without any luck of finding what I'm looking for. So if there is a resource out there for it, I apologize for the duplicate question and would appreciate being directed that way.

My goal is simply to build a web scraper. I'm using chromedp, which has features to focus on elements, fill in text, etc. I want to create a test environment/server to test it with during development. The main reason being that I do not want to constantly create GET requests from a website (out of common courtesy), but also be able to work offline and in addition it should also make testing a little faster. I stumbled across the go-vcr library and have been trying to get that to work, but to no avail. I can get it to record and create a .yaml, but I can't figure out how to test beyond the raw html that bounces back and gets stored in the .yaml file. My understanding is that it's possible to replicate the website and functionality of it using the library, but I'm unable to piece together how to do that.

Is what I'm trying to do possible, or is the go-vcr library (or any test/fake server for that matter) only capable of returning static data, therefore rendering anything I want to test with the web scraper not possible?

I haven't posted any code simply because I haven't pieced together much more than the examples given from the repository for the go-vcr.

I hope I was able to explain that in a way that made sense. If not I'd be happy to answer questions to clarify.

Aucun commentaire:

Enregistrer un commentaire