Package mocks is a generated GoMock package.

NewCrawler returns a new crawler instance.

Config encapsulates the configuration options for creating a new Crawler.

Crawler implements a web-page crawling pipeline consisting of the following stages: - Given a URL, retrieve the web-page contents from the remote server.

Graph is implemented by objects that can upsert links and edges into a link graph instance.

Indexer is implemented by objects that can index the contents of web-pages retrieved by the crawler pipeline.

PrivateNetworkDetector is implemented by objects that can detect whether a host resolves to a private network address.

URLGetter is implemented by objects that can perform HTTP GET requests.