Returns an instance of CrawlOptions with the values set to sensible defaults.

Utility to create a NodeChSet and get full access to the Quit & Node Channel.

Utility function to create a NodeChSet given a created Node and Quit Channel.

New - Create an Instance of the Octopus with the given CrawlOptions.

NewWithDefaultOptions - Create an Instance of the Octopus with the default CrawlOptions.

CrawlOptions is used to house options for crawling.

Node encloses a NodeInfo and its Body (HTML) Content.

NodeChSet is the standard set of channels used to build the concurrency pipelines in the crawler.

NodeInfo is used to represent each crawled link and its associated crawl depth.

StdChannels are used to hold the standard set of channels that are used for special operations.

OutputAdapter is the interface that has to be implemented in order to handle outputs from the octopus crawler.