package
0.0.0-20241001134721-f44fe3ff2f61
Repository: https://github.com/dmitry-vovk/wcrawler.git
Documentation: pkg.go.dev
# Functions
NewFilter returns and instance of NormalizingFilter with sane defaults.
# Variables
Set some sensible normalisation flags, may be tweaked to suit a specific site requirements.
# Structs
NormalizingFilter is a URL filter with basic normalisation rules.
# Interfaces
RobotsChecker defines a method to tell if this path allowed to be visited by this agent.