ContentDeduplicationJoint used to check the hash of page body, if duplicated hash already exists, will break the pipeline.

EmptyJoint is a place holder.

FilterCheckJointused to check the task url if it is already in the filter, if not in the filter, then add it to task filter, and make sure won't add it next time.

IndexJoint is used to send snapshot and task info into index.

LanguageDetectJoint used to detect the language of the webpage.

TaskDeduplicationJoint is used to find whether the task already in the database.

UrlFilterJoint used to validate urls, include host,path,file and file extension.

UrlNormalizationJoint used to cleanup url and do normalization.