CheckErr checks if err is nil, if not, panic with msg and err.

CheckErrNop checks if err is nil, if not, print log with msg and err.

Compress the given interface.

ConvTimeUnit converts time value from one unit to another unit.

CPUUsage returns current cpu busy percentage.

DaysSince returns how many days since the provided date.

DecodeBytes to the given interface.

DecodeCompressed bytes to the given interface.

Decompress the given byte array.

EncodeToBytes transforms given object to bytes.

FileExists checks (with optional retry) whether the specified file existsin the provided directory (or optionally its sub-directory).

GetPartitionsFor the specified schema and table.

HTTPGet initiates HTTP get request and returns its response.

HTTPGetResponse initiates HTTP get request and returns its response.

HTTPPostJSON visits url using http post method, optionally using provided headers.

LogReturn calculates log return based on previous value, current value and bias.

MkDirAll similar to os.MkDirAll, but with retry when failed.

NewGCSClient creates a new storage.Client instance.

NumOfFiles counts files matching the provided patternunder the specified directory (or optionally its sub-directory).

ParseLines parses target file specified by absolute path.

PickProxy randomly chooses a proxy from database.The queried proxy will be cached for conf.Args.Network.RotateProxyRefreshInterval.

PickProxyDirect randomly chooses a proxy from the pool directly.

PickUserAgent picks a user agent string from the pool randomly.if the pool is not populated, it will trigger the initialization processto fetch user agent lists from remote server.

RandomProxy randomly decides whether to use a proxy, master proxy,or rotate proxy based on the given weights for respective chances.

SameWeek checks if two dates denoted by respective strings are in the same week.

Str2Fnull converts string value to sql.NullFloat64.

Str2IntfSlice converts string slice to slice of interface{}.

TimeStr returns date and time in project standard formats respectively.

UnixMilliseconds returns milliseconds since Unix Ephoch.

UpdateProxyScore for the specified proxy.

WriteJSONFile writes provided payload object pointer as (gzipped) json formatted file.it first tries to write to a *.tmp file, then renames it to *.json(.gz),then returns the final path of the written file.

GCSClient may serve as a handy wrapper for google cloud storage client,Caller can reuse the same storage.Client instance in concurrent goroutines.Initialization is performed automatically, and is thread-safe.Get the GCSClient via NewGCSClient function.

Proxy represents the table structure of proxy_list.

UserAgent represents user_agent table structure.