Categorygithub.com/Laisky/go-utils/v4
modulepackage
4.10.0
Repository: https://github.com/laisky/go-utils.git
Documentation: pkg.go.dev
Overview
Versions
5
Dependencies
59
Dependents
8
Files
3.7k SLOC

# README

Go-Utils

Many useful golang tools

VersionSupport Go
v1>= v1.16
v2>= v1.18
v3>= v1.19
v4>= v1.20

License: MIT Commitizen friendly Go Report Card GoDoc Build Status codecov

Use as command-line executable binary

Install

go install github.com/Laisky/go-utils/v4/cmd/gutils@latest

Add $HOME/.go/bin to your PATH.

Usage

# find and delete duplicate files/ similar images
gutils remove-dup examples/images --dry

# move files to hash-based hierach directories
gutils md5dir -i examples/md5dir/ --dry

# show x509 certificate details
gutils certinfo -r blog.laisky.com:443
gutils certinfo -f ./cert.pem

# encrypt by aes
gutils encrypt aes -i <file_path> -s <password>

# sign or verify by rsa
gutils rsa sign
gutils rsa verify

Use as SDK

Install

go get github.com/Laisky/go-utils/v4@latest

Usage

import (
    gutils "github.com/Laisky/go-utils/v4"
)

Modules

Contains some useful tools in different directories:

  • settings: move go github.com/Laisky/go-config
  • color.go: colorful code
  • compressor.go: compress and extract dir/files
  • email/: SMTP email sdk
  • encrypt/: some tools for encrypt and decrypt, support AES, RSA, ECDSA, MD5, SHA128, SHA256
    • configserver.go: load configs from file or config-server
  • fs.go: some tools to read, move, walk dir/files
  • http.go: some tools to send http request
  • jwt/: some tools to generate and parse JWT
  • log/: enhanched zap logger
  • math.go: some math tools to deal with int, round
  • net.go: some tools to deal with tcp/udp
  • random.go: generate random string, int
  • sort.go: easier to sort
  • sync.go: some locks depends on atomic
  • throttle.go: faster rate limiter
  • time.go: faster clock (if you do not enable vdso)
  • utils: some useful tools

# Packages

Package algorithm contains some useful algorithms.
Package cmd some useful tools for command argument.
Package common global shared utils.
Package compress contains some useful tools to compress/decompress data or files.
Package counter contains varias counter tools.
Package crypto is a collection of cryptographic algorithms and protocols, providing hash functions, block and stream ciphers, public key cryptography and authentication.
Package email simple email sender.
Package gorm some useful tools for gorm.
Package json implements encoding and decoding of JSON as defined in RFC 7159.
Package jwt all in one JWT sdk.
Package log enhanced zap logger.
No description provided by the author

# Functions

AutoGC auto trigger GC when memory usage exceeds the custom ration default to /sys/fs/cgroup/memory/memory.limit_in_bytes.
CheckResp check HTTP response's status code and return the error with body message.
CloseWithLog close and log error.
Color wrap with ANSI color inspired by github.com/fatih/color.
CombineSortedChain return the intersection of multiple sorted chans.
Contains if collection contains ele.
ConvertMap2StringKey convert any map to `map[string]any`.
CopyFile copy file content from src to dst.
CostSecs convert duration to string like `0.25s`.
DecodeByBase64 decode string to bytes by base64.
Dedent removes leading whitespace or tab from the beginning of each line will replace all tab to 4 blanks.
DeepClone deep clone a struct will ignore all unexported fields.
DirSize calculate directory size inspired by https://stackoverflow.com/a/32482941/2368737.
EmptyAllChans receive all thins in all chans.
EncodeByBase64 encode bytes to string by base64.
FallBack return the fallback when orig got error utils.FallBack(func() any { return getIOStatMetric(fs) }, &IOStat{}).(*IOStat).
FileExists is path a valid file.
FileHash generate file signature by hash.
FileHashSharding get file hash sharding path.
FileMD5 read file and calculate MD5 Deprecated: use Hash instead.
FileSHA1 read file and calculate sha1 return hashed string in 40 bytes Deprecated: use Hash instead.
FilterSlice filters a slice inplace.
FlattenMap make embedded map into flatten map.
FlushWithLog flush and log error.
ForceGCBlocking force to run blocking manual gc.
ForceGCUnBlocking trigger GC unblocking.
GetEnvInsensitive get env case insensitive.
GetFuncName return the name of func.
GetStructFieldByName get struct field by name.
GracefulCancel is a function that will be called when the process is about to be terminated.
HasField check is struct has field inspired by https://mrwaggel.be/post/golang-reflect-if-initialized-struct-has-member-method-or-fields/.
Hash generate signature by hash.
HashSHA128String calculate string's hash by sha256 Deprecated: use Hash instead.
HashSHA256String calculate string's hash by sha256 Deprecated: use Hash instead.
HashVerify verify by hash.
HashXxhashString calculate string's hash by sha256 Deprecated: use Hash instead.
HasMethod check is struct has method inspired by https://mrwaggel.be/post/golang-reflect-if-initialized-struct-has-member-method-or-fields/.
HTTPInvalidStatusError return error about status code.
Input reads input from stdin.
InputPassword reads password from stdin input and returns it as a string.
InputYes require user input `y` or `Y` to continue.
IsDir is path exists as dir.
IsDirWritable if dir is writable.
IsEmpty is empty.
IsFile is path exists as file.
IsFileATimeChanged check is file's atime equal to expectATime.
IsPanic is `f()` throw panic if you want to get the data throwed by panic, use `IsPanic2`.
IsPanic2 check is `f()` throw panic, and return panic as error.
IsPtr check if t is pointer.
IsRemoteUDPPortOpen check is remote udp port open.
JoinFilepath join paths and check if result is escaped basedir basedir is the first nonempty path in paths.
JumpHash fatest consistent hashing created by google.
ListFilesInDir list files in dir.
ListFilesInDirFilter filter files, only return files that filter returns true.
ListFilesInDirRecursive list files in dir recursively.
LogErr invoke f and log error if got error.
Max return the maximal value.
MD5JSON calculate md5(jsonify(data)).
Min return the minimal value.
MockStdout mock stdout to a temp file Example: func TestMockStdout(t *testing.T) { recover, stdout, err := MockStdout() require.NoError(t, err) defer recover() fmt.Println("hello") stdout.Seek(0, 0) buf, err := io.ReadAll(stdout) require.NoError(t, err) require.Equal(t, "hello\n", string(buf)) }.
MoveFile move file from src to dst by copy sometimes move file by `rename` not work.
NewTask new async task ctx must keep alive for whole lifecycle of AsyncTask.
NewAsyncTaskStoreMemory new default memory store.
NewClock create new Clock.
NewDelay ensures the execution time of a function is not less than a predefined threshold.
NewExpCache new cache manager use with generic: cc := NewExpCache[string](context.Background(), 100*time.Millisecond) cc.Store("key", "val") val, ok := cc.Load("key").
NewExpiredRLock new ExpiredRLock.
NewFlock new file lock.
NewGoroutineTest new test for goroutine any fail will call cancel().
NewHTTPClient create http client.
NewJaegerTracingID generate jaeger tracing id Args: - traceID: trace id, 64bit number, will encode to hex string - spanID: span id, 64bit number, will encode to hex string - parentSpanID: parent span id, 64bit number, will encode to hex string - flag: 8bit number, one byte bitmap, as one or two hex digits (leading zero may be omitted) Even if some of the parameters have incorrect formatting, it won't result in an error; instead, it will generate a new random value.
NewLruCache new lru cache.
NewLRUExpiredMap new ExpiredMap.
NewMutex create new mutex.
NewPermissionTree new permission tree only contains root node.
NewRand new individual random to aviod global mutex.
NewRateLimiter create new Throttle 90x faster than `rate.NewLimiter`.
NewSingleItemExpCache new expcache contains single data.
NewTmpFile write content to tmp file and return path.
NewTmpFileForContent write content to tmp file and return path deprecated: use NewTmpFileForReader instead.
NewTtlCache new cache with ttl.
NilInterface make sure data is nil interface or another type with nil value Example: type foo struct{} var f *foo var v any v = f v == nil // false NilInterface(v) // true.
NotEmpty val should not be empty, with pretty error msg.
OpenURLInDefaultBrowser opens the specified URL in the default browser of the user.
OptionalVal return optionval if not empty.
Overwrite overwrite file if target existed.
PaddingLeft padding string to left.
PanicIfErr panic if err is not nil.
ParseHex2UTC parse hex to UTC time.
ParseHexNano2UTC parse hex contains nano to UTC time.
ParseObjectIdentifier parse oid.
ParseTimeWithTruncate parse time with truncate.
ParseUnix2String can parse unix timestamp(int64) to string.
ParseUnix2UTC convert unix to UTC time.
ParseUnixNano2UTC convert unixnano to UTC time.
ParseUUID7 parse uuid7.
Pipeline run f(v) for all funcs.
PrettyBuildInfo get build info in formatted json Print: { "Path": "github.com/Laisky/go-ramjet", "Version": "v0.0.0-20220718014224-2b10e57735f1", "Sum": "h1:08Ty2gR+Xxz0B3djHVuV71boW4lpNdQ9hFn4ZIGrhec=", "Replace": null }.
RaceErr return when any goroutine returned.
RaceErrWithCtx return when any goroutine returned or ctx canceled.
RandomBytesWithLength generate random bytes.
RandomChoice selects a random subset of elements from an input array of any type.
RandomNonZeroUint64 generate random uint64 number.
RandomStringWithLength generate random string with specific length.
Recursive list files recursively Deprecated: use ListFilesInDirRecursive instead.
RegexNamedSubMatch extract key:val map from string by group match Deprecated: use RegexNamedSubMatch2 instead.
RegexNamedSubMatch2 extract key:val map from string by group match.
RemoveEmpty remove duplicate string in slice.
RemoveEmptyVal remove empty value in map.
RenderTemplate render template with args.
RenderTemplateFile render template file with args.
ReplaceFile replace file with content atomatically this function is not goroutine-safe.
ReplaceFileAtomic replace file with content atomatically write content to a tmp file, then rename it to dst file.
RequestJSON request JSON and return JSON by default client.
RequestJSONWithClient request JSON and return JSON with specific client.
ReverseSlice reverse slice.
RunCMD run command script.
RunCMD2 run command script and handle stdout/stderr by pipe.
RunCMDWithEnv run command with environments # Args - envs: []string{"FOO=BAR"}.
RunWithTimeout run func with timeout.
SanitizeCMDArgs sanitizes the given command arguments.
SecRandInt generate security int.
SecRandomBytesWithLength generate crypto random bytes.
SecRandomStringWithLength generate random string with specific length.
SetInternalClock set internal Clock with refresh interval.
SetStructFieldsBySlice set field value of structs slice by values slice.
SilentClose close and ignore error Example defer SilentClose(fp).
SilentFlush flush and ignore error.
SleepWithContext sleep duration with context, if context is done, return.
SortBiggest sort from biggest to smallest.
SortSmallest sort from smallest to biggest.
StopSignal registered for SIGTERM and SIGINT.
TemplateWithMap replace `${var}` in template string.
TemplateWithMapAndRegexp replace `${var}` in template string.
TimeEqual compare two time with difference, return true if time difference less than difference.
TrimEleSpaceAndRemoveEmpty remove duplicate string in slice.
UniqueStrings remove duplicate string in slice.
URLMasking masking password in url.
UTCNow get current time in utc.
UUID1 get uuid version 1 Deprecated: use UUID7 instead.
UUID4 get uuid version 4.
UUID7 get uuid version 7.
ValidateFileHash validate file content with hashed string Args: - filepath: file path to check - hashed: hashed string, like `sha256: xxxx`.
WaitComplete wait all goroutines complete or ctx canceled, returns the first non-nil error (if any) from them, or return ctx.Err() if ctx canceled.
WaitTCPOpen wait tcp open.
WatchFileChanging watch file changing when file changed, callback will be called, callback will only received fsnotify.Write no matter what happened to changing a file.
WithFileFlag how to write dst file.
WithFileMode if create new dst file, set the file's mode.
WithGCMemLimitFilePath set memory limit file.
WithGCMemRatio set mem ratio trigger for GC default to 85.
WithHTTPClientInsecure set http client igonre ssl issue default to false Deprecated: use WithHTTPTlsConfig instead.
WithHTTPClientMaxConn set http client max connection default to 20.
WithHTTPClientProxy set http client proxy.
WithHTTPClientTimeout set http client timeout default to 30s.
WithHTTPTlsConfig set tls config.
WithPrettyBuildInfoDeps include deps in build info.
WithReplaceTabBySpaces replace tab to spaces.
WithStopSignalCloseSignals set signals that will trigger close.
Wrap wrap error without stack.

# Constants

Background text colors.
Background text colors.
Background text colors.
Background text colors.
Background Hi-Intensity text colors.
Background Hi-Intensity text colors.
Background Hi-Intensity text colors.
Background Hi-Intensity text colors.
Background Hi-Intensity text colors.
Background Hi-Intensity text colors.
Background Hi-Intensity text colors.
Background Hi-Intensity text colors.
Background text colors.
Background text colors.
Background text colors.
Background text colors.
Base attributes.
Base attributes.
Base attributes.
Base attributes.
Base attributes.
ANSIColorEscape escape string for ANSI color.
Base attributes.
Foreground text colors.
Foreground text colors.
Foreground text colors.
Foreground text colors.
Foreground Hi-Intensity text colors.
Foreground Hi-Intensity text colors.
Foreground Hi-Intensity text colors.
Foreground Hi-Intensity text colors.
Foreground Hi-Intensity text colors.
Foreground Hi-Intensity text colors.
Foreground Hi-Intensity text colors.
Foreground Hi-Intensity text colors.
Foreground text colors.
Foreground text colors.
Foreground text colors.
Foreground text colors.
Base attributes.
Base attributes.
Base attributes.
Base attributes.
AsyncTaskStatusDone task done.
AsyncTaskStatusFailed task failed.
AsyncTaskStatusPending task pending.
AsyncTaskStatusUnspecified unknown.
BaseHex 16.
BitSize64 64.
HashTypeMD5 MD5.
HashTypeSha1 Sha1.
HashTypeSha256 Sha256.
HashTypeSha512 Sha512.
HashTypeXxhash Xxhash.
HTTPHeaderContentType HTTP header name.
HTTPHeaderContentTypeValJSON HTTP header value.
HTTPHeaderHost HTTP header name.
HTTPHeaderReferer HTTP header name.
Nano2Sec 1e9.
TimeFormatDate "2006-01-02".
TracingKey default trace key https://www.jaegertracing.io/docs/1.22/client-libraries/#key `{trace-id}:{span-id}:{parent-span-id}:{flags}`.

# Variables

AbsInt32 abs(v) ignore int exceeds limit error, abs(MinInt32) == MaxInt32.
AbsInt64 abs(v) ignore int exceeds limit error, abs(MinInt64) == MaxInt64.
Bytes2Str unsafe convert bytes to str.
Clock high performance time utils, replace Clock1.
DecodeByHex decode string to bytes by hex.
EncodeByHex encode bytes to string by hex.
ErrAsyncTask root error for async tasks.
ForceGC force to start gc blocking.
HumanReadableByteCount convert bytes to human readable string Args: - bytes: - si: `si ? 1024 : 1000` Example: `HumanReadableByteCount(1005, false) -> "1.01KB"`.
JSON effective json Deprecated: use github.com/Laisky/go-utils/v4/json instead.
NewThrottleWithCtx create new Throttle Deprecated: use `NewRateLimiter` instead.
Number2Roman convert number to roman.
ParseTs2String can parse unix timestamp(int64) to string.
ParseTs2Time can parse unix timestamp(int64) to time.Time.
ParseTs2UTC can parse unix timestamp(int64) to time.Time.
RegexpOidFormat check if oid is valid.
ReplaceFileStream replace file with content atomatically Deprecated: use ReplaceFileAtomic instead.
Round round float64 Round(1.005, 2) -> 1.01.
Str2Bytes unsafe convert str to bytes.
TimeZoneShanghai timezone Shanghai TimeZoneShanghai = time.FixedZone("Asia/Shanghai", 8*3600).
TimeZoneUTC timezone UTC.
TriggerGC force to start gc unblocking.
UnixNano2UTC convert unixnano to UTC time.

# Structs

AsyncTask async task manager.
AsyncTaskResult result of async task.
AsyncTaskStoreMemory example store in memory.
ClockT high performance ClockT with lazy refreshing.
Delayer create by NewDelay do not use this type directly.
ExpCache cache with expires can Store/Load like map.
ExpiredRLock Lock with expire time.
GoroutineTest testing.T support goroutine.
LRUExpiredMap map with expire time, auto delete expired item.
Mutex mutex that support unblocking lock.
RateLimiter current limitor.
RateLimiterArgs Throttle's configuration.
RBACPermissionElem element node of permission tree the whole permission tree can represented by the head node.
RequestData http request.
RWManager auto create rwlock if not exists.
SingleItemExpCache single item with expires.
TtlCache cache with ttl.

# Interfaces

asyncTask async task.
AsyncTaskStoreInterface persistency storage for async task.
ClockItf high performance lazy clock.
ExpCacheInterface cache with expire duration.
FLock lock by file.
HashTypeInterface hashs.
RWManagerInterface auto create rwlock if not exists.
SortItemItf interface of sort item.
UUID7Itf general uuid7 interface.

# Type aliases

AsyncTaskStatus status of async task.
CopyFileOptionFunc set options for copy file.
DedentOptFunc dedent option.
GcOptFunc option for GC utils.
HashType hashs.
HTTPClientOptFunc http client options.
JaegerTracingID jaeger tracing id.
ListFilesInDirOptionFunc options for ListFilesInDir.
Number is a number type.
PairList array of sort items.
PrettyBuildInfoOption options for PrettyBuildInfo.
RBACPermFullKey key with ancesters.
RBACPermKey permission identity keyword format like `a`.
Sortable Data types that can be compared by >, <, ==.
StopSignalOptFunc options for StopSignal.
Throttle rate limitor Deprecated: use `RateLimiter` instead.
ThrottleCfg Throttle's configuration Deprecated: use `RateLimiterArgs` instead.