Categorygithub.com/influxdata/gosnowflake
modulepackage
1.9.0
Repository: https://github.com/influxdata/gosnowflake.git
Documentation: pkg.go.dev

# README

Support

For official support and urgent, production-impacting issues, please contact Snowflake Support.

Go Snowflake Driver

Coverage

This topic provides instructions for installing, running, and modifying the Go Snowflake Driver. The driver supports Go's database/sql package.

Prerequisites

The following software packages are required to use the Go Snowflake Driver.

Go

The latest driver requires the Go language 1.19 or higher. The supported operating systems are Linux, Mac OS, and Windows, but you may run the driver on other platforms if the Go language works correctly on those platforms.

Installation

If you don't have a project initialized, set it up.

go mod init example.com/snowflake

Get Gosnowflake source code, if not installed.

go get -u github.com/snowflakedb/gosnowflake

Docs

For detailed documentation and basic usage examples, please see the documentation at godoc.org.

Sample Programs

Snowflake provides a set of sample programs to test with. Set the environment variable $GOPATH to the top directory of your workspace, e.g., ~/go and make certain to include $GOPATH/bin in the environment variable $PATH. Run the make command to build all sample programs.

make install

In the following example, the program select1.go is built and installed in $GOPATH/bin and can be run from the command line:

SNOWFLAKE_TEST_ACCOUNT=<your_account> \
SNOWFLAKE_TEST_USER=<your_user> \
SNOWFLAKE_TEST_PASSWORD=<your_password> \
select1
Congrats! You have successfully run SELECT 1 with Snowflake DB!

Development

The developer notes are hosted with the source code on GitHub.

Testing Code

Set the Snowflake connection info in parameters.json:

{
    "testconnection": {
        "SNOWFLAKE_TEST_USER":      "<your_user>",
        "SNOWFLAKE_TEST_PASSWORD":  "<your_password>",
        "SNOWFLAKE_TEST_ACCOUNT":   "<your_account>",
        "SNOWFLAKE_TEST_WAREHOUSE": "<your_warehouse>",
        "SNOWFLAKE_TEST_DATABASE":  "<your_database>",
        "SNOWFLAKE_TEST_SCHEMA":    "<your_schema>",
        "SNOWFLAKE_TEST_ROLE":      "<your_role>"
    }
}

Install jq so that the parameters can get parsed correctly, and run make test in your Go development environment:

make test

Capturing Code Coverage

Configure your testing environment as described above and run make cov. The coverage percentage will be printed on the console when the testing completes.

make cov

For more detailed analysis, results are printed to coverage.txt in the project directory.

To read the coverage report, run:

go tool cover -html=coverage.txt

Submitting Pull Requests

You may use your preferred editor to edit the driver code. Make certain to run make fmt lint before submitting any pull request to Snowflake. This command formats your source code according to the standard Go style and detects any coding style issues.

Runaway dbus-daemon processes on certain OS

This only affects certain Linux distributions, one of them is confirmed to be RHEL. Due to a bug in one of the dependencies (keyring), on the affected OS, each invocation of a program depending on gosnowflake (or any other program depending on the same keyring), will generate a new instance of dbus-daemon fork which can, due to not being cleaned up, eventually fill the fd limits.

Until we replace the offending dependency with one which doesn't have the bug, a workaround needs to be applied, which can be:

  • cleaning up the runaway processes periodically
  • setting envvar DBUS_SESSION_BUS_ADDRESS=$XDG_RUNTIME_DIR/bus (if that socket exists, or create it) or even DBUS_SESSION_BUS_ADDRESS=/dev/null

The driver will try to detect automatically, whether your runtime is susceptible for this bug or not, and if so, log a message on Warning loglevel.

Details in issue 773

# Packages

No description provided by the author
No description provided by the author

# Functions

Array takes in a column of a row to be inserted via array binding, bulk or otherwise, and converts it into a native snowflake type for binding.
CreateDefaultLogger return a new instance of SFLogger with default config.
DSN constructs a DSN for Snowflake db.
GetConfigFromEnv is used to parse the environment variable values to specific fields of the Config.
GetFromEnv is used to get the value of an environment variable from the system.
GetLogger return logger that is not public.
Location returns an offset (minutes) based Location object for Snowflake database.
LocationWithOffsetString returns an offset based Location object.
NewConnector creates a new connector with the given SnowflakeDriver and Config.
NewUUID creates a new snowflake UUID.
ParseDSN parses the DSN string to a Config.
ParseUUID parses a string of xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx into its UUID form.
ScanSnowflakeParameter binds SnowflakeParameter variable with an array of column buffer.
SetLogger set a new logger of SFLogger interface for gosnowflake.
SFCallerPrettyfier to provide base file name and function name from calling frame used in SFLogger.
WithArrowAllocator returns a context embedding the provided allocator which will be utilized by chunk downloaders when constructing Arrow objects.
WithArrowBatches returns a context that allows users to retrieve arrow.Record download workers upon querying.
WithArrowBatchesTimestampOption in combination with WithArrowBatches returns a context that allows users to retrieve arrow.Record with different timestamp options.
WithArrowBatchesUtf8Validation in combination with WithArrowBatches returns a context that will validate and replace invalid UTF-8 characters in string columns with the replacement character Theoretically, this should not be necessary, because arrow string column is only intended to contain valid UTF-8 characters.
WithAsyncMode returns a context that allows execution of query in async mode.
WithDescribeOnly returns a context that enables a describe only query.
WithFetchResultByID returns a context that allows retrieving the result by query ID.
WithFileStream returns a context that contains the address of the file stream to be PUT.
WithFileTransferOptions returns a context that contains the address of file transfer options.
WithHigherPrecision returns a context that enables higher precision by returning a *big.Int or *big.Float variable when querying rows for column types with numbers that don't fit into its native Golang counterpart When used in combination with WithArrowBatches, original BigDecimal in arrow batches will be preserved.
WithMultiStatement returns a context that allows the user to execute the desired number of sql queries in one query.
WithOriginalTimestamp in combination with WithArrowBatches returns a context that allows users to retrieve arrow.Record with original timestamp struct returned by Snowflake.
WithQueryIDChan returns a context that contains the channel to receive the query ID.
WithQueryTag returns a context that will set the given tag as the QUERY_TAG parameter on any queries that are run.
WithRequestID returns a new context with the specified snowflake request id.
WithStreamDownloader returns a context that allows the use of a stream based chunk downloader.

# Constants

AuthTypeExternalBrowser is to use a browser to access an Fed and perform SSO authentication.
AuthTypeJwt is to use Jwt to perform authentication.
AuthTypeOAuth is the OAuth authentication.
AuthTypeOkta is to use a native okta URL to perform SSO authentication on Okta.
AuthTypeSnowflake is the general username password authentication.
AuthTypeTokenAccessor is to use the provided token accessor and bypass authentication.
AuthTypeUsernamePasswordMFA is to use username and password with mfa.
ConfigBoolFalse represents false for the config field.
ConfigBoolTrue represents true for the config field.
DateType denotes a date type for array binds.
ErrAsync is an error code for an unknown async error.
ErrBindSerialization is an error code for a failed serialization of bind variables.
ErrBindUpload is an error code for the uploading process of bind elements to the stage.
ErrCodeClientConfigFailed is an error code for the case where clientConfigFile is invalid or applying client configuration fails.
ErrCodeEmptyAccountCode is an error code for the case where a DNS doesn't include account parameter.
ErrCodeEmptyPasswordCode is an error code for the case where a DNS doesn't include password parameter.
ErrCodeEmptyUsernameCode is an error code for the case where a DNS doesn't include user parameter.
ErrCodeFailedToConnect is an error code for the case where a DB connection failed due to wrong account name.
ErrCodeFailedToParseAuthenticator is an error code for the case where a DNS includes an invalid authenticator.
ErrCodeFailedToParseHost is an error code for the case where a DNS includes an invalid host name.
ErrCodeFailedToParsePort is an error code for the case where a DNS includes an invalid port number.
ErrCodeIdpConnectionError is an error code for the case where a IDP connection failed.
ErrCodePrivateKeyParseError is an error code for the case where the private key is not parsed correctly.
ErrCodeRegionOverlap is an error code for the case where a region is specified despite an account region present.
ErrCodeServiceUnavailable is an error code for the case where service is unavailable.
ErrCodeSSOURLNotMatch is an error code for the case where a SSO URL doesn't match.
ErrCommandNotRecognized is an error code denoting the PUT/GET command was not recognized.
ErrCompressionNotSupported is an error code denoting the user specified compression type is not supported.
ErrFailedToAuth is an error code for the case where authentication failed for unknown reason.
ErrFailedToAuthOKTA is an error code for the case where authentication via OKTA failed for unknown reason.
ErrFailedToAuthSAML is an error code for the case where authentication via SAML failed for unknown reason.
ErrFailedToCancelQuery is an error code for the case where cancel query failed.
ErrFailedToCloseSession is an error code for the case where close session failed.
ErrFailedToConvertToS3Client is an error code denoting the failure of an interface to s3.Client conversion.
ErrFailedToDownloadFromStage is an error code denoting the failure to download a file from the stage.
ErrFailedToGetChunk is an error code for the case where it failed to get chunk of result set.
ErrFailedToGetExternalBrowserResponse is an error code for when there's an error reading from the open socket.
ErrFailedToGetSSO is an error code for the case where authentication via OKTA failed for unknown reason.
ErrFailedToHeartbeat is an error code when a heartbeat fails.
ErrFailedToParseResponse is an error code for when we cannot parse an external browser response from Snowflake.
ErrFailedToPostQuery is an error code for the case where HTTP POST failed.
ErrFailedToRenewSession is an error code for the case where session renewal failed.
ErrFailedToUploadToStage is an error code denoting the failure to upload a file to the stage.
ErrFileNotExists is an error code denoting the file to be transferred does not exist.
ErrInternalNotMatchEncryptMaterial is an error code denoting the encryption material specified does not match.
ErrInvalidBinaryHexForm is an error code for the case where a binary data in hex form is invalid.
ErrInvalidOffsetStr is an error code for the case where a offset string is invalid.
ErrInvalidPadding is an error code denoting the invalid padding of decryption key.
ErrInvalidStageFs is an error code denoting an invalid stage in the file system.
ErrInvalidStageLocation is an error code denoting an invalid stage location.
ErrInvalidTimestampTz is an error code for the case where a returned TIMESTAMP_TZ internal value is invalid.
ErrLocalPathNotDirectory is an error code denoting a local path that is not a directory.
ErrNoDefaultTransactionIsolationLevel is an error code for the case where non default isolation level is specified.
ErrNoReadOnlyTransaction is an error code for the case where readonly mode is specified.
ErrNoResultIDs is an error code for empty result IDs for multi statement queries.
ErrNotImplemented is an error code denoting the file transfer feature is not implemented.
ErrObjectNotExistOrAuthorized is a GS error code for the case that the server-side object specified does not exist.
ErrOCSPInvalidValidity is an error code for the case where the OCSP response validity is invalid.
ErrOCSPNoOCSPResponderURL is an error code for the case where the OCSP responder URL is not attached.
ErrOCSPStatusRevoked is an error code for the case where the certificate is revoked.
ErrOCSPStatusUnknown is an error code for the case where the certificate revocation status is unknown.
ErrQueryIDFormat the query ID given to fetch its result is not valid.
ErrQueryIsRunning the query is still running.
ErrQueryReportedError server side reports the query failed with error.
ErrQueryStatus when check the status of a query, receive error or no status.
ErrRoleNotExist is a GS error code for the case that the role specified does not exist.
ErrSessionGone is an GS error code for the case that session is already closed.
ErrTooHighTimestampPrecision is an error code for the case where cannot convert Snowflake timestamp to arrow.Timestamp.
OCSPFailOpenFalse represents OCSP fail closed mode.
OCSPFailOpenTrue represents OCSP fail open mode.
QueryFailed denotes a failed query.
QueryStatusComplete denotes a completed query execution.
QueryStatusInProgress denotes a query execution in progress.
Query Status defined at server side.
Query Status defined at server side.
SFQueryBlocked is when a statement is waiting on a lock on resource held by another statement.
Query Status defined at server side.
Query Status defined at server side.
Query Status defined at server side.
Query Status defined at server side.
Query Status defined at server side.
SFQueryQueueRepairingWarehouse present in QueryDTO.java.
Query Status defined at server side.
Query Status defined at server side.
Query Status defined at server side.
Query Status defined at server side.
SFSessionIDKey is context key of session id.
SFSessionUserKey is context key of user id of a session.
SnowflakeGoDriverVersion is the version of Go Snowflake Driver.
SQLStateConnectionFailure is a SQL State code indicating connection failed.
SQLStateConnectionRejected is a SQL State code indicating connection was rejected.
SQLStateConnectionWasNotEstablished is a SQL State code indicating connection was not established.
SQLStateFeatureNotSupported is a SQL State code indicating the feature is not enabled.
SQLStateInvalidDataTimeFormat is a SQL State code indicating DataTime format is invalid.
SQLStateNumericValueOutOfRange is a SQL State code indicating Numeric value is out of range.
TimestampLTZType denotes a LTZ timezoneType for array binds.
TimestampNTZType denotes a NTZ timezoneType for array binds.
TimestampTZType denotes a TZ timezoneType for array binds.
TimeType denotes a time type for array binds.
UseMicrosecondTimestamp uses arrow.Timestamp in microsecond precision.
UseMillisecondTimestamp uses arrow.Timestamp in millisecond precision.
UseNanosecondTimestamp uses arrow.Timestamp in nanosecond precision, could cause ErrTooHighTimestampPrecision if arrow.Timestamp cannot fit original timestamp values.
UseOriginalTimestamp uses original timestamp struct returned by Snowflake.
UseSecondTimestamp uses arrow.Timestamp in second precision.

# Variables

CustomJSONDecoderEnabled has the chunk downloader use the custom JSON decoder to reduce memory footprint.
DataTypeArray is a ARRAY datatype.
DataTypeBinary is a BINARY datatype.
DataTypeBoolean is a BOOLEAN datatype.
DataTypeDate is a Date datatype.
DataTypeFixed is a FIXED datatype.
DataTypeObject is a OBJECT datatype.
DataTypeReal is a REAL datatype.
DataTypeText is a TEXT datatype.
DataTypeTime is a TIME datatype.
DataTypeTimestampLtz is a TIMESTAMP_LTZ datatype.
DataTypeTimestampNtz is a TIMESTAMP_NTZ datatype.
DataTypeTimestampTz is a TIMESTAMP_TZ datatype.
DataTypeVariant is a TEXT datatype.
LogKeys these keys in context should be included in logging messages when using logger.WithContext.
MaxChunkDownloadWorkers specifies the maximum number of goroutines used to download chunks.
SnowflakeTransport includes the certificate revocation check with OCSP in sequential.
SnowflakeTransportTest includes the certificate revocation check in parallel.

# Structs

ArrowBatch object represents a chunk of data, or subset of rows, retrievable in arrow.Record format.
ArrowStreamBatch is a type describing a potentially yet-to-be-downloaded Arrow IPC stream.
ClientConfig config root.
ClientConfigCommonProps properties from "common" section.
Config is a set of configuration parameters.
ConfigParam is used to bind the name of the Config field with the environment variable and set the requirement for it.
Connector creates Driver with the specified Config.
SnowflakeDriver is a context of Go Driver.
SnowflakeError is a error type including various Snowflake specific information.
SnowflakeFileTransferOptions enables users to specify options regarding files transfers such as PUT/GET.
SnowflakeParameter includes the columns output from SHOW PARAMETER command.
SnowflakeQueryStatus is the query status metadata of a snowflake query.
TypedNullTime is required to properly bind the null value with the snowflakeType as the Snowflake functions require the type of the field to be provided explicitly for the null values.

# Interfaces

ArrowStreamLoader is a convenience interface for downloading Snowflake results via multiple Arrow Record Batch streams.
InternalClient is implemented by HTTPClient.
InternalSnowflakeDriver is the interface for an internal Snowflake driver.
SFLogger Snowflake logger interface to expose FieldLogger defined in logrus.
SnowflakeConnection is a wrapper to snowflakeConn that exposes API functions.
SnowflakeResult provides an API for methods exposed to the clients.
SnowflakeRows provides an API for methods exposed to the clients.
SnowflakeStmt represents the prepared statement in driver.
TokenAccessor manages the session token and master token.

# Type aliases

AuthType indicates the type of authentication in Snowflake.
ConfigBool is a type to represent true or false in the Config.
OCSPFailOpenMode is OCSP fail open mode.
UUID is a RFC4122 compliant uuid type.