package
0.3.7
Repository: https://github.com/aliyun/aliyun-odps-go-sdk.git
Documentation: pkg.go.dev

# README

Apache Arrow for Go

GoDoc

Apache Arrow is a cross-language development platform for in-memory data. It specifies a standardized language-independent columnar memory format for flat and hierarchical data, organized for efficient analytic operations on modern hardware. It also provides computational libraries and zero-copy streaming messaging and inter-process communication.

Reference Counting

The library makes use of reference counting so that it can track when memory buffers are no longer used. This allows Arrow to update resource accounting, pool memory such and track overall memory usage as objects are created and released. Types expose two methods to deal with this pattern. The Retain method will increase the reference count by 1 and Release method will reduce the count by 1. Once the reference count of an object is zero, any associated object will be freed. Retain and Release are safe to call from multiple goroutines.

When to call Retain / Release?

  • If you are passed an object and wish to take ownership of it, you must call Retain. You must later pair this with a call to Release when you no longer need the object. "Taking ownership" typically means you wish to access the object outside the scope of the current function call.

  • You own any object you create via functions whose name begins with New or Copy or when receiving an object over a channel. Therefore you must call Release once you no longer need the object.

  • If you send an object over a channel, you must call Retain before sending it as the receiver is assumed to own the object and will later call Release when it no longer needs the object.

Performance

The arrow package makes extensive use of c2goasm to leverage LLVM's advanced optimizer and generate PLAN9 assembly functions from C/C++ code. The arrow package can be compiled without these optimizations using the noasm build tag. Alternatively, by configuring an environment variable, it is possible to dynamically configure which architecture optimizations are used at runtime. See the cpu package README for a description of this environment variable.

Example Usage

The following benchmarks demonstrate summing an array of 8192 values using various optimizations.

Disable no architecture optimizations (thus using AVX2):

$ INTEL_DISABLE_EXT=NONE go test -bench=8192 -run=. ./math
goos: darwin
goarch: amd64
pkg: github.com/apache/arrow/go/arrow/math
BenchmarkFloat64Funcs_Sum_8192-8   	 2000000	       687 ns/op	95375.41 MB/s
BenchmarkInt64Funcs_Sum_8192-8     	 2000000	       719 ns/op	91061.06 MB/s
BenchmarkUint64Funcs_Sum_8192-8    	 2000000	       691 ns/op	94797.29 MB/s
PASS
ok  	github.com/apache/arrow/go/arrow/math	6.444s

NOTE: NONE is simply ignored, thus enabling optimizations for AVX2 and SSE4


Disable AVX2 architecture optimizations:

$ INTEL_DISABLE_EXT=AVX2 go test -bench=8192 -run=. ./math
goos: darwin
goarch: amd64
pkg: github.com/apache/arrow/go/arrow/math
BenchmarkFloat64Funcs_Sum_8192-8   	 1000000	      1912 ns/op	34263.63 MB/s
BenchmarkInt64Funcs_Sum_8192-8     	 1000000	      1392 ns/op	47065.57 MB/s
BenchmarkUint64Funcs_Sum_8192-8    	 1000000	      1405 ns/op	46636.41 MB/s
PASS
ok  	github.com/apache/arrow/go/arrow/math	4.786s

Disable ALL architecture optimizations, thus using pure Go implementation:

$ INTEL_DISABLE_EXT=ALL go test -bench=8192 -run=. ./math
goos: darwin
goarch: amd64
pkg: github.com/apache/arrow/go/arrow/math
BenchmarkFloat64Funcs_Sum_8192-8   	  200000	     10285 ns/op	6371.41 MB/s
BenchmarkInt64Funcs_Sum_8192-8     	  500000	      3892 ns/op	16837.37 MB/s
BenchmarkUint64Funcs_Sum_8192-8    	  500000	      3929 ns/op	16680.00 MB/s
PASS
ok  	github.com/apache/arrow/go/arrow/math	6.179s

# Packages

Package array provides implementations of various Arrow array types.
Package arrio exposes functions to manipulate records, exposing and using interfaces not unlike the ones defined in the stdlib io package.
No description provided by the author
Package csv reads CSV files and presents the extracted data as records, also writes data as record into CSV files.
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
Package math provides optimized mathematical functions for processing Arrow arrays.
Package memory provides support for allocating and manipulating memory at a low level.
No description provided by the author
Package tensor provides types that implement n-dimensional arrays.

# Functions

CheckMetadata is an option for TypeEqual that allows checking for metadata equality besides type equality.
FixedSizeListOf returns the list type with element type t.
No description provided by the author
FixedSizeListOfNonNullable is like FixedSizeListOf but NullableElem defaults to false indicating that the child type should be marked as non-nullable.
GetExtensionType retrieves and returns the extension type of the given name from the global extension type registry.
No description provided by the author
ListOf returns the list type with element type t.
No description provided by the author
ListOfNonNullable is like ListOf but NullableElem defaults to false, indicating that the child type should be marked as non-nullable.
No description provided by the author
No description provided by the author
No description provided by the author
NewSchema returns a new Schema value from the slice of fields and metadata.
RegisterExtensionType registers the provided ExtensionType by calling ExtensionName to use as a Key for registrying the type.
StructOf returns the struct type with fields fs.
TypeEqual checks if two DataType are the same, optionally checking metadata equality for STRUCT types.
UnregisterExtensionType removes the type with the given name from the registry causing any messages with that type which come in to be expressed with their metadata and underlying type instead of the extension type that isn't known.

# Constants

BINARY is a Variable-length byte type (no guarantee of UTF8-ness).
BOOL is a 1 bit, LSB bit-packed ordering.
DATE32 is int32 days since the UNIX epoch.
Date32SizeBytes specifies the number of bytes required to store a single Date32 in memory.
DATE64 is int64 milliseconds since the UNIX epoch.
Date64SizeBytes specifies the number of bytes required to store a single Date64 in memory.
DayTimeIntervalSizeBytes specifies the number of bytes required to store a single DayTimeInterval in memory.
Alias to ensure we do not break any consumers.
DECIMAL128 is a precision- and scale-based decimal type.
Decimal128SizeBytes specifies the number of bytes required to store a single decimal128 in memory.
DECIMAL256 is a precision and scale based decimal type, with 256 bit max.
DENSE_UNION of logical types.
DICTIONARY aka Category type.
Measure of elapsed time in either seconds, milliseconds, microseconds or nanoseconds.
DurationSizeBytes specifies the number of bytes required to store a single Duration in memory.
Custom data type, implemented by user.
FIXED_SIZE_BINARY is a binary where each value occupies the same number of bytes.
Fixed size list of some logical type.
FLOAT16 is a 2-byte floating point value.
Float16SizeBytes specifies the number of bytes required to store a single float16 in memory.
FLOAT32 is a 4-byte floating point value.
Float32SizeBytes specifies the number of bytes required to store a single float32 in memory.
FLOAT64 is an 8-byte floating point value.
Float64SizeBytes specifies the number of bytes required to store a single float64 in memory.
INT16 is a Signed 16-bit little-endian integer.
Int16SizeBytes specifies the number of bytes required to store a single int16 in memory.
INT32 is a Signed 32-bit little-endian integer.
Int32SizeBytes specifies the number of bytes required to store a single int32 in memory.
INT64 is a Signed 64-bit little-endian integer.
Int64SizeBytes specifies the number of bytes required to store a single int64 in memory.
INT8 is a Signed 8-bit little-endian integer.
Int8SizeBytes specifies the number of bytes required to store a single int8 in memory.
INTERVAL could be any of the interval types, kept to avoid breaking anyone after switching to individual type ids for the interval types that were using it when calling MakeFromData or NewBuilder Deprecated and will be removed in the next major version release.
INTERVAL_DAY_TIME is DAY_TIME in SQL Style.
calendar interval with three fields.
INTERVAL_MONTHS is YEAR_MONTH interval in SQL style.
like BINARY but with 64-bit offsets, not yet implemented.
like LIST but with 64-bit offsets.
like STRING, but 64-bit offsets.
LIST is a list of some logical data type.
MAP is a repeated struct logical type.
No description provided by the author
No description provided by the author
MonthDayNanoIntervalSizeBytes specifies the number of bytes required to store a single DayTimeInterval in memory.
MonthIntervalSizeBytes specifies the number of bytes required to store a single MonthInterval in memory.
No description provided by the author
NULL type having no physical storage.
No description provided by the author
SPARSE_UNION of logical types.
STRING is a UTF8 variable-length string.
STRUCT of logical types.
TIME32 is a signed 32-bit integer, representing either seconds or milliseconds since midnight.
Time32SizeBytes specifies the number of bytes required to store a single Time32 in memory.
TIME64 is a signed 64-bit integer, representing either microseconds or nanoseconds since midnight.
Time64SizeBytes specifies the number of bytes required to store a single Time64 in memory.
TIMESTAMP is an exact timestamp encoded with int64 since UNIX epoch Default unit millisecond.
TimestampSizeBytes specifies the number of bytes required to store a single Timestamp in memory.
UINT16 is an Unsigned 16-bit little-endian integer.
Uint16SizeBytes specifies the number of bytes required to store a single uint16 in memory.
UINT32 is an Unsigned 32-bit little-endian integer.
Uint32SizeBytes specifies the number of bytes required to store a single uint32 in memory.
UINT64 is an Unsigned 64-bit little-endian integer.
Uint64SizeBytes specifies the number of bytes required to store a single uint64 in memory.
UINT8 is an Unsigned 8-bit little-endian integer.
Uint8SizeBytes specifies the number of bytes required to store a single uint8 in memory.

# Variables

No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
Decimal128 traits.
No description provided by the author
No description provided by the author
Float16 traits.
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author

# Structs

No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
DayTimeInterval represents a number of days and milliseconds (fraction of day).
DayTimeIntervalType is encoded as a pair of 32-bit signed integer, representing a number of days and milliseconds (fraction of day).
Decimal128Type represents a fixed-size 128-bit decimal type.
DurationType is encoded as a 64-bit signed integer, representing an amount of elapsed time without any relation to a calendar artifact.
ExtensionBase is the base struct for user-defined Extension Types which must be embedded in any user-defined types like so: type UserDefinedType struct { arrow.ExtensionBase // any other data } .
No description provided by the author
No description provided by the author
FixedSizeListType describes a nested type in which each array slot contains a fixed-size sequence of values, all having the same relative type.
Float16Type represents a floating point value encoded with a 16-bit precision.
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
ListType describes a nested type in which each array slot contains a variable-size sequence of values, all having the same relative type.
No description provided by the author
No description provided by the author
MonthDayNanoInterval represents a number of months, days and nanoseconds (fraction of day).
MonthDayNanoIntervalType is encoded as two signed 32-bit integers representing a number of months and a number of days, followed by a 64-bit integer representing the number of nanoseconds since midnight for fractions of a day.
MonthIntervalType is encoded as a 32-bit signed integer, representing a number of months.
NullType describes a degenerate array, with zero physical storage.
Schema is a sequence of Field values, describing the columns of a table or a record batch.
No description provided by the author
StructType describes a nested type parameterized by an ordered sequence of relative types, called its fields.
Time32Type is encoded as a 32-bit signed integer, representing either seconds or milliseconds since midnight.
Time64Type is encoded as a 64-bit signed integer, representing either microseconds or nanoseconds since midnight.
TimestampType is encoded as a 64-bit signed integer since the UNIX epoch (2017-01-01T00:00:00Z).
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author

# Interfaces

No description provided by the author
DataType is the representation of an Arrow type.
ExtensionType is an interface for handling user-defined types.
FixedWidthDataType is the representation of an Arrow type that requires a fixed number of bits in memory for each element.

# Type aliases

No description provided by the author
No description provided by the author
No description provided by the author
MonthInterval represents a number of months.
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
Type is a logical type.
TypeEqualOption is a functional option type used for configuring type equality checks.