AWS Glue Construct Library

The APIs of higher level constructs in this module are experimental and under active development. They are subject to non-backward compatible changes or removal in any future version. These are not subject to the Semantic Versioning model and breaking changes will be announced in the release notes. This means that while you may use them, you may need to update your source code when upgrading to a newer version of this package.

This module is part of the AWS Cloud Development Kit project.

Job

A Job encapsulates a script that connects to data sources, processes them, and then writes output to a data target.

There are 3 types of jobs supported by AWS Glue: Spark ETL, Spark Streaming, and Python Shell jobs.

The glue.JobExecutable allows you to specify the type of job, the language to use and the code assets required by the job.

glue.Code allows you to refer to the different code assets required by the job, either from an existing S3 location or from a local file path.

glue.ExecutionClass allows you to specify FLEX or STANDARD. FLEX is appropriate for non-urgent jobs such as pre-production jobs, testing, and one-time data loads.

Spark Jobs

These jobs run in an Apache Spark environment managed by AWS Glue.

ETL Jobs

An ETL job processes data in batches using Apache Spark.

var bucket bucket

glue.NewJob(this, jsii.String("ScalaSparkEtlJob"), &JobProps{
	Executable: glue.JobExecutable_ScalaEtl(&ScalaJobExecutableProps{
		GlueVersion: glue.GlueVersion_V4_0(),
		Script: glue.Code_FromBucket(bucket, jsii.String("src/com/example/HelloWorld.scala")),
		ClassName: jsii.String("com.example.HelloWorld"),
		ExtraJars: []code{
			glue.*code_*FromBucket(bucket, jsii.String("jars/HelloWorld.jar")),
		},
	}),
	WorkerType: glue.WorkerType_G_8X(),
	Description: jsii.String("an example Scala ETL job"),
})

Streaming Jobs

A Streaming job is similar to an ETL job, except that it performs ETL on data streams. It uses the Apache Spark Structured Streaming framework. Some Spark job features are not available to streaming ETL jobs.

glue.NewJob(this, jsii.String("PythonSparkStreamingJob"), &JobProps{
	Executable: glue.JobExecutable_PythonStreaming(&PythonSparkJobExecutableProps{
		GlueVersion: glue.GlueVersion_V4_0(),
		PythonVersion: glue.PythonVersion_THREE,
		Script: glue.Code_FromAsset(path.join(__dirname, jsii.String("job-script"), jsii.String("hello_world.py"))),
	}),
	Description: jsii.String("an example Python Streaming job"),
})

Python Shell Jobs

A Python shell job runs Python scripts as a shell and supports a Python version that depends on the AWS Glue version you are using. This can be used to schedule and run tasks that don't require an Apache Spark environment. Currently, three flavors are supported:

• PythonVersion.TWO (2.7; EOL)
• PythonVersion.THREE (3.6)
• PythonVersion.THREE_NINE (3.9)

var bucket bucket

glue.NewJob(this, jsii.String("PythonShellJob"), &JobProps{
	Executable: glue.JobExecutable_PythonShell(&PythonShellExecutableProps{
		GlueVersion: glue.GlueVersion_V1_0(),
		PythonVersion: glue.PythonVersion_THREE,
		Script: glue.Code_FromBucket(bucket, jsii.String("script.py")),
	}),
	Description: jsii.String("an example Python Shell job"),
})

Ray Jobs

These jobs run in a Ray environment managed by AWS Glue.

glue.NewJob(this, jsii.String("RayJob"), &JobProps{
	Executable: glue.JobExecutable_PythonRay(&PythonRayExecutableProps{
		GlueVersion: glue.GlueVersion_V4_0(),
		PythonVersion: glue.PythonVersion_THREE_NINE,
		Runtime: glue.Runtime_RAY_TWO_FOUR(),
		Script: glue.Code_FromAsset(path.join(__dirname, jsii.String("job-script"), jsii.String("hello_world.py"))),
	}),
	WorkerType: glue.WorkerType_Z_2X(),
	WorkerCount: jsii.Number(2),
	Description: jsii.String("an example Ray job"),
})

Enable Spark UI

Enable Spark UI setting the sparkUI property.

glue.NewJob(this, jsii.String("EnableSparkUI"), &JobProps{
	JobName: jsii.String("EtlJobWithSparkUIPrefix"),
	SparkUI: &SparkUIProps{
		Enabled: jsii.Boolean(true),
	},
	Executable: glue.JobExecutable_PythonEtl(&PythonSparkJobExecutableProps{
		GlueVersion: glue.GlueVersion_V3_0(),
		PythonVersion: glue.PythonVersion_THREE,
		Script: glue.Code_FromAsset(path.join(__dirname, jsii.String("job-script"), jsii.String("hello_world.py"))),
	}),
})

The sparkUI property also allows the specification of an s3 bucket and a bucket prefix.

See documentation for more information on adding jobs in Glue.

Enable Job Run Queuing

AWS Glue job queuing monitors your account level quotas and limits. If quotas or limits are insufficient to start a Glue job run, AWS Glue will automatically queue the job and wait for limits to free up. Once limits become available, AWS Glue will retry the job run. Glue jobs will queue for limits like max concurrent job runs per account, max concurrent Data Processing Units (DPU), and resource unavailable due to IP address exhaustion in Amazon Virtual Private Cloud (Amazon VPC).

Enable job run queuing by setting the jobRunQueuingEnabled property to true.

glue.NewJob(this, jsii.String("EnableRunQueuing"), &JobProps{
	JobName: jsii.String("EtlJobWithRunQueuing"),
	Executable: glue.JobExecutable_PythonEtl(&PythonSparkJobExecutableProps{
		GlueVersion: glue.GlueVersion_V4_0(),
		PythonVersion: glue.PythonVersion_THREE,
		Script: glue.Code_FromAsset(path.join(__dirname, jsii.String("job-script"), jsii.String("hello_world.py"))),
	}),
	JobRunQueuingEnabled: jsii.Boolean(true),
})

Connection

A Connection allows Glue jobs, crawlers and development endpoints to access certain types of data stores. For example, to create a network connection to connect to a data source within a VPC:

var securityGroup securityGroup
var subnet subnet

glue.NewConnection(this, jsii.String("MyConnection"), &ConnectionProps{
	Type: glue.ConnectionType_NETWORK(),
	// The security groups granting AWS Glue inbound access to the data source within the VPC
	SecurityGroups: []iSecurityGroup{
		securityGroup,
	},
	// The VPC subnet which contains the data source
	Subnet: Subnet,
})

For RDS Connection by JDBC, it is recommended to manage credentials using AWS Secrets Manager. To use Secret, specify SECRET_ID in properties like the following code. Note that in this case, the subnet must have a route to the AWS Secrets Manager VPC endpoint or to the AWS Secrets Manager endpoint through a NAT gateway.

var securityGroup securityGroup
var subnet subnet
var db databaseCluster

glue.NewConnection(this, jsii.String("RdsConnection"), &ConnectionProps{
	Type: glue.ConnectionType_JDBC(),
	SecurityGroups: []iSecurityGroup{
		securityGroup,
	},
	Subnet: Subnet,
	Properties: map[string]*string{
		"JDBC_CONNECTION_URL": fmt.Sprintf("jdbc:mysql://%v/databasename", db.clusterEndpoint.socketAddress),
		"JDBC_ENFORCE_SSL": jsii.String("false"),
		"SECRET_ID": db.secret.secretName,
	},
})

If you need to use a connection type that doesn't exist as a static member on ConnectionType, you can instantiate a ConnectionType object, e.g: new glue.ConnectionType('NEW_TYPE').

See Adding a Connection to Your Data Store and Connection Structure documentation for more information on the supported data stores and their configurations.

SecurityConfiguration

A SecurityConfiguration is a set of security properties that can be used by AWS Glue to encrypt data at rest.

glue.NewSecurityConfiguration(this, jsii.String("MySecurityConfiguration"), &SecurityConfigurationProps{
	CloudWatchEncryption: &CloudWatchEncryption{
		Mode: glue.CloudWatchEncryptionMode_KMS,
	},
	JobBookmarksEncryption: &JobBookmarksEncryption{
		Mode: glue.JobBookmarksEncryptionMode_CLIENT_SIDE_KMS,
	},
	S3Encryption: &S3Encryption{
		Mode: glue.S3EncryptionMode_KMS,
	},
})

By default, a shared KMS key is created for use with the encryption configurations that require one. You can also supply your own key for each encryption config, for example, for CloudWatch encryption:

var key key

glue.NewSecurityConfiguration(this, jsii.String("MySecurityConfiguration"), &SecurityConfigurationProps{
	CloudWatchEncryption: &CloudWatchEncryption{
		Mode: glue.CloudWatchEncryptionMode_KMS,
		KmsKey: key,
	},
})

See documentation for more info for Glue encrypting data written by Crawlers, Jobs, and Development Endpoints.

Database

A Database is a logical grouping of Tables in the Glue Catalog.

glue.NewDatabase(this, jsii.String("MyDatabase"), &DatabaseProps{
	DatabaseName: jsii.String("my_database"),
	Description: jsii.String("my_database_description"),
})

Table

A Glue table describes a table of data in S3: its structure (column names and types), location of data (S3 objects with a common prefix in a S3 bucket), and format for the files (Json, Avro, Parquet, etc.):

var myDatabase database

glue.NewS3Table(this, jsii.String("MyTable"), &S3TableProps{
	Database: myDatabase,
	Columns: []column{
		&column{
			Name: jsii.String("col1"),
			Type: glue.Schema_STRING(),
		},
		&column{
			Name: jsii.String("col2"),
			Type: glue.Schema_Array(glue.Schema_STRING()),
			Comment: jsii.String("col2 is an array of strings"),
		},
	},
	DataFormat: glue.DataFormat_JSON(),
})

By default, a S3 bucket will be created to store the table's data but you can manually pass the bucket and s3Prefix:

var myBucket bucket
var myDatabase database

glue.NewS3Table(this, jsii.String("MyTable"), &S3TableProps{
	Bucket: myBucket,
	S3Prefix: jsii.String("my-table/"),
	// ...
	Database: myDatabase,
	Columns: []column{
		&column{
			Name: jsii.String("col1"),
			Type: glue.Schema_STRING(),
		},
	},
	DataFormat: glue.DataFormat_JSON(),
})

Glue tables can be configured to contain user-defined properties, to describe the physical storage of table data, through the storageParameters property:

var myDatabase database

glue.NewS3Table(this, jsii.String("MyTable"), &S3TableProps{
	StorageParameters: []storageParameter{
		glue.*storageParameter_SkipHeaderLineCount(jsii.Number(1)),
		glue.*storageParameter_CompressionType(glue.CompressionType_GZIP),
		glue.*storageParameter_Custom(jsii.String("separatorChar"), jsii.String(",")),
	},
	// ...
	Database: myDatabase,
	Columns: []column{
		&column{
			Name: jsii.String("col1"),
			Type: glue.Schema_STRING(),
		},
	},
	DataFormat: glue.DataFormat_JSON(),
})

Glue tables can also be configured to contain user-defined table properties through the parameters property:

var myDatabase database

glue.NewS3Table(this, jsii.String("MyTable"), &S3TableProps{
	Parameters: map[string]*string{
		"key1": jsii.String("val1"),
		"key2": jsii.String("val2"),
	},
	Database: myDatabase,
	Columns: []column{
		&column{
			Name: jsii.String("col1"),
			Type: glue.Schema_STRING(),
		},
	},
	DataFormat: glue.DataFormat_JSON(),
})

Partition Keys

To improve query performance, a table can specify partitionKeys on which data is stored and queried separately. For example, you might partition a table by year and month to optimize queries based on a time window:

var myDatabase database

glue.NewS3Table(this, jsii.String("MyTable"), &S3TableProps{
	Database: myDatabase,
	Columns: []column{
		&column{
			Name: jsii.String("col1"),
			Type: glue.Schema_STRING(),
		},
	},
	PartitionKeys: []*column{
		&column{
			Name: jsii.String("year"),
			Type: glue.Schema_SMALL_INT(),
		},
		&column{
			Name: jsii.String("month"),
			Type: glue.Schema_SMALL_INT(),
		},
	},
	DataFormat: glue.DataFormat_JSON(),
})

Partition Indexes

Another way to improve query performance is to specify partition indexes. If no partition indexes are present on the table, AWS Glue loads all partitions of the table and filters the loaded partitions using the query expression. The query takes more time to run as the number of partitions increase. With an index, the query will try to fetch a subset of the partitions instead of loading all partitions of the table.

The keys of a partition index must be a subset of the partition keys of the table. You can have a maximum of 3 partition indexes per table. To specify a partition index, you can use the partitionIndexes property:

var myDatabase database

glue.NewS3Table(this, jsii.String("MyTable"), &S3TableProps{
	Database: myDatabase,
	Columns: []column{
		&column{
			Name: jsii.String("col1"),
			Type: glue.Schema_STRING(),
		},
	},
	PartitionKeys: []*column{
		&column{
			Name: jsii.String("year"),
			Type: glue.Schema_SMALL_INT(),
		},
		&column{
			Name: jsii.String("month"),
			Type: glue.Schema_SMALL_INT(),
		},
	},
	PartitionIndexes: []partitionIndex{
		&partitionIndex{
			IndexName: jsii.String("my-index"),
			 // optional
			KeyNames: []*string{
				jsii.String("year"),
			},
		},
	},
	 // supply up to 3 indexes
	DataFormat: glue.DataFormat_JSON(),
})

Alternatively, you can call the addPartitionIndex() function on a table:

var myTable table

myTable.AddPartitionIndex(&PartitionIndex{
	IndexName: jsii.String("my-index"),
	KeyNames: []*string{
		jsii.String("year"),
	},
})

Partition Filtering

If you have a table with a large number of partitions that grows over time, consider using AWS Glue partition indexing and filtering.

var myDatabase database

glue.NewS3Table(this, jsii.String("MyTable"), &S3TableProps{
	Database: myDatabase,
	Columns: []column{
		&column{
			Name: jsii.String("col1"),
			Type: glue.Schema_STRING(),
		},
	},
	PartitionKeys: []*column{
		&column{
			Name: jsii.String("year"),
			Type: glue.Schema_SMALL_INT(),
		},
		&column{
			Name: jsii.String("month"),
			Type: glue.Schema_SMALL_INT(),
		},
	},
	DataFormat: glue.DataFormat_JSON(),
	EnablePartitionFiltering: jsii.Boolean(true),
})

Glue Connections

Glue connections allow external data connections to third party databases and data warehouses. However, these connections can also be assigned to Glue Tables, allowing you to query external data sources using the Glue Data Catalog.

Whereas S3Table will point to (and if needed, create) a bucket to store the tables' data, ExternalTable will point to an existing table in a data source. For example, to create a table in Glue that points to a table in Redshift:

var myConnection connection
var myDatabase database

glue.NewExternalTable(this, jsii.String("MyTable"), &ExternalTableProps{
	Connection: myConnection,
	ExternalDataLocation: jsii.String("default_db_public_example"),
	 // A table in Redshift
	// ...
	Database: myDatabase,
	Columns: []column{
		&column{
			Name: jsii.String("col1"),
			Type: glue.Schema_STRING(),
		},
	},
	DataFormat: glue.DataFormat_JSON(),
})

Encryption

You can enable encryption on a Table's data:

• S3Managed - (default) Server side encryption (SSE-S3) with an Amazon S3-managed key.

var myDatabase database

glue.NewS3Table(this, jsii.String("MyTable"), &S3TableProps{
	Encryption: glue.TableEncryption_S3_MANAGED,
	// ...
	Database: myDatabase,
	Columns: []column{
		&column{
			Name: jsii.String("col1"),
			Type: glue.Schema_STRING(),
		},
	},
	DataFormat: glue.DataFormat_JSON(),
})

• Kms - Server-side encryption (SSE-KMS) with an AWS KMS Key managed by the account owner.

var myDatabase database

// KMS key is created automatically
// KMS key is created automatically
glue.NewS3Table(this, jsii.String("MyTable"), &S3TableProps{
	Encryption: glue.TableEncryption_KMS,
	// ...
	Database: myDatabase,
	Columns: []column{
		&column{
			Name: jsii.String("col1"),
			Type: glue.Schema_STRING(),
		},
	},
	DataFormat: glue.DataFormat_JSON(),
})

// with an explicit KMS key
// with an explicit KMS key
glue.NewS3Table(this, jsii.String("MyTable"), &S3TableProps{
	Encryption: glue.TableEncryption_KMS,
	EncryptionKey: kms.NewKey(this, jsii.String("MyKey")),
	// ...
	Database: myDatabase,
	Columns: []*column{
		&column{
			Name: jsii.String("col1"),
			Type: glue.Schema_STRING(),
		},
	},
	DataFormat: glue.DataFormat_JSON(),
})

• KmsManaged - Server-side encryption (SSE-KMS), like Kms, except with an AWS KMS Key managed by the AWS Key Management Service.

var myDatabase database

glue.NewS3Table(this, jsii.String("MyTable"), &S3TableProps{
	Encryption: glue.TableEncryption_KMS_MANAGED,
	// ...
	Database: myDatabase,
	Columns: []column{
		&column{
			Name: jsii.String("col1"),
			Type: glue.Schema_STRING(),
		},
	},
	DataFormat: glue.DataFormat_JSON(),
})

• ClientSideKms - Client-side encryption (CSE-KMS) with an AWS KMS Key managed by the account owner.

var myDatabase database

// KMS key is created automatically
// KMS key is created automatically
glue.NewS3Table(this, jsii.String("MyTable"), &S3TableProps{
	Encryption: glue.TableEncryption_CLIENT_SIDE_KMS,
	// ...
	Database: myDatabase,
	Columns: []column{
		&column{
			Name: jsii.String("col1"),
			Type: glue.Schema_STRING(),
		},
	},
	DataFormat: glue.DataFormat_JSON(),
})

// with an explicit KMS key
// with an explicit KMS key
glue.NewS3Table(this, jsii.String("MyTable"), &S3TableProps{
	Encryption: glue.TableEncryption_CLIENT_SIDE_KMS,
	EncryptionKey: kms.NewKey(this, jsii.String("MyKey")),
	// ...
	Database: myDatabase,
	Columns: []*column{
		&column{
			Name: jsii.String("col1"),
			Type: glue.Schema_STRING(),
		},
	},
	DataFormat: glue.DataFormat_JSON(),
})

Note: you cannot provide a Bucket when creating the S3Table if you wish to use server-side encryption (KMS, KMS_MANAGED or S3_MANAGED).

Types

A table's schema is a collection of columns, each of which have a name and a type. Types are recursive structures, consisting of primitive and complex types:

var myDatabase database

glue.NewS3Table(this, jsii.String("MyTable"), &S3TableProps{
	Columns: []column{
		&column{
			Name: jsii.String("primitive_column"),
			Type: glue.Schema_STRING(),
		},
		&column{
			Name: jsii.String("array_column"),
			Type: glue.Schema_Array(glue.Schema_INTEGER()),
			Comment: jsii.String("array<integer>"),
		},
		&column{
			Name: jsii.String("map_column"),
			Type: glue.Schema_Map(glue.Schema_STRING(), glue.Schema_TIMESTAMP()),
			Comment: jsii.String("map<string,string>"),
		},
		&column{
			Name: jsii.String("struct_column"),
			Type: glue.Schema_Struct([]*column{
				&column{
					Name: jsii.String("nested_column"),
					Type: glue.Schema_DATE(),
					Comment: jsii.String("nested comment"),
				},
			}),
			Comment: jsii.String("struct<nested_column:date COMMENT 'nested comment'>"),
		},
	},
	// ...
	Database: myDatabase,
	DataFormat: glue.DataFormat_JSON(),
})

Primitives

Numeric

Date and time

String

Miscellaneous

Complex

Data Quality Ruleset

A DataQualityRuleset specifies a data quality ruleset with DQDL rules applied to a specified AWS Glue table. For example, to create a data quality ruleset for a given table:

glue.NewDataQualityRuleset(this, jsii.String("MyDataQualityRuleset"), &DataQualityRulesetProps{
	ClientToken: jsii.String("client_token"),
	Description: jsii.String("description"),
	RulesetName: jsii.String("ruleset_name"),
	RulesetDqdl: jsii.String("ruleset_dqdl"),
	Tags: map[string]*string{
		"key1": jsii.String("value1"),
		"key2": jsii.String("value2"),
	},
	TargetTable: glue.NewDataQualityTargetTable(jsii.String("database_name"), jsii.String("table_name")),
})

For more information, see AWS Glue Data Quality.