tf-aws-module_collection-ecs_appmesh_ingress

Overview

This terraform module creates an ingress route into the ECS Cluster using App Mesh through an Application Load Balancer and a Virtual Gateway.

This module provisions an Application Load Balancer (ALB) with a HTTPs listener, listening on port 443. All incoming traffic to this App Mesh are routed through this load balancer. The ALB routes traffic to the Virtual Gateway through the envoy proxy running as an ECS Service. Once the traffic reaches the Virtual Gateway listener, there can be multiple gateway routes configured at the Virtual gateway to route the traffic to respective backends (ECS Service) through the configured Virtual Services that uses CloudMap service discovery to discover the ECS task instances. This module just configures a single gateway route /health for the ALB to perform health checks on. It is a single HTTP server configured to return 200 OK.

The following AWS resources are provisioned in this module

• Application Load Balancer
  • Listeners
  • Target Group
  • Security Group
• ACM Certificate for Load Balancer
• Public DNS record for Load Balancer. Optionally, creates a vanity URL (CName record)
• Virtual Gateway
  • ECS Service
  • Task Definition
  • Security Group
  • IAM roles for Task and Task Execution
• Health Check ECS application
  • ECS Service
  • Task Definition
  • Virtual Service
  • Virtual Node
  • Service Discovery Service
  • Private Certificate for Service Discovery Endpoint
• Virtual Gateway route for the above Health Check service

Below are some important considerations for the App Mesh Ingress configuration on ECS cluster

• All ingress traffic must pass through the Virtual Gateway
• In order for the ALB health checks to not fail for a deployed Virtual Gateway in ECS (without any actual apps deployed), it must have a backend configured for health-checks. In this module, we deploy a simple HTTP server for health checks of Virtual gateway.
• The backend services that do not need route from outside the Mesh may not have a route configured in Virtual Gateway.
  • These services can be accessed by other services inside the Mesh using the Service Discovery endpoints (configured using CloudMap)
  • The security group of these services must allow ingress traffic on the application port for these services
• All the backend services that need a route from outside the Mesh must have a gateway route configured in the Virtual gateway
  • Virtual gateway supports 2 kinds of routing
    • Path based routing
    • Matching Hostname header based routing
• ALB is configured with HTTPs listener. In order to provision certificates for TLS, a custom domain name must be configured for the ALB. TLS certificates cannot be provisioned for the default AWS URL.
  • We use an internal ALB in this project
  • An HTTP listener is also configured to redirect all traffic at port 80 to 443.
  • An A record is created in the public hosted zone in the AWS account
  • A certificate is provisioned for the A record as the first domain name in the ACM certificate
    • Additional SANs can be added for other CNAME records pointing to the ALB. This can be useful for hostname header based routing.
• ALB Target Group must be configured for HTTPS at port 443. The health check port should ideally be 443 but can be any other port
  • The health check port must match the application port of the envoy proxy container in the task definition for Virtual Gateway ECS Service. The Virtual Gateway listener port and the health check port should also be the same.
  • The health check path can be anything, defaulting to /. If anything else is selected like /health, the corresponding gateway route must be configured to have the same match_path_prefix
• ALB Security group should allow ingress traffic on port 443 and optionally 80 (will be auto redirected by HTTP listener)
• Virtual Gateway Security Group configured on the ECS Service allows ingress traffic only from the ALB Security Group
  • Optionally, it can allow ingress traffic on 9901 port to check stats on the App Mesh.

Usage

Several sample variable files tfvars are available in the root directory which can be used to test this module. User needs to follow the below steps to execute this module

1. Update the tfvars file to manually enter values for all fields marked within <> to make the variable file usable
2. Create a file provider.tf with the below contents

    provider "aws" {
      profile = "<profile_name>"
      region  = "<region_name>"
    }
   

   If using SSO, make sure you are logged in aws sso login --profile <profile_name>
3. Make sure terraform binary is installed on your local. Use command type terraform to find the installation location. If you are using asdf, you can run asfd install and it will install the correct terraform version for you. .tool-version contains all the dependencies.
4. Run the terraform to provision infrastructure on AWS

   # Initialize
   terraform init
   # Plan
   terraform plan -var-file example.tfvars
   # Apply (this is create the actual infrastructure)
   terraform apply -var-file example.tfvars -auto-approve
   

Known Issues and Guidelines

1. The ALB is currently set to only work with TLS listener in this module. Unable to make it work for both HTTP and TLS listener in if/else. The parent module is not supporting
2. If tls_enforce=true in this module, then ensure that any applications deployed to this cluster that uses this ingress must also have tls_enforce=true, else those modules will fail to talk to the ingress
3. Sometimes the terraform destroy is unable to destroy all resources and fails with error shown below. But, this is mitigated by running terraform destroy again.

│ Error: deleting App Mesh Virtual Router (a0602f30-46ea-49ce-b545-a9255087d0dd): operation error App Mesh: DeleteVirtualRouter, https response error StatusCode: 409, RequestID: 0f756a18-5adc-43a4-9622-7590885de530, ResourceInUseException: VirtualRouter with name xxxx cannot be deleted because it is a VirtualService provider

Pre-Commit hooks

.pre-commit-config.yaml file defines certain pre-commit hooks that are relevant to terraform, golang and common linting tasks. There are no custom hooks added.

commitlint hook enforces commit message in certain format. The commit contains the following structural elements, to communicate intent to the consumers of your commit messages:

• fix: a commit of the type fix patches a bug in your codebase (this correlates with PATCH in Semantic Versioning).
• feat: a commit of the type feat introduces a new feature to the codebase (this correlates with MINOR in Semantic Versioning).
• BREAKING CHANGE: a commit that has a footer BREAKING CHANGE:, or appends a ! after the type/scope, introduces a breaking API change (correlating with MAJOR in Semantic Versioning). A BREAKING CHANGE can be part of commits of any type. footers other than BREAKING CHANGE: may be provided and follow a convention similar to git trailer format.
• build: a commit of the type build adds changes that affect the build system or external dependencies (example scopes: gulp, broccoli, npm)
• chore: a commit of the type chore adds changes that don't modify src or test files
• ci: a commit of the type ci adds changes to our CI configuration files and scripts (example scopes: Travis, Circle, BrowserStack, SauceLabs)
• docs: a commit of the type docs adds documentation only changes
• perf: a commit of the type perf adds code change that improves performance
• refactor: a commit of the type refactor adds code change that neither fixes a bug nor adds a feature
• revert: a commit of the type revert reverts a previous commit
• style: a commit of the type style adds code changes that do not affect the meaning of the code (white-space, formatting, missing semi-colons, etc)
• test: a commit of the type test adds missing tests or correcting existing tests

Base configuration used for this project is commitlint-config-conventional (based on the Angular convention)

If you are a developer using vscode, this plugin may be helpful.

detect-secrets-hook prevents new secrets from being introduced into the baseline. TODO: INSERT DOC LINK ABOUT HOOKS

In order for pre-commit hooks to work properly

• You need to have the pre-commit package manager installed. Here are the installation instructions.
• pre-commit would install all the hooks when commit message is added by default except for commitlint hook. commitlint hook would need to be installed manually using the command below

pre-commit install --hook-type commit-msg

To test the resource group module locally

1. For development/enhancements to this module locally, you'll need to install all of its components. This is controlled by the configure target in the project's Makefile. Before you can run configure, familiarize yourself with the variables in the Makefile and ensure they're pointing to the right places.

make configure

This adds in several files and directories that are ignored by git. They expose many new Make targets.

2. The first target you care about is env. This is the common interface for setting up environment variables. The values of the environment variables will be used to authenticate with cloud provider from local development workstation.

make configure command will bring down aws_env.sh file on local workstation. Developer would need to modify this file, replace the environment variable values with relevant values.

These environment variables are used by terratest integration suit.

Then run this make target to set the environment variables on developer workstation.

make env

3. The first target you care about is check.

Pre-requisites Before running this target it is important to ensure that, developer has created files mentioned below on local workstation under root directory of git repository that contains code for primitives/segments. Note that these files are aws specific. If primitive/segment under development uses any other cloud provider than AWS, this section may not be relevant.

• A file named provider.tf with contents below

provider "aws" {
  profile = "<profile_name>"
  region  = "<region_name>"
}

• A file named terraform.tfvars which contains key value pair of variables used.

Note that since these files are added in gitignore they would not be checked in into primitive/segment's git repo.

After creating these files, for running tests associated with the primitive/segment, run

make check

If make check target is successful, developer is good to commit the code to primitive/segment's git repo.

make check target

• runs terraform commands to lint,validate and plan terraform code.
• runs conftests. conftests make sure policy checks are successful.
• runs terratest. This is integration test suit.
• runs opa tests

Know Issues

Currently, the encrypt at transit is not supported in terraform. There is an open issue for this logged with Hashicorp - https://github.com/hashicorp/terraform-provider-aws/pull/26987

Requirements

Providers

Modules

Resources

Inputs

map(object(
    {
      name       = string
      max_length = optional(number, 60)
    }
  ))

{
  "acm": {
    "max_length": 60,
    "name": "acm"
  },
  "alb": {
    "max_length": 32,
    "name": "alb"
  },
  "alb_sg": {
    "max_length": 60,
    "name": "albsg"
  },
  "alb_tg": {
    "max_length": 60,
    "name": "albtg"
  },
  "health_check_app_ecs_sg": {
    "max_length": 60,
    "name": "hcappsg"
  },
  "health_check_ecs_app": {
    "max_length": 60,
    "name": "hcsvc"
  },
  "health_check_ecs_td": {
    "max_length": 60,
    "name": "hctd"
  },
  "s3_logs": {
    "max_length": 60,
    "name": "alblogs"
  },
  "sds_vg": {
    "max_length": 60,
    "name": "sdsvg"
  },
  "task_exec_policy": {
    "max_length": 60,
    "name": "execplcy"
  },
  "task_policy": {
    "max_length": 60,
    "name": "taskplcy"
  },
  "vgw_ecs_app": {
    "max_length": 60,
    "name": "vgwsvc"
  },
  "vgw_ecs_sg": {
    "max_length": 60,
    "name": "vgwsg"
  },
  "vgw_ecs_td": {
    "max_length": 60,
    "name": "vgwtd"
  },
  "virtual_gateway": {
    "max_length": 60,
    "name": "vgw"
  }
}

object({
    ingress_rules            = optional(list(string))
    ingress_cidr_blocks      = optional(list(string))
    ingress_with_cidr_blocks = optional(list(map(string)))
    egress_rules             = optional(list(string))
    egress_cidr_blocks       = optional(list(string))
    egress_with_cidr_blocks  = optional(list(map(string)))
    ingress_with_sg          = optional(list(map(string)))
    egress_with_sg           = optional(list(map(string)))
  })

object({
    description              = optional(string)
    ingress_rules            = optional(list(string))
    ingress_cidr_blocks      = optional(list(string))
    egress_rules             = optional(list(string))
    egress_cidr_blocks       = optional(list(string))
    ingress_with_cidr_blocks = optional(list(map(string)))
    egress_with_cidr_blocks  = optional(list(map(string)))
  })

list(object({
    # Need to use name_prefix instead of name as the lifecycle property create_before_destroy is set
    name_prefix      = optional(string, "albtg-")
    backend_protocol = optional(string)
    backend_port     = optional(number)
    target_type      = optional(string)
    health_check     = optional(map(string), {})
  }))

object({
    ingress_rules            = optional(list(string))
    ingress_cidr_blocks      = optional(list(string))
    ingress_with_cidr_blocks = optional(list(map(string)))
    egress_rules             = optional(list(string))
    egress_cidr_blocks       = optional(list(string))
    egress_with_cidr_blocks  = optional(list(map(string)))
  })

Outputs