Cluster discovery

Motivation
Introduction
Usage
- AWS
- Google cloud
- Consul

Motivation

Part of using a cloud infrastructure is need to discover/identify a collection of nodes, i.e. a cluster, that is used to implement certain functionality.

Introduction

Cluster discovery can be based on some attributes, e.g. EC2 tags in the AWS environment.
Additionaly, the discovered nodes represented by their IP addresses can be filtered by running health checks on the nodes. Currently, cluster discovery is implemented for Amazon, Google and Consul environments.

Usage

To discover a cluster, one has to pass the following parameters:

Cluster name (informational)
API ("AWS", "GCP", "CONSUL", "LOCAL"), a case insensitive string. "LOCAL" can be used for unit testing. It returns a single node cluster with the IP address of 127.0.0.1.
Criteria
Health checks

Health checks can be implemented as

HTTP check. Issues a GET request with predefined URL and checks for the HTTP status code.
Age check. Verifies whether the node is sufficiently "old".

AWS

package main

import (
	"github.com/viant/cloudless/compute/cluster"
	_ "github.com/viant/cloudless/compute/cluster/aws"
	"time"
)

func main() (*cluster.Cluster, error) {
	// Discovery object contains criteria and health checks definitions
	discovery := &cluster.Discovery{
		Api:     "AWS",
		Cluster: "Cluster1",
		Criteria: cluster.Criteria{
			Region: "us-west-2",
			Tags:   []string{"mytag"},
		},
		HealthChecks: []cluster.HealthCheck{
			{
				URL:            "http://{IP}:8080/x/y/z",
				TimeoutMs:      1000,
				ExpectedStatus: 200,
				MinAge:         time.Minute * 10,
			},
		},
	}

	s := cluster.New()

	// discover cluster
	cluster, err := s.Discover(discovery)

}

Google cloud

package compute

import (
	"github.com/viant/cloudless/compute/cluster"
	_ "github.com/viant/cloudless/compute/cluster/gcp"
	"time"
)

func main() {
	// Discovery object contains criteria and health checks definitions
	discovery := &cluster.Discovery{
		Api:     "GCP",
		Cluster: "Cluster2",
		Criteria: cluster.Criteria{
			Project: "ppp",
			Zone:    "us-east1-b",
			Tags:    []string{"aerospike"},
			Labels:  map[string]string,
		},
		HealthChecks: []cluster.HealthCheck{
			{
				MinAge: time.Duration(time.Minute * 10),
			},
		},
	}

	s := cluster.New()

	// discover cluster
	cluster, err := s.Discover(discovery)

}

Multiple "Labels" criteria imply AND logic whle "Tags" do OR logic.

Consul

package compute

import (
	"github.com/viant/cloudless/compute/cluster"
	_ "github.com/viant/cloudless/compute/cluster/consul"
	"time"
)

func main() {
	// Discovery object contains criteria and health checks definitions
	discovery := &cluster.Discovery{
		Api:     "CONSUL",
		Cluster: "Cluster3",
		Criteria: cluster.Criteria{
			URL:     "consul.vianttech.com:8500",
			Service: "consul",
		},
	}

	s := cluster.New()

	// discover cluster
	cluster, err := s.Discover(discovery)

}

Consul can run health checks inside its own infrastructure and return only healthy IP addresses. However, it is possible to specify additional checks using the same mechanism as AWS and GCP.

# README