# README
Table of Contents
About The Project
- Backend Master Class [Golang + Postgres + Kubernetes + gRPC]
- Learn everything about backend web development: Golang, Postgres, Gin, gRPC, Docker, Kubernetes, AWS, GitHub Actions
- Original Repo - simple_bank
- techschool
SQL
- dbdiagram.io
- CRUD
- SQL
- Very fast & straightforward
- Manual mapping SQL fields to variables
- GORM
- CRUD functions already implemented
- Run slowly on high load
- SQLX
- Quite fast & easy to use
- Fields mapping via query text & struct tags
- SQLC
- Very fast & easy to use
- Automatic code generation
- Catch SQL query errors before generating codes
- Full support Postgres. MySQL is experimental
- SQL
# Login using root
psql simple_bank -U root
# Check connection info
\conninfo
# In db folder
sqlc generate
-- Set idle session limit as superuser
ALTER system SET idle_in_transaction_session_timeout='5min';
-- Disable idle session limit as superuser
ALTER system SET idle_in_transaction_session_timeout=0;
Deadlock
- store_test.go: Logs for deadlock bug
- Refer to commit
45de7cb5a930e3bcdddae513d968b4943327983e
=== RUN TestTransferTx
>> before: 807 9
tx 2 create transfer
tx 2 create entry 1
tx 2 create entry 2
tx 2 get account 1
tx 2 update account 1
tx 2 get account 2
tx 2 update account 2
tx 1 create transfer
>> tx: 797 19
tx 1 create entry 1
tx 1 create entry 2
tx 1 get account 1
tx 1 update account 1
tx 1 get account 2
tx 1 update account 2
>> tx: 787 29
>> after: 787 29
--- PASS: TestTransferTx (0.21s)
PASS
BEGIN;
INSERT INTO transfers (from_account_id, to_account_id, amount) VALUES (1, 2, 10) RETURNING *;
INSERT INTO entries (account_id, amount) VALUES (1, -10) RETURNING *;
INSERT INTO entries (account_id, amount) VALUES (2, 10) RETURNING *;
SELECT * FROM accounts WHERE id = 1 FOR UPDATE;
UPDATE accounts SET balance = 90 WHERE id = 1 RETURNING *;
SELECT * FROM accounts WHERE id = 2 FOR UPDATE;
UPDATE accounts SET balance = 110 WHERE id = 2 RETURNING *;
ROLLBACK
-- edited code snippets from link above
-- see ShareLock under mode
SELECT a.application_name,
l.relation::regclass,
l.transactionid,
l.mode,
l.locktype,
l.GRANTED,
a.usename,
a.query,
a.pid
FROM pg_stat_activity a
JOIN pg_locks l ON l.pid = a.pid
ORDER BY a.pid;
- Run the 4 queries in the order below to simulate deadlock
-- Tx1: transfer $10 from account 1 to account 2
BEGIN;
-- 1
UPDATE accounts SET balance = balance - 10 WHERE id = 1 RETURNING *;
-- 3
UPDATE accounts SET balance = balance - 10 WHERE id = 2 RETURNING *;
ROLLBACK;
-- Tx2: transfer $10 from account 2 to account 1
BEGIN;
-- 2
UPDATE accounts SET balance = balance - 10 WHERE id = 2 RETURNING *;
-- 4
UPDATE accounts SET balance = balance - 10 WHERE id = 1 RETURNING *;
ROLLBACK;
Transaction Isolation Level
- Read Phenomena
- Dirty Read: A transaction reads data written by other concurrent uncommitted transaction
- Non-Repeatable Read: A transaction reads the same row twice and sees different value because it has been modified by other committed transaction
- Phantom Read: A transaction re-executes a query to find rows that satisfy a condition and sees a different set of rows, due to changes by other committed transaction
- Serialization Anomaly: The result of a group of concurrent committed transactions is impossible to achieve if we try to run them sequentially in any order without overlapping
- 4 Standard Isolation Levels
- Read Uncommitted: Can see data written by uncommitted transaction
- Read Committed: Only see data written by committed transaction
- Repeatable Read: Same read query always returns same result
- Serializable: Can achieve same result if execute transactions serially in some order instead of concurrently
Read Uncommitted | Read Committed | Repeatable Read | Serializable | |
---|---|---|---|---|
Dirty Read | ✅ | ❌ | ❌ | ❌ |
Non-Repeatable Read | ✅ | ✅ | ❌ | ❌ |
Phantom Read | ✅ | ✅ | ❌ | ❌ |
Serialization Anomaly | ✅ | ✅ | ✅ | ❌ |
MySQL | Postgres |
---|---|
4 Isolation Levels | 3 Isolation Levels |
Locking Mechanism | Dependencies Detection |
Repeatable Read | Read Committed |
- High Level Isolation Methods
- Retry Mechanism: There might be errors, timeout or deadlock
- Read documentation: Each database engine might implement isolation level differently
GitHub Actions
- Actions -> Configure in Go -> Copy code to vscode
- GitHub - Creating PostgreSQL service containers
- GitHub - migrate CLI usage
- GitHub - migrate releases
mockgen
# Go 1.16+
go install github.com/golang/mock/[email protected]
mockgen -build_flags=--mod=mod -package mockdb -destination db/mock/store.go github.com/DarrelASandbox/go-simple-bank/db/sqlc Store
Migration
migrate create -ext sql -dir db/migration -seq add_users
PASETO
- Platform-Agnostic Security Tokens
- Only 2 most recent PASTO versions are accepted
- Non-trivial Forgery
- No more "alg" header or "none" algorithm
- Everything is authenticated
- Encrypted payload for local use <symmetric key>
Docker
- sh-compatible wait-for
- Download from release and place it in root folder (
wait-for.sh
)
- Download from release and place it in root folder (
docker build -t simple_bank:latest .
docker run --name simple_bank -p 4000:4000 -e GIN_MODE=release simple_bank:latest
docker compose up
# Make script executable
chmod +x start.sh
chmod +x wait-for.sh
AWS ECR with GitHub Actions
- AWS ECR -> Create repository for URI
- Instead of using the commands given by AWS (View push commands), we will use GitHub Actions
- GitHub Marketplace Actions
- Search for
ECR
- Refer to
deploy.yml
file
- Search for
- AWS IAM -> Add user
- User name: github-ci
- Access type: Programmatic access
- Next -> Create group
- Group name: deployment
- Filter policies: elastic container registry
- Check AmazonEC2ContainerRegistryFullAccess -> Create group -> Next -> Review
- GitHub simple_bank repo Settings -> Secrets -> New repository secret
- Refer to the code snippets below
- Name:
AWS_ACCESS_KEY_ID
- Value:
AWS Access Key ID
- Name:
AWS_SECRET_ACCESS_KEY
- Value:
AWS Secret access key
# From tutorial video
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v1
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{secrets.AWS_SECRET_ACCESS_KEY }}
# From deploy.yml
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v1
with:
role-to-assume: arn:aws:iam::123456789012:role/my-github-actions-role
AWS RDS
- AWS RDS -> Create database
- Choose a database creation method: Standard create
- Engine options: PostgreSQL
- DB instance identifier: simple_bank
- Master username: root
- Check Auto generate a password
- DB instance class: db.t2.micro
- Uncheck Storage autoscaling
- Use Default VPC for Connectivity
- Public access: Yes
- VPC security group:
- Create new
- New VPC security group name: access-postgres-anywhere
- Database authentication: Password authentication
- Initial database name: simple_bank
- Create database
- View credential details
- Register new Server/Connection for PostgreSQL
- Name: AWS Postgres
- Host: From AWS connection details endpoint (See point 19)
- Username: root
- Password: From AWS RDS credential details
- Database: simple_bank
- Check VPC Security Groups from AWS RDS
- Via the Security group ID -> Edit inbound rules
- Source: Anywhere (Unless ip is static)
- Save rules
- View connection details from AWS RDS -> Copy Endpoint
- Update
Makefile
postgres url
AWS Secrets Manager
# Generate a 32 characters string
openssl rand -hex 64 | head -c 32
- AWS Secrets Manager -> Store a new secret
- Other type of secrets
- DB_DRIVER: postgres
- DB_SOURCE: Use AWS credentials for the postgres connection
- SERVER_ADDRESS: 0.0.0.0:4000
- TOKEN_SYMMETRIC_KEY: Generate a 32 characters string
- ACCESS_TOKEN_DURATION: 15m
- Next -> Secret name: simple_bank -> Next -> Next -> Store
- Install AWS CLI
aws configure
(Refer to IAM profile)ls -l ~/.aws
cat ~/.aws/credentials
aws secretsmanager help
aws secretsmanager get-secret-value --secret-id simple_bank
Might want to try with arn as well- AWS IAM User groups -> Permissions -> Add permissions -> Attach Policies: SecretsManagerReadWrite
aws secretsmanager get-secret-value --secret-id simple_bank --query SecretString --output text
brew install jq
(To output into json format)aws secretsmanager get-secret-value --secret-id simple_bank --query SecretString --output text | jq -r 'to_entries|map("\(.key)=\(.value)")|.[]' > app.env
- After GitHub Actions are completed, check AWS ECR for the newly built image
- AWS CLI Command Reference: get-login-password
aws ecr get-login-password | docker login --username AWS --password-stdin <aws_account_id>.dkr.ecr.<region>.amazonaws.com
docker pull <AWS ECR Image URI>
AWS EKS
- AWS EKS -> clusters -> Create cluster
- Name: simple_bank
- Kubernetes version: 1.20
- Cluster Service Role: AWSEKSClusterRole
- AWS IAM -> Create role ->
EKS - Cluster
use case -> Next (AmazonEKSClusterPolicy
) -> Next - Role name: AWSEKSClusterRole
- AWS IAM -> Create role ->
- Next -> Use default VPC
- Cluster endpoint access: Public and private
- Amazon VPC CNI Version:
v1.8.0-eksbuild. 1
- Next -> Next -> Create -> Refresh later when cluster is created
- Add Node Group
- Name: simple_bank
- Node IAM Role: AWSEKSNodeRole
- AWS IAM -> Create role ->
EC2
use case -> Next - Pick
AmazonEKS_CNI_Policy
&AmazonEKSWorkerNodePolicy
&AmazonEC2ContainerRegistryReadOnly
-> Next - Role name: AWSEKSNodeRole
- Next
- AMI type: Amazon Linux 2 (AL2_x86_64)
- Capacity type: On-Demand
- Instance types: t3.small
- Disk size: 10 GiB
- Node Group scaling configuration
- Minimum size: 1 nodes
- Maximum size: 2 nodes
- Desired size: 1 nodes
- Next -> Disable Allow remote access to nodes -> Next -> Create
- AWS IAM -> Create role ->
- Refresh later when node is created
kubectl & k9s
- Command line tool (kubectl)
kubectl cluster-info
- AWS IAM -> Users -> Groups -> Pick
deployment
group that was created before - Permissions -> Add permissions -> Create inline policy
- Service: EKS
- Actions: All EKS actions
- Resources: All resources
- Review policy
- Name: EKSFullAccess
- Create policy
aws eks update-kubeconfig --name simple_bank --region us-east-1
ls -l ~/.kube
- Switch between clusters:
kubectl config use-context arn:aws:eks:ap-southeast-1:560476749134:cluster/dep-aws-eks
- How do I provide access to other IAM users and roles after cluster creation in Amazon EKS?
- AWS Profile -> My Security Credentials -> Access Keys (access key ID and secret access key) -> Create New Access Key
vi ~/.aws/credentials
to add the new credentialsexport AWS_PROFILE=github
orexport AWS_PROFILE=default
- Input User ARN from AWS IAM Users into
aws-auth.yaml
kubectl apply -f eks/aws-auth.yaml
- k9s
k8s with AWS EKS
- Kubernetes Documentation - Concepts - Workloads - Workload Resources - Deployments
kubectl apply -f eks/deployment.yaml
- AWS EKS -> Clusters -> Compute ->
simple_bank
Node Group -> Autoscaling group name -> Edit group details- Desired capacity: 1
- Update -> Activity -> Refresh
- The points below are for switching instance type:
- AWS EC2 - IP addresses per network interface per instance type: t3.micro
- Edit -> Edit Node Group: simple_bank page -> Cannot change eni
- So we must delete node group
- Add Node Group
- Name: simple_bank
- Role name: AWSEKSNodeRole
- Next
- AMI type: Amazon Linux 2 (AL2_x86_64)
- Capacity type: On-Demand
- Instance types: t3.small
- Disk size: 10 GiB
- Node Group scaling configuration
- Minimum size: 0 nodes
- Maximum size: 2 nodes
- Desired size: 1 nodes
- Next -> Next -> Create
- Kubernetes Documentation - Concepts - Services - Load Balancing, and Networking - Service
kubectl apply -f eks/service.yaml
- Linux nslookup command
AWS route 53
- Amazon Route 53 pricing
- AWS Route 53 -> Register domain (e.g. simple-bank.org) -> Check -> Continue
- Fill in Registrant Contact form -> Continue -> Email Verification
- Automatically Renew Domain -> Terms and Conditions -> Complete Order
- Enable Transfer lock (Confirm in Registered domains page)
- Hosted zones -> domain name
- Name Server (NS) record
- Start of Authority (SOA) record
- Create record:
- Record Name: api.domain-name.com
- Record Type: Address (A) Record
- Value: Check
Alias
- Alias to Network Load Balancer
- Region of Load Balancer: us-east-1
- URL of Load Balancer from API Service
nslookup api.simple-bank.org
k8s Ingress
- Kubernetes Documentation - Concepts - Services, Load Balancing, and Networking - Ingress #hostname-wildcards
- Kubernetes Documentation - Concepts - Services, Load Balancing, and Networking - Ingress Controllers
- Ingress NGINX Controller
- Kubernetes Documentation - Concepts - Services, Load Balancing, and Networking - Ingress #ingress-class
kubectl apply -f eks/service.yaml
(Change from LoadBalancer to ClusterIP because we do not want to expose it outside world)kubectl apply -f eks/ingress.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.5.1/deploy/static/provider/aws/deploy.yaml
- Copy
ingress-nginx
namespace's address from k9s -> AWS Route 53 Hosted zones page -> Checkapi.simple-bank.org
Record name -> Edit record- Replace the URL of Load Balancer with the copied address
- Save
nslookup api.simple-bank.org
kubectl apply -f eks/ingress.yaml
(Apply #ingress-class settings)
k8s cert-manager & Let's Encrypt
- Kubernetes Documentation - Concepts - Services, Load Balancing, and Networking - Ingress #tls
- cert-manager
- Let's Encrypt
- cert-manager - Automated Certificate Management Environment (ACME)
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.10.0/cert-manager.yaml
- Check
cert-manager
namespace kubectl apply -f eks/issuer.yaml
- Search for
clusterissuer
in k9s - Search for
secrets
in k9s - Search for
certificates
in k9s - Check
all+
namespace then search foringress
- Describe simple-bank-ingress
k8s with GitHub Actions
- GitHub Marketplace - Kubectl tool installer
- Find latest version using this URL:
storage.googleapis.com/kubernetes-release/release/stable.txt
- Find latest version using this URL:
dbdocs.io
- dbdocs.io
- Refer to
db.dbml
- Login ->
dbdocs build doc/db.dbml
- Refer to
gRPC
- Remote Procedure Call Framework
- The client can execute a remote procedure on the server
- The remote interaction code is handled by gRPC
- The API & data structure code is automatically generated
- Support multiple programming languages
- How it works?
- Define API & data structure
- The RPC and its request/ response structure are defined using protobuf
- Generate gRPC stubs
- Generate codes for the server and client in the language of your choice
- Implement the server
- Implement the RPC handler on the server side
- Use the client
- Use the generated client stubs to call the RPC on the serser
- Define API & data structure
- Why GRPC?
- High performance: HTTP/2: binary framing, multiplexing, header compression, bidirectional communication
- Strong API contract: Server & client share the same protobuf RPC definition with strongly typed data
- Automatic code generation: Codes that serialize/ deserialize data, or transfer data between client & server are automatically generated
- 4 Types of GRPC
- Unary gRPC
- Client streaming gRPC
- Server streaming gRPC
- Bidirectional streaming gRPC
- gRPC Gateway: Serve both gRPC and HTTP requests at once
- A plugin of protobuf compiler
- Generate proxy codes from protobuf
- In-process translation: only for unary
- Separate proxy server: both unary and streaming
- Write code once, serve both gRPC and HTTP requests
protoc-gen-grpc-gateway --help
- gRPC-Gateway - Using proto names in JSON
- gRPC - Introduction to gRPC
- gRPC - Docs - Languages - Go - Quick start
brew install protobuf
protoc --version
"protoc": {
"options": ["--proto_path=protos/v3"]
}
api
folder: REST api - HTTP requestapi_rpc
folder: gRPC api - grpc message- Why do we need to register reflection service on gRPC server
- List of TCP and UDP port numbers
evans --host localhost --port=5000 -r repl
show service
call CreateUser
exit
Swagger
# From source `~/Downloads/grpc-gateway/protoc-gen-openapiv2/options`
# Copy required files
cp *.proto ~/Projects/go-simple-bank/proto/protoc-gen-openapiv2/options/
# From `~/Downloads/swagger-ui`
ls -l dist
cp -r dist/* ~/Projects/go-simple-bank/doc/swagger/
# Change url in `swagger-initializer.js` file to `simple_bank.swagger.json`
- Go Standard Library - Embed instead of statik
- Embed a directory of static files into your Go binary to be later served from an http.FileSystem
authorization
- We can implement authorization logic using a gRPC interceptor but if we do that, it won't work for the HTTP gateway.
- And we will have to implement a separate HTTP middleware as well.
- Hence, we have implemented the logic inside the RPC handler method, it will work out of the box for both gRPC & HTTP gateway server.
pm.test('Status code is 200', function () {
pm.response.to.have.status(200);
});
const jsonData = JSON.parse(responseBody);
pm.collectionVariables.set('access_token', jsonData.access_token);
logger
- zerolog
- What logs should be?
- gRPC method
- Request duration
- Response status code
- JSON structured
- Easily parsed and indexed like log management tools:
- Logstash
- fluentd
- Grafana Loki
pb.RegisterSimpleBankHandlerServer
function performs an in-process translation between HTTP and gRPC.- which means it will call the handler function of the gRPC server directly, without going through any gRPC interceptor.
- So that's why the gRPC logger function doesn't get executed for HTTP requests
- Hence we will use a separate HTTP logger middleware function