Skip to main content

Deploying Tigris on EKS

This section outlines the deployment of Tigris on an AWS managed Elastic Kubernetes Service (EKS).

The installation will use recommended settings for redundancy.

If you would rather watch a video, check out the deployment in action on YouTube:

Requirements

Installation Host

The following components will need to be installed on the machine you are performing the deployment steps on:

  • Helm
  • AWS Cli

Helm will be used to install the Tigris Stack Chart:

❯ helm version
version.BuildInfo{Version:"v3.10.1", GitCommit:"9f88ccb6aee40b9a0535fcc7efea6055e1ef72c9", GitTreeState:"clean", GoVersion:"go1.19.2"}

AWS Cli will be used to setup a wildcard certificate:

❯ aws --version
aws-cli/2.8.5 Python/3.10.8 Darwin/21.6.0 source/arm64 prompt/off

EKS

Outside of the above you will need an EKS cluster with access and sufficient resources available for deployment.

❯ kubectl get nodes -L beta.kubernetes.io/instance-type
NAME STATUS ROLES AGE VERSION INSTANCE-TYPE
ip-10-2-10-204.us-west-2.compute.internal Ready <none> 18h v1.21.14-eks-ba74326 r6i.2xlarge
ip-10-2-10-6.us-west-2.compute.internal Ready <none> 6d v1.21.14-eks-ba74326 r6i.2xlarge
ip-10-2-11-166.us-west-2.compute.internal Ready <none> 18h v1.21.14-eks-ba74326 r6i.2xlarge
ip-10-2-11-169.us-west-2.compute.internal Ready <none> 18h v1.21.14-eks-ba74326 r6i.2xlarge
ip-10-2-12-79.us-west-2.compute.internal Ready <none> 6d v1.21.14-eks-ba74326 r6i.2xlarge
ip-10-2-13-192.us-west-2.compute.internal Ready <none> 6d v1.21.14-eks-ba74326 r6i.2xlarge

Most of the resources will be consumed by FoundationDB and TypeSense, both of which are main building blocks to Tigris.

The EKS cluster must have the following components installed:

  • AWS Load Balancer Controller
  • Cert Manager
  • EBS CSI Controller
❯ helm list -A
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
aws-load-balancer-controller kube-system 1 2022-10-20 11:00:30.800715 -0400 EDT deployed aws-load-balancer-controller-1.4.4 v2.4.3
cert-manager kube-system 1 2022-10-20 11:00:27.772092 -0400 EDT deployed cert-manager-v1.6.1 v1.6.1
cert-manager-ca kube-system 1 2022-10-20 11:01:45.36529 -0400 EDT deployed cert-manager-ca-0.2.0 v0.1.0
external-dns kube-system 1 2022-10-20 11:00:25.898907 -0400 EDT deployed external-dns-6.7.1 0.12.1
metrics-server kube-system 1 2022-10-20 11:00:26.973139 -0400 EDT deployed metrics-server-3.8.1 0.6.1

Deployment

The installation deploys the following components:

  • Kubernetes Operator for FoundationDB
  • FoundationDB
  • Tigris Search (TypeSense)
  • Tigris Server

You can install the components individually or together, using the encompassing tigris-stack Helm Chart. Below I’m going to use this Chart to install Tigris.

Create Certificate for TLS

First, we need to generate a certificate for TLS. This certificate will be used on the load balancer to terminate TLS connections:

❯ aws acm request-certificate --domain-name='*.example.com'
{
"CertificateArn": "arn:aws:acm:us-west-2:878843336588:certificate/fe257207-b117-4db0-ad6b-eef8d66308cd"
}

Prepare For Deployment

Next, check out the deploy script repository:

git clone git@github.com:tigrisdata/tigris-deploy.git
Cloning into 'tigris-deploy'...
remote: Enumerating objects: 177, done.
remote: Counting objects: 100% (97/97), done.
remote: Compressing objects: 100% (60/60), done.
remote: Total 177 (delta 43), reused 68 (delta 34), pack-reused 80
Receiving objects: 100% (177/177), 87.68 KiB | 568.00 KiB/s, done.
Resolving deltas: 100% (63/63), done.

Navigate to the folder which contains the helm chart of tigris-stack:

cd tigris-deploy/helm/tigris-stack

Deploy Tigris Stack

As we are deploying to EKS, we are going to use an ALB as our load balancer. The host for this installation will be set to api.example.com.

⚠️ You will want to make sure that above hostname matches the domain of the wildcard certificate you created previously!

Finally, to ensure there is initial quorum for Tigris Search, we deploy it initially with a single replica:

❯ helm install tigris-stack . -f ./values.yaml --set tigris-server.ingress_aws.enabled=true --set tigris-server.tls_hostname="api.example.com" --set tigris-search.replicas=1

NAME: tigris-stack
LAST DEPLOYED: Tue Oct 25 18:58:53 2022
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None

That’s it, your Tigris deployment should be now on its way coming up!

Validate Deployment

Generally speaking, there are two high level checkboxes to check. First, we should ensure that all the resources were scheduled and are available and running. Second, we will want to make sure that Tigris API is accessible using the Ingress resource allocated. These steps are expanded upon below.

Resources Validation

Allow the resources such as FoundationDB a couple minutes to initialize. In a Production-ready installation FoundationDB would allocate significant resources. You will want to make sure that the FDB Pods were able to be scheduled.

Tigris will not enter the Running state until FoundationDB becomes fully functional. It might take a couple minutes for FoundationDB to become unavailable, even when all of its Pods appear to be in the Running state.

You can validate if the FDB key-value store is available using fdbcli:

tigris@tigris-server-58ccd7bb9f-czcbb:/server$ fdbcli -C /mnt/fdb-config-volume/cluster-file
Using cluster file `/mnt/fdb-config-volume/cluster-file'.

The database is available.

Welcome to the fdbcli. For help, type `help'.
fdb>

Look for the message “The database is available” in fdbcli’s output.

Final Overview

Unless there were additional customizations, your output should be similar to below:

❯ kubectl get all
NAME READY STATUS RESTARTS AGE
pod/fdb-cluster-log-1 2/2 Running 0 5m53s
pod/fdb-cluster-log-2 2/2 Running 0 5m53s
pod/fdb-cluster-log-3 2/2 Running 0 5m53s
pod/fdb-cluster-log-4 2/2 Running 0 5m53s
pod/fdb-cluster-log-5 2/2 Running 0 5m53s
pod/fdb-cluster-stateless-1 2/2 Running 0 5m54s
pod/fdb-cluster-stateless-10 2/2 Running 0 5m53s
pod/fdb-cluster-stateless-2 2/2 Running 0 5m54s
pod/fdb-cluster-stateless-3 2/2 Running 0 5m54s
pod/fdb-cluster-stateless-4 2/2 Running 0 5m54s
pod/fdb-cluster-stateless-5 2/2 Running 0 5m54s
pod/fdb-cluster-stateless-6 2/2 Running 0 5m54s
pod/fdb-cluster-stateless-7 2/2 Running 0 5m54s
pod/fdb-cluster-stateless-8 2/2 Running 0 5m54s
pod/fdb-cluster-stateless-9 2/2 Running 0 5m53s
pod/fdb-cluster-storage-1 2/2 Running 0 5m54s
pod/fdb-cluster-storage-2 2/2 Running 0 5m54s
pod/fdb-cluster-storage-3 2/2 Running 0 5m54s
pod/fdb-cluster-storage-4 2/2 Running 0 5m54s
pod/fdb-cluster-storage-5 2/2 Running 0 5m54s
pod/tigris-search-0 2/2 Running 1 6m49s
pod/tigris-server-58ccd7bb9f-czcbb 1/1 Running 4 6m49s
pod/tigris-server-58ccd7bb9f-ngjk5 1/1 Running 4 6m49s
pod/tigris-server-58ccd7bb9f-rnbxb 1/1 Running 4 6m49s
pod/tigris-stack-fdb-operator-5d9dbc4c9d-ptlng 1/1 Running 0 6m49s

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 172.20.0.1 <none> 443/TCP 5d8h
service/tigris-grpc NodePort 172.20.60.127 <none> 80:30440/TCP 6m50s
service/tigris-headless ClusterIP None <none> 8080/TCP 6m50s
service/tigris-http NodePort 172.20.82.191 <none> 80:30675/TCP 6m50s
service/tigris-search NodePort 172.20.130.194 <none> 80:31720/TCP 6m50s
service/ts ClusterIP None <none> 8108/TCP 6m50s

NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/tigris-server 3/3 3 3 6m50s
deployment.apps/tigris-stack-fdb-operator 1/1 1 1 6m50s

NAME DESIRED CURRENT READY AGE
replicaset.apps/tigris-server-58ccd7bb9f 3 3 3 6m50s
replicaset.apps/tigris-stack-fdb-operator-5d9dbc4c9d 1 1 1 6m50s

NAME READY AGE
statefulset.apps/tigris-search 1/1 6m50s

Ingress Validation

The most EKS installation specific piece to a Tigris installation is generally the load balancer and related resources.

The installation will create an annotated Ingress resource:

❯ kubectl get ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
tigris-server <none> * 80 3m25s
❯ kubectl get ingress tigris-server -o yaml | grep -A17 annotations:
annotations:
alb.ingress.kubernetes.io/backend-protocol: HTTP
alb.ingress.kubernetes.io/conditions.tigris-grpc: |
[{"field":"http-header","httpHeaderConfig":{"httpHeaderName": "Content-Type", "values":["application/grpc"]}}]
alb.ingress.kubernetes.io/group.name: tigris-server-lb
alb.ingress.kubernetes.io/healthcheck-path: /v1/health
alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
alb.ingress.kubernetes.io/load-balancer-attributes: routing.http.drop_invalid_header_fields.enabled=true
alb.ingress.kubernetes.io/load-balancer-name: tigris-server-lb
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/ssl-policy: ELBSecurityPolicy-TLS-1-2-2017-01
alb.ingress.kubernetes.io/tags: service=tigris-server
alb.ingress.kubernetes.io/target-group-attributes: stickiness.enabled=true,stickiness.app_cookie.duration_seconds=10,stickiness.type=app_cookie,stickiness.app_cookie.cookie_name=Tigris-Tx-Id
alb.ingress.kubernetes.io/target-type: ip
external-dns.alpha.kubernetes.io/ingress-hostname-source: annotation-only
kubernetes.io/ingress.class: alb
meta.helm.sh/release-name: tigris-stack
meta.helm.sh/release-namespace: default

AWS Load Balancer Controller will create an ALB based on above annotation:

❯ aws elbv2 describe-load-balancers --names tigris-server-lb | grep -i dnsname
"DNSName": "tigris-server-lb-<redacted>.us-west-2.elb.amazonaws.com",

Make sure that your load balancer is healthy and operational before you proceed!

Preparing For Production

There is one last step that is required for a proper, production-ready installation.

In order to ensure there is proper redundancy under Tigris Search, you will want to increase the number of replicas to an odd number of replicas post initial installation.

An odd number of replicas is required to ensure that quorum can be reached. An even number of replicas could end up being without a tie breaker during network partitioning, where both partitions may end up with the same number of replicas.

Below command will increase the number of Tigris Search replicas to 5:

❯ helm upgrade tigris-stack . -f ./values.yaml --set tigris-server.ingress_aws.enabled=true --set tigris-server.tls_hostname="api.example.com" --set tigris-search.replicas=5
Release "tigris-stack" has been upgraded. Happy Helming!
NAME: tigris-stack
LAST DEPLOYED: Tue Oct 25 19:13:33 2022
NAMESPACE: default
STATUS: deployed
REVISION: 2
TEST SUITE: None

We recommend using at least 3 replicas for Production installations with 5 being recommended for load handling purposes. This is the default setting in the tigris-stack Chart that is only reduced during install, to ensure quorum can be achieved during initialization.

You can verify that there are now 5 replicas running with a simple kubectl command:

❯ kubectl get pods | grep tigris
tigris-search-0 2/2 Running 1 17m
tigris-search-1 2/2 Running 0 2m39s
tigris-search-2 2/2 Running 0 2m39s
tigris-search-3 2/2 Running 0 2m39s
tigris-search-4 2/2 Running 0 2m39s
tigris-server-58ccd7bb9f-czcbb 1/1 Running 4 17m
tigris-server-58ccd7bb9f-ngjk5 1/1 Running 4 17m
tigris-server-58ccd7bb9f-rnbxb 1/1 Running 4 17m
tigris-stack-fdb-operator-5d9dbc4c9d-ptlng 1/1 Running 0 17m

Final Thoughts

In this section we have covered the deployment of Tigris to one of the most common Managed Kubernetes platforms available in the Cloud. As you can see from above the deployment process is fairly easy and straightforward on EKS.

Enjoy using Tigris as much as we enjoy building it!!!