Rails on Kubernetes - Part 2

This post is part of a series about setting up a Rails app with Kubernetes. Check out Part 1 - Rails and Docker Compose.

Code

You can skip straight to the completed deployment here Rails app on Github.

Rails Docker Image

If you followed my previous post you should have a Rails application working with Docker Compose.

In order to prep for the Kubernetes deploy we need to package the Rails Docker image and push to a Docker registry. I'll use a public Docker Hub repo for simplicity, but you should always store your images in a private registry if your source code lives in the image.

bin/rails g task docker push_image  

We create a rake task for this. It's very simple, we're grabbing the latest git revision hash and using that as an image tag. We're also attaching a latest. Note that it's recommended you use the actual commit hash in your Kube deploys to increase visibility into what's running on each pod - I'm just tagging latest for simplicity.

namespace :docker do  
  desc "Push docker images to DockerHub"
  task :image do
    TAG = `git rev-parse --short HEAD`.strip

    puts "Building Docker image"
    sh "docker build -t tzumby/rails-app:#{TAG} ."

    IMAGE_ID = `docker images | grep tzumby\/rails-app | head -n1 | awk '{print $3}'`.strip

    puts "Tagging latest image"
    sh "docker tag #{IMAGE_ID} tzumby/rails-app:latest"

    puts "Pushing Docker image"
    sh "docker push tzumby/rails-app:#{TAG}"
    sh "docker push tzumby/rails-app:latest"

    puts "Done"
  end

end  

Now we can run rake docker:push_image every time we want to push a new image version.

Kubernetes

We are ready to translate the docker-compose.yml config into Kubernetes resources. For our local development we accessed the app under the classic port 3000 but with Kubernetes we will setup an Ingress resource running Nginx and proxy requests to a Rails service.

Postgres

We'll start with our Postgres server. Here are the Kube resources we will use to create the DB:

  • Service
    • The service maps traffic from the Ingress to our pods. There are a several ways to do this via the type option: NodePort, ClusterIP, LoadBalancer or ExternalName. If we don't specify anything, Kube will create the service as ClusterIP - meaning it will only be accessible from within the cluster. This is what we want for Postgres as we will only be connecting to it from the Rails pods.
  • Secret
    • This is an object used for storing sensitive information such as passwords or TLS certificates. We will create one for our Postgres username and password.
  • Persistent Volume (PV).
    • Pod storage is ephemeral just like the container file system. Using Kube's API we can allocate space using a number of different file systems (local, EBS, CephFS, iSCSI etc.)
  • Persistent Volume Claim (PVC)
    • If the PV allocates the space, the PVC binds that resource to our Pods.
  • Replication Controller (RC)
    • The RC controls the life cycle of our Pods. We specify what image to pull, how to mount the persistent volumes, what commands to run and define ENV variables to be used by our app.

Let's create the secret to store our username and password first:

$ kubectl create secret generic db-user-pass --from-literal=password=mysecretpass
$ kubectl create secret generic db-user --from-literal=username=postgres

Here is the yaml file containing the Service, PV, PVC and RC objects:

apiVersion: v1  
kind: Service  
metadata:  
  name: postgres
  labels:
    app: rails-kube-app
spec:  
  ports:
    - port: 5432
  selector:
    app: rails-kube-app
    tier: postgres
---
kind: PersistentVolume  
apiVersion: v1  
metadata:  
  name: postgres-pv
  labels:
    type: local
spec:  
  capacity:
    storage: 4Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/tmp/data"
---
kind: PersistentVolumeClaim  
apiVersion: v1  
metadata:  
  name: postgres-pvc
spec:  
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 4Gi
---
apiVersion: v1  
kind: ReplicationController  
metadata:  
  name: postgres
  labels:
    app: rails-kube-app
spec:  
  replicas: 1
  selector:
    app: rails-kube-app
    tier: postgres
  template:
    metadata:
      name: postgres
      labels:
        app: rails-kube-app
        tier: postgres
    spec:
      volumes:
      - name: postgres-pv
        persistentVolumeClaim:
          claimName: postgres-pvc
      containers:
      - name: postgres
        image: postgres:latest
        env:
        - name: POSTGRES_USER
          valueFrom:
            secretKeyRef:
              name: db-user
              key: username
        - name: POSTGRES_PASSWORD
          valueFrom:
            secretKeyRef:
              name: db-user-pass
              key: password
        - name: POSTGRES_DB
          value: rails-kube-demo_development
        - name: PGDATA
          value: /var/lib/postgresql/data
        ports:
        - containerPort: 5432
        volumeMounts:
        - mountPath: "/var/lib/postgresql/data"
          name: postgres-pv

Names and labels

name - uniquely identifies the object we are creating. A Service or a Replaication Controller will use the value directly because Kube will create one one resource for each. The Pods created by the RC will append a random string to the name because the number of pods is dynamic and depends on the number of replicas we specify in a Replication Controller. If name the Postgres RC postgres this is what we'll also use in our database.yml for example - Kube's DNS will resolve that to the proper resource.

labels - are key-value pairs used to organize and group resources. For example: environment (dev, staging, production) or tier (frontend, backend, database, cache). Given those examples we could run queries against our system such as:

kubectl get pods -l environment=production,tier=frontend  

Back to our Postgres deploy, let's run this to create the Kube resources:

kubectl create -f postgres.yaml  

We can verify that the pod ran successfully:

kubectl get pods -w  
NAME             READY     STATUS    RESTARTS   AGE  
postgres-k3mqv   1/1       Running   0          1m  

Redis Deployment

Next is the Redis deployment. We're not using volumes here for simplicity's sake. This could be a problem in a production environment: we will loose the memory stored Redis data if we re-deploy. If there are unprocessed jobs store in there that could be a big problem. I would recommend looking a setting up a Redis cluster with Kube for a production environment.

apiVersion: v1  
kind: Service  
metadata:  
  name: redis
  labels:
    app: rails-kube-app
spec:  
  ports:
    - port: 6379
  selector:
    app: rails-kube-app
    tier: redis
---
apiVersion: v1  
kind: ReplicationController  
metadata:  
  name: redis
spec:  
  replicas: 1
  selector:
    app: rails-kube-app
    tier: redis
  template:
    metadata:
      name: redis
      labels:
        app: rails-kube-app
        tier: redis
    spec:
      containers:
      - name: redis
        image: redis:latest
        ports:
        - containerPort: 6379

We'll run this yaml file as well:

kubectl create -f redis.yaml  

Now we should have both Postgres and Redis running:

NAME             READY     STATUS    RESTARTS   AGE  
postgres-k3mqv   1/1       Running   0          5m  
redis-dz20q      1/1       Running   0          1m  

Rails Migrations

In my previous post I ran the migrations using a separate docker-compose service and passing it a bash script that waited for the Postgres server to go up and then ran the migrations.

Kubernetes provides a special Job resource for this.

This is pretty neat, Kube will bring this Pod up, run the command and then shut it down. Note that we're running the db:create and db:migrate with a single Job. Normally you would create a separate job for the db creation and another one for ongoing jobs like running migrations.

We already have the kube secrets for the Postgres DB. Let's create another one for the secret key base that Rails will use in production as well:

$ kubectl create secret generic secret-key-base --from-literal=secret-key-base=50dae16d7d1403e175ceb2461605b527cf87a5b18479740508395cb3f1947b12b63bad049d7d1545af4dcafa17a329be4d29c18bd63b421515e37b43ea43df64

And this is our Kube Job:

apiVersion: batch/v1  
kind: Job  
metadata:  
  name: setup
spec:  
  template:
    metadata:
      name: setup
    spec:
      containers:
      - name: setup
        image: tzumby/rails-app:latest
        args: ["rake db:create && rake db:migrate"]
        env:
        - name: DATABASE_NAME
          value: rails-kube-demo_production
        - name: DATABASE_PORT
          value: 5432
        - name: DATABASE_USER
          valueFrom:
            secretKeyRef:
              name: db-user
              key: username
        - name: DATABASE_PASSWORD
          valueFrom:
            secretKeyRef:
              name: db-user-pass
              key: password         
        - name: RAILS_APP
          value: production
        - name: REDIS_URL
          value: redis
        - name: REDIS_PORT
          value: "6379"
        - name: SECRET_KEY_BASE
          valueFrom:
            secretKeyRef:
              name: secret_key_base
              key: secret_key_base          
      restartPolicy: Never

Running the job is predictable:

kubectl create -f setup.yaml  

Let's check if everything ran successfully:

kubectl get jobs  
NAME      DESIRED   SUCCESSFUL   AGE  
setup     1         1            1m  

Rails app

We are now ready to deploy the Rails app. It's a pretty standard config with a Service and a Replication Controller. We're using the handy RAILS_LOG_TO_STDOUT environment flag to trigger Rails logging to stdout. Since the Nginx server runs on a separate server we'll have to serve the static assets from Rails. There are a few ways to go around this if we wanted Nginx to serve the assets without hitting the Rails server:

  • We could run an Nginx instance on the Rails pods and configure it to serve the assets.
  • We could customize the Nginx Ingress Controller and copy the assets there on each deploy (this one doesn't seem feasable).

For now let's configure Rails to server its own assets via RAILS_SERVE_STATIC_FILES. I think the best compromise for a production setup would be to configure the Pagespeed module in the Nginx Intress Controller - but that's a topic for another post.

apiVersion: v1  
kind: Service  
metadata:  
  name: rails
  labels:
    app: rails-kube-app
spec:  
  ports:
    - port: 3000
  selector:
    app: rails-kube-app
    tier: rails
---
apiVersion: v1  
kind: ReplicationController  
metadata:  
  name: rails
spec:  
  replicas: 1
  selector:
    app: rails-kube-app
    tier: rails
  template:
    metadata:
      name: rails
      labels:
        app: rails-kube-app
        tier: rails
    spec:
      containers:
      - name: rails
        image: tzumby/rails-app:latest
        args: ["rails s -p 3000 -b 0.0.0.0"]
        env:
        - name: DATABASE_NAME
          value: rails-kube-demo_production
        - name: DATABASE_PORT
          value: 5432
        - name: DATABASE_USER
          valueFrom:
            secretKeyRef:
              name: db-user
              key: username
        - name: DATABASE_PASSWORD
          valueFrom:
            secretKeyRef:
              name: db-user-pass
              key: password         
        - name: RAILS_APP
          value: production
        - name: RAILS_LOG_TO_STDOUT
          value: true
        - name: RAILS_SERVE_STATIC_ASSETS
          value: true
        - name: REDIS_URL
          value: redis
        - name: REDIS_PORT
          value: "6379"
        - name: SECRET_KEY_BASE
          valueFrom:
            secretKeyRef:
              name: secret_key_base
              key: secret_key_base
        ports:
        - containerPort: 3000

The args parameter is equivalent to a Dockerfile's CMD and it will add that command as an argument to our ENTRYPOINT bash script we defined in our Dockerfile:

FROM ruby:2.3.1

RUN apt-get update -qq && apt-get install -y build-essential libpq-dev nodejs netcat  
RUN mkdir /myapp

WORKDIR /myapp

ADD Gemfile /myapp/Gemfile  
ADD Gemfile.lock /myapp/Gemfile.lock

RUN bundle install

ADD . /myapp

COPY docker-entrypoint.sh /usr/local/bin

ENTRYPOINT ["docker-entrypoint.sh"]  

Our docker-entrypoint.sh is very basic at this point:

#!/bin/sh

set -e

if [ -f tmp/pids/server.pid ]; then  
  rm tmp/pids/server.pid
fi

exec bundle exec "$@"  

Let's run this one as well:

kubectl create -f rails.yaml  

Sidekiq

The Sidekiq deploy is very similar to Rails. The only exception is the args we pass to the ENTRYPOINT:

apiVersion: v1  
kind: ReplicationController  
metadata:  
  name: sidekiq
spec:  
  replicas: 1
  selector:
    app: rails-kube-app
    tier: sidekiq
  template:
    metadata:
      name: sidekiq
      labels:
        app: rails-kube-app
        tier: sidekiq
    spec:
      containers:
      - name: sidekiq
        image: tzumby/rails-app:v2.3
        args: ["sidekiq -C config/sidekiq.yml"]
        env:
        - name: DATABASE_NAME
          value: rails-kube-demo_production
        - name: DATABASE_PORT
          value: 5432
        - name: DATABASE_USER
          valueFrom:
            secretKeyRef:
              name: db-user
              key: username
        - name: DATABASE_PASSWORD
          valueFrom:
            secretKeyRef:
              name: db-user-pass
              key: password         
        - name: RAILS_APP
          value: production
        - name: REDIS_URL
          value: redis
        - name: REDIS_PORT
          value: "6379"
        - name: SECRET_KEY_BASE
          valueFrom:
            secretKeyRef:
              name: secret_key_base
              key: secret_key_base

Note that I'm also not specifying a port number. We will not connect to this Pod directly: the sidekiq worker connects to the Redis service and listens for jobs.

Ingress Controller

We're almost ready to access our newly deployed Rails application. I created a simple Ingress resource that listens for rails.local HOST header and targets a service called rails on port 3000. This matches what I defined in my Rails kube deployment:

apiVersion: extensions/v1beta1  
kind: Ingress  
metadata:  
  name: rails-demo-ing
spec:  
  rules:
    - host: rails.local
      http:
        paths:
          - backend:
              serviceName: rails
              servicePort: 3000
            path: /

We use the handy create command to spin this one:

kubectl create -f ingress.yaml  

Now we can access our app at http://rails.local. I added an entry in my /etc/hosts file that points the minikube IP to the rails.local domain:

~ minikube ip
192.168.64.2  

And my hosts file:

192.168.64.2 rails.local  

If you deploy this on AWS or GCE you can have the option to spin a Load Balancer when you create an Ingress. You would then take the LB's Address and create the proper DNS records in your domain.

What's next

In the next post we will look at a few deployments and rolling updates scenarios. We'll also use apache bench to load test our setup and see how quickly we can respond to increase in load.