Kubernetes for Celery Developers

Kubernetes is an open-source container-orchestration system for automating deployment, scaling and management of containerised apps. And if you want to sound really 🆒, you should refer to it as k8s. Just in case you want to know why it's called k8s, check out this brief explanation.

k8s' job is to run, track and monitor containers at scale. It has become the de facto tool for container management. Kubernetes is the largest and fastest growing open-source container orchestration software. This blog post is the first part of a series: Kubernetes for Python developers.

Our goal is to migrate a Celery app app we developed in a previous blog post from Docker Compose to Kubernetes. You do not need any Kubernetes knowlegde to follow this blog post. You should have some experience with Docker. In this first part of the series, you will learn how to set up RabbitMQ as your Celery message broker on Kubernetes. You will learn about kubectl, the Kubernetes command line interface. And by the end of this article you will know how to deploy a self-healing RabbitMQ application with a stable IP address and DNS name into the cluster.

To run Kubernetes on your machine, make sure to enable it. You can find instructions here.

kubectl

First thing you need to know is kubectl. kubectl is the kubernetes command line tool. It is the docker-compose equivalent and lets you interact with your kubernetes cluster. For example, run kubectl cluster-info to get basic information about your kubernetes cluster. Or kubectl logs worker to get stdout/stderr logs. Very similar to docker-compose logs worker.

Pods

You cannot run a container directly on Kubernetes. A container must always run inside a Pod. A Pod is the smallest and most basic building block in the Kubernetes world. A Pod is an environment for a single container. Or a small number of tightly coupled containers (think log forwarding container). A Pod shares some of the properties of a Docker Compose service. A Pod specifies the docker image and command to run. It allows you to define environment variables, memory and CPU resources.

Unlike a Docker Compose service, a Pod does not provide self-healing functionality. It is ephemeral. When a Pod dies, it's gone. Nor does a Pod come with DNS capabilities. Survival, restart, network accessbility, port mapping and all that is handled by a so-called Service object. We will cover services further down. Pods are much lower level compared to Docker Compose services. Let's create a RabbitMQ Pod. We use the RabbitMQ image from Docker Hub, tag 3.7.8.

# rabbitmq-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: rabbitmq-pod
spec:
  containers:
    - name: rabbitmq-container
      image: rabbitmq:3.7.8

Create the Pod with kubectl and confirm it is up and running.

# apply rabbitmq-pod.yaml
~$ kubectl apply -f rabbitmq-pod.yaml
pod/rabbitmq-pod created

# list pods
~$ kubectl get pods
NAME                            READY   STATUS    RESTARTS   AGE
rabbitmq-pod                    1/1     Running   0          10s

Delete the Pod and confirm it's gone.

# delete pod from rabbitmq-pod.yaml
~$ kubectl delete -f rabbitmq-pod.yaml
pod "rabbitmq-pod" deleted

# list pods
~$ kubectl get pods
No resources found.

ReplicaSets

When the container running inside the Pod dies, the Pod is gone. Pods do not self-heal. Nor do they scale. The lack of self-healing capabilities means that it is not a good idea to create a Pod directly. This is where ReplicaSets come in. A ReplicaSet ensures that a specified number of Pod replicas are running at any given time. A ReplicaSet is a management wrapper around a Pod. If a Pod, that is managed by a ReplicaSet, dies, the ReplicaSet brings up a new Pod instance.

# rabbitmq-rs.yaml
apiVersion: apps/v1
kind: ReplicaSet
metadata:
  name: rabbitmq-rs
  labels:
    app: rabbitmq-rs
spec:
  replicas: 1
  selector:
    matchLabels:
      name: rabbitmq-pod
  template:
    metadata:
      labels:
        name: rabbitmq-pod
    spec:
      restartPolicy: Always
      containers:
        - name: rabbitmq-container
          image: rabbitmq:3.7.8

Instead of a Pod yaml, we now create a ReplicaSet yaml. We define the Pod inside the .spec.template property of the ReplicaSet yaml - which is the RabbitMQ Pod manifest from above. That is,.spec.template has exactly the same schema as the Pod manifest. Except that it is nested and does not have the apiVersion and kind properties. We also rearranged the Pod's metadata slightly. We now attach the label name: rabbitmq-pod to the RabbitMQ Pod. This matches the ReplicaSet's .spec.selector.matchLabels selector. This means the ReplicaSet can manage the RabbitMQ Pods as the selector matches. We set the number of RabbitMQ Pods we want to run concurrently in .spec.replicas to 1. Let's create the ReplicaSet with kubectl.

# apply rabbitmq-rs.yaml
~$ kubectl apply -f rabbitmq-rs.yaml
replicaset.apps/rabbitmq-rs created

# list replicatsets
~$ kubectl get rs
NAME          DESIRED   CURRENT   READY   AGE
rabbitmq-rs   1         1         1       5s

# list pods
~$ kubectl get pods
NAME                READY   STATUS    RESTARTS   AGE
rabbitmq-rs-fxdqp   1/1     Running   0          7s

Let's find out what happens when we delete the Pod rabbitmq-rs-fxdqp.

# delete pod
~$ kubectl delete pod rabbitmq-rs-fxdqp
pod "rabbitmq-rs-fxdqp" deleted

# list pods
~$ kubectl get pods
NAME                READY   STATUS    RESTARTS   AGE
rabbitmq-rs-5sldl   1/1     Running   0          24s

What happened here? We deleted the ephemeral Pod rabbitmq-rs-fxdqp. The ReplicaSet then noticed that the actual number of RabbitMQ Pods running was 0. And it created a new RabbitMQ Pod instance named rabbitmq-rs-5sldl. We have a self-healing RabbitMQ instance. Nice. Now, let's try and delete delete the ReplicaSet.

# delete replicatset from rabbitmq-rs.yaml
~$ kubectl delete -f rabbitmq-rs.yaml
replicaset.apps "rabbitmq-rs" deleted

# list replicasets
~$ kubectl get rs
No resources found.

# list pods
~$ kubectl get pods
No resources found.

Deployments

Deploying ReplicaSet updates directly is only possible in an imperative way. It is much easier to define the desired state. This is the use case for Deployments. A Deployment provides declarative updates for ReplicaSets and Pods. Create a Deployment to create a ReplicaSet which, in turn, brings up one RabbitMQ Pod. In other words: ReplicaSets manage Pods. Deployments manage ReplicaSets.

# rabbitmq-deploy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: rabbitmq-deploy
spec:
  replicas: 1
  selector:
    matchLabels:
      name: rabbitmq-pod
  template:
    metadata:
      labels:
        name: rabbitmq-pod
    spec:
      restartPolicy: Always
      containers:
        - name: rabbitmq-container
          image: rabbitmq:3.7.8

Now, let's say we need RabbitMQ with the management plugin. We need to replace rabbitmq:3.7.8 with rabbitmq:3.7.8-management. The new Deployment manifest defines the updated desired state for rabbitmq-deploy.

# rabbitmq-management-deploy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: rabbitmq-deploy
spec:
  replicas: 1
  selector:
    matchLabels:
      name: rabbitmq-pod
  template:
    metadata:
      labels:
        name: rabbitmq-pod
    spec:
      restartPolicy: Always
      containers:
        - name: rabbitmq-container
          image: rabbitmq:3.7.8-management

Deploy the new Deployment version and see how it updates the ReplicaSet and Pod.

# apply rabbitmq-management-deploy.yaml
~$ kubectl apply -f rabbitmq-management-deploy.yaml
deployment.apps/rabbitmq-deploy configured

# list pods
~$ kubectl get pods
NAME                               READY   STATUS              RESTARTS   AGE
rabbitmq-deploy-7f86fcd959-fgtxr   1/1     Running             0          8m
rabbitmq-deploy-f98989967-qmxzn    0/1     ContainerCreating   0          2s

# list pods
~$ kubectl get pods
NAME                               READY   STATUS        RESTARTS   AGE
rabbitmq-deploy-7f86fcd959-fgtxr   0/1     Terminating   0          8m
rabbitmq-deploy-f98989967-qmxzn    1/1     Running       0          19s

# list replicasets
~$ kubectl get rs
NAME                         DESIRED   CURRENT   READY   AGE
rabbitmq-deploy-7f86fcd959   0         0         0       13m
rabbitmq-deploy-f98989967    1         1         1       1m

# get details for rabbitmq-deploy-f98989967-qmxzn pod
~$ kubectl get pod rabbitmq-deploy-f98989967-qmxzn -o yaml

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: 2018-11-23T16:33:38Z
  generateName: rabbitmq-deploy-f98989967-
  labels:
    name: rabbitmq-pod
    pod-template-hash: "954545523"
  name: rabbitmq-deploy-f98989967-qmxzn
  namespace: default
  ownerReferences:
  - apiVersion: extensions/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicaSet
    name: rabbitmq-deploy-f98989967
    uid: 87be145f-ef3d-11e8-886a-025000000001
  resourceVersion: "594134"
  selfLink: /api/v1/namespaces/default/pods/rabbitmq-deploy-f98989967-qmxzn
  uid: 87c0e8ca-ef3d-11e8-886a-025000000001
spec:
  containers:
  - image: rabbitmq:3.7.8-management
    imagePullPolicy: IfNotPresent
    name: rabbitmq-container
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-r7js4
      readOnly: true

RabbitMQ 3.7.8-management is successfully deployed, replacing RabbitMQ 3.7.8 and giving you access to the RabbitMQ management plugin. You now know how to create and deploy a self-healing RabbitMQ Kubernetes instance!

Services

We still lack a stable Pod IP address or DNS name. Remember that Pods are not durable. When a Pod dies, the ReplicaSet creates a new Pod instance. The new Pod's IP address differs from the old Pod's IP address. In order to run a Celery worker Pod, we need a stable connection to the RabbitMQ Pod.

Enter Services. A Kubernetes Service is another Kubernetes object. A service gets its own stable IP address, a stable DNS name and a stable port.Services provide service discovery, load-balancing, and features to support zero-downtime deployments. Kubernetes provides two types of Services. A ClusterIP service gives you a service inside your cluster. Your apps inside your cluster can access that service via a stable IP address, DNS name and port. A ClusterIP service does not provide access from outside the cluster. A NodePort service provides access to a Pod from outside the cluster. And everything a ClusterIP service provides.

Make the RabbitMQ Pod available inside the cluster under the service name rabbitmq and expose 5672. Expose the RabbitMQ management UI externally on port 30672.

# rabbitmq-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: rabbitmq
spec:
  type: NodePort
  selector:
    name: rabbitmq-pod
  ports:
    - protocol: TCP
      port: 15672
      nodePort: 30672
      targetPort: 15672
      name: http
    - protocol: TCP
      port: 5672
      targetPort: 5672
      name: amqp

Deploy with kubectl and check the service's status:

# apply rabbitmq-service.yaml
~$ kubectl apply -f rabbitmq-service.yaml
service/rabbitmq created

# get details for rabbitmq service
~$ kubectl get service rabbitmq
NAME       TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)                          AGE
rabbitmq   NodePort   10.105.37.247   <none>        15672:30672/TCP,5672:32610/TCP   1m

The RabbitMQ management UI is now available on http://localhost:30672. From within the cluster, RabbitMQ is now accessible on amqp://guest:guest@rabbitmq/5672.

Next up

In this blog post, we built the foundations for migrating our Docker Compose Celery app to Kubernetes. We set up a self-healing RabbitMQ Deployment and a RabbitMQ service that gives us a stable URL. Now that we have a stable RabbitMQ URL, we can set up our Celery worker on Kubernetes. I will cover that in the next blog post.