Growing Pains

Deployment

Before deployment can start, you need to federate the resource types you want to deploy. These can be pods, services, namespaces, and so on. Again, the kubefedctl tool is used for this step. For example,

kubecfedctl enable pods

lets you distribute resources of the Pod type. If a Kubernetes namespace exists that you want to federate as a whole, you can do so with the command:

# kubefedctl federate namespace demo --contents --enable-type

The --enable-type option ensures that the missing types are also federated immediately. On top of this, you can create a FederatedNamespace type resource that lets you control at namespace level the clusters on which K8s creates resources. This even works at the single deployment level, but it does cause more administrative overhead. However, if you rely on cloud providers where privacy concerns exist, this method gives you clear control over where containers are running.

Setting up a namespace the right way for use in a federation also means having a YAML file like the one in Listing 2. The fedtestns namespace has to exist before the create step. The placement parameter lets you manage the clusters to which K8s rolls out the resources. To distribute the pods on both sides of the cluster, you need to feed them in differently.

---
apiVersion: types.kubefed.io/v1beta1
kind: FederatedNamespace
metadata:
  name: fedns
  namespace: fedtestns
spec:
  placement:
    clusters:
    - name: earth
    - name: vulcan

A simple web server is used as a test case. Normal unfederated deployment results in the pods running in the local cluster only. The command for federating the namespace along with its contents distributes all the existing resources but does not ensure that newly created resources are also evenly distributed in the same way. You now have the option of creating a deployment directly as a FederatedDeployment instead. The YAML file for this is shown in Listing 3. It looks very similar to the one for a simple deployment, but the placement parameter ensures that the specified clusters also inherit the deployment.

apiVersion: types.kubefed.io/v1beta1
kind: FederatedDeployment
metadata:
  name: fedhttp
  namespace: fedtestns
spec:
  template:
    metadata:
      name: http
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: httpbin
          version: v1
      template:
        metadata:
          labels:
            app: httpbin
            version: v1
        spec:
          containers:
          - image: docker.io/kennethreitz/httpbin
            imagePullPolicy: IfNotPresent
            name: httpbin
            ports:
            - containerPort: 80
  placement:
    clusters:
    - name: earth
    - name: vulcan

After a short wait, you will now have a pod and a deployment with the web service on both clusters. The pods on both clusters can also be accessed on port 80. In the case of AWS, however, you also need to enable this port in the correct security group of the AWS configuration.

For the pods in the clusters to work together, it is crucial to set up routing between the two clusters so that they can reach each other. You also need to configure firewall rules and security groups appropriately. It is advisable here to rely on an automation tool such as Ansible if the cluster at the cloud provider's end will be on-demand only.

Like the deployment, you also need to roll out the service as a FederatedService, if you use it. Listing 4 shows the service for deployment from Listing 3, with service name resolution running locally; that is, fed-service resolves to the cluster IP in the local cluster. This feature is something to keep in mind when designing the service.

apiVersion: types.kubefed.io/v1beta1
kind: FederatedService
metadata:
  name: fed-service
  namespace: fedtestns
spec:
  template:
    spec:
      selector:
        app: httpbin
      type: NodePort
      ports:
        - name: http
          port: 80
  placement:
    clusters:
    - name: earth
    - name: vulcan

Components such as persistent volumes are also created locally on the clusters and must be populated there. Where pods access remote resources, you need to make sure they are accessible on both clusters. It is important to consider both IP accessibility and name resolution.

If you remove the federated resources, you remove the resources at the same time. To remove only one cluster from the deployment, remove the cluster under placement. You can remove the second cluster from the configuration with the command:

# kubefedctl unjoin vulcan --host-cluster-context earth --v=2

In the test, however, I did have to manually clean up the resources rolled out in the second cluster retroactively. You can save yourself this step by completely deleting the cluster from the commercial public cloud provider.

Conclusions

Kubernetes Cluster Federation makes it easy to link up one or multiple clusters as resource extensions. This solution doesn't mean you don't have to pay attention to what will be running where, and it is important to set up the extension such that the service's consumers can use it. In the case of a special promo in a web store, for example, the customers' requests need to reach the containers in the new cluster in the same way they did at the previously existing clusters; otherwise, federation will not offer you any performance benefits. If you expect such situations to occur frequently, it is advisable to run your own cluster as a federation with a single member. Then, you only need to complete the steps required for the extension.