Kubernetes GCP HTTP Load-Balancer Tips

We are in the midst of deploying a Kubernetes setup for one of our products on GCP using GKE and so far it’s been a great experience. I thought I would share a couple of tips around how the ingress resource works with the GCP HTTP(s) Load-balancer.

Google provides some great instructions on getting ingress up and running however we had a couple of things which I thought would be worth sharing to help others.

• Custom health check URL for GCP healtcheck resource
• Ingress with multiple clusters in a GCP project
• Firewall rules to your services
• Beware the Backend Services quota

Health check & cluster-uid

Fortunately the first two of these are features which can be enabled by manually editing the GLBC replication controller, you can the cluster-uid example here.

$ kubectl get rc --namespace=kube-system
NAME                             DESIRED   CURRENT   AGE
elasticsearch-logging-v1         2         2         26m
heapster-v1.0.0                  1         1         26m
kibana-logging-v1                1         1         26m
kube-dns-v11                     1         1         26m
kubernetes-dashboard-v1.0.0      1         1         26m
l7-lb-controller-v0.6.0          1         1         26m
monitoring-influxdb-grafana-v3   1         1         26m

$ kubectl edit rc l7-lb-controller-v0.6.0 --namespace=kube-system

You can then edit the args array in the yml to include the following:

- args:
  - --default-backend-service=kube-system/default-http-backend
  - --sync-period=300s
  - --cluster-uid=uid
  - --health-check-path=/healthcheck/

Once saved delete the old pod and let the RC recreate it

$ kubectl delete pod -l name=glbc --namespace=kube-system
pod "l7-lb-controller-v0.6.0-ud9ix" deleted

The uid can be anything you like in our case this references the environment. Once this is set all the various resources in your project will have --uid apended to them allowing meaning cluster wont clash.

One gotcha about changing the healtcheck here is that this will apply to all you ingress resources, so you’ll need to make sure all your services return a 200 on this path.

Firewall

The Google guide suggests configuring your firewall by extracting the NodePort assigned by Kubernetes and then creating firewall rules for this, but once you have more than a few services we have found it far easier to statically assign within a range and create a generic firewall rule which covers these and all future services.

For example if we assign our services to use 30000-31000 as their nodePort:

apiVersion: v1
kind: Service
metadata:
  name: app-svc
  labels:
    app: app
spec:
  type: NodePort
  ports:
  - port: 80
    nodePort: 30000
  selector:
    app: app

We can then create a single run like this:

$ gcloud compute firewall-rules create allow-http-lb \
  --source-ranges 130.211.0.0/22 \
  --target-tags <cluster-tag> \
  --allow tcp:30000-31000

Backend Service Quota

Finally not really a tip as much as a heads up. Google by default gives you 3 Backend Services, however this is really inadequate once you start creating multiple services in your ingress resources.

You can request more, however it appears there is a cap of 30 per project without special approval.

I hope some of the manual step and limitations above will soon be replaced with automated options as Ingress moves from beta to production ready in 1.3, but even now in beta we have been impressed with how well everything fit together and works.