Automating certificate ordering in Azure using Kubernetes and Keyvault Acmebot

As a part of our transition to a fully cloud based application platform for our public facing web applications, we were moving our apps from old, on-prem servers to our shiny new Azure environment.

Our new cloud setup consisted of multiple Kubernetes clusters running in azure, and a fully managed public facing load-balancer that accepted and distributed incoming traffic from our end users.

Both the Azure Application Gateway and the Kubernetes clusters use certificates stored in the Azure Keyvault.

In short:
Our end users hit the Azure Load Balancer where the intial TLS connection is terminated. A new, second TLS connection is established from the Azure Load Balancer and to the Kubernetes cluster.
In our setup both these connections use the same certificates and we are using azure key vault to store the certificates.

In order to facilitate the process of ordering and renewing certificates, we have set up Keyvault Acmebot as a tool to help us manage the process. Keyvault acmebot allows us to manually order certificates for a given domain name and will provision this certificate in our keyvault using the let’s encrypt apis.

The only thing we are missing using this setup is automation whenever a new application is deployed in the kubernetes clusters, we would like to automatically provision the cert so that this can be used in the kubernetes ingress to terminate the incoming tls traffic coming from the Azure Load Balancer to the Kubernetes cluster.

Full flow regarding certificates.

1. Azure Application Gateway fetches the certs needed from the azure keyvault.
2. Whenever a new endpoint is discovered in the kubernetes cluster, our open source kubernetes controller ingress2acmebotreflector orders checks if a corresponding tls cert for the domain exists in the keyvault.
   If it does not, it will call the REST-API of Keyvault Acmebot to order the new cert.
3. When the new cert has been provisioned it will be saved in the keyvault, and we are using akv2k8s to fetch the certificate from the azure key vault to the Kubernetes cluster.

Using this setup we have been able to fully automate the provisioning and management of tls certificates in our cloud setup.