ArgoCD bootstrap cluster December 16, 2019
valeblog
0 Comment

I wish it could be completely automated… But for now I’ve just automated as much as possible (and convenient). The ingredients:

  • Helm
  • Sealed Secrets
  • Argo CD and Argo Rollouts
  • traefik
  • Prometheus and Grafana

I have a repository for the purposes of playing around with Kubernetes tooling like this – and hopefully turn it into an actual application eventually. I have big plans and lots of stuff I want to try out, but time is limited. All the code examples in this post use the namespaces and naming choices in the repository. The folder structure (relevant to this bit) is like…

system
 ├┬ apps
 │└─ (bootstrapped Argo CD app manifests)
 ├┬ argo
 │└─ the local "umbrella chart" for Argo CD and Argo Rollouts
 ├┬ bootstrap
 │└─ boilerplate project and application manifests
 └┬ manifests
  └─ manifests I didn't bother turning into a Helm chart
     referenced by the raw-manifests.yaml application

Helm

helm init

Depending on the system configuration, it may be necessary to give the Tiller ServiceAccount the necessary ClusterRoleBinding. Since I’m not running a bank’s infrastructure on this cluster, I just give the kube-system:default account cluster-admin privileges.

ArgoCD bootstrap

Since ArgoCD can’t install itself if it’s not on the cluster already, it has to be installed first.

helm install --namespace argo --name argo-cd .

I have a local “umbrella chart” that covers ArgoCD so that it can later manage itself once already on the cluster. I’m still not 100% sure if this is a good idea considering that it can inadvertently lead to an infinite loop of resource (app) updates.

I’ve actually murdered a bunch of clusters accidentally like this… One misstep (miscommit) and it DOSes the control plane to hell and back with a flood of updates that never end (cf infinite loop). Things would probably be safer with the Argo Application manifests in a separate repository.

Sealed Secrets bootstrap

Another thing that’s easier to push on the cluster manually in the first steps (and never touch later) is sealed-secrets.

helm install --namespace kube-system --name sealed-secrets stable/sealed-secrets

Once the installation is done, be sure to re-generate all secrets that the cluster (system apps) will need. Commit and push them before actually initializing the “app of apps” or things will go red.

kubeseal \
 --format yaml \
 --controller-namespace=kube-system \
 --controller-name=sealed-secrets \
 < secret.raw.yaml > secret.yaml

While this definitely isn’t as automatic as it could be, as often as I set up a new cluster it’s easier to do manually than to figure out how to automate it. Also you’d probably want a separate API key to use for a new cluster anyway.

Launch ArgoCD to orbit

Login

The ArgoCD Helm chart’s after-install message describes the process well. Since traefik isn’t live at this point, I need to port-forward the Argo server pod’s 443 and connect to that.

kubectl port-forward svc/argo-cd-argocd-server -n argo 8080:443
kubectl get pods -n argo -l app.kubernetes.io/name=argocd-server -o name | cut -d'/' -f 2
argocd login localhost:8080

Then use the name of the server pod (the default password) to login. And be sure to change the password with argocd account update-password before moving on.

Init project and bootstrap “app of apps”

  1. argocd proj create -f system.yaml
  2. argocd proj create -f monitoring.yaml
  3. argocd app create -f bootstrap.yaml

I prefer not to use the default project for system boilerplate like this, so I create projects for general system stuff (like traefik) and monitoring (Grafana and Prometheus). Be sure to create any namespaces necessary too (in my case kubectl create namespace monitoring).

Then comes the actual dive with the bootstrap app. This “bootstrap” app will be the only one not controlled by the git repo. So if I wanted to pin it to a certain commit for whatever reason, I’d need to upsert the app from the YAML again manually.

The only remaining task is that once traefik goes up and the cloud provider (in my case Digital Ocean) brings up the external load balancer, make sure to edit all the A(AAA) records for the domains to point to the new IP.

As I configure traefik to use Digital Ocean’s API and automatically acquire TLS certificates for all my stuff using Let’s Encrypt, by the time I open the browser, usually the certificates are already there and waiting. I’m using the DNS challenge, so it doesn’t even need said domains to be actually pointed at the load balancer while it’s in progress.

Since ArgoCD is itself included in the bootstrap, the server pod might (probably will) get recreated, meaning the port-forward started earlier will break. Just reconnect – the password won’t be reset.

Fine-tuning

After this point pretty much the only thing left is to log in to Grafana and set things up. The Helm install outputs the instructions how to get the initial password, but of course that won’t show if it’s Argo applying the chart.

kubectl get secret --namespace monitoring grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

I usually create myself a new account and delete the old “admin”. Then add the Prometheus service as data-source and set up the Traefik and Node Exporter dashboards from available templates.

With my setup the Prometheus server is at prometheus-server.monitoring.svc.cluster.local. Node Exporter templates I’ve tried and liked: 11074, 1860 and 405. Traefik template: 4475.

Hit the road, Jack

What’s left? Actually deploying something into the cluster. I’ve already tested Argo Rollouts and confirmed that it works perfectly for my demands. What I still have to figure out is how/where to keep the Rollout (Deployment) files, and how to automate releases on merges to the “master” branch.

Rather than a technical difficulty, this is mostly up to the team working on the project (in my case, me, myself and I). It shouldn’t be hard to set up any of the major CI providers to “roll” with Rollouts (or Konfigure – as I said, things are yet to be decided).

There’s also the issue of the developer environment. While Argo Rollouts is great for managing (and pulling off) releases, I’d prefer much faster feedback while developing. I’ve earlier tried out Garden and left with a positive impression. Since then I was approached by the CEO of Tilt to try that out as well, so might give that a shot (though I’m not fond of its style of configuration).