mardi 6 octobre 2020

Running and inspecting sanity of deployments on Kubernetes

TL;DR How can it be periodically checked that deployments configuration loading and integration with external/other services is sane on production clusters?

For example, assume the following deployment was created on some minikube cluster:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: "example-deployment"
spec:
  selector:
    matchLabels:
      app: "example"
  replicas: 1
  template:
    metadata:
      labels:
        app: "example"
    spec:
      containers:
      - name: "example-container"
        image: alpine
        command:
          - "sleep"
          - "100000000"

Then, I wish to periodically check this deployment is still sane. For instance, I want to find as soon as possible if one of the following(or more) happened:

  1. Someone changed the command to SOME_NON_EXISTING_CMD.
  2. Image was replaced with SOME_NOT_REAL_IMAGE.

Now, production K8s cluster of course could contain much more complicated deployments and eventually pods could fail(or worse - some silent fails) for various reasons - just to name a few:

  1. Invalid mounts of secrets/config mounts
  2. Missing env vars
  3. Assigning non existent service account
  4. Network issues(within the cluster or with the "outer world")
  5. External dependency like DB are down or under heavy load. That is to say, run some code(which should be added) to check that configuration was loaded well(through mounts etc.) and infrastructure is reachable.

Searching for such, closest I got were cronjob, helm tests and init containers. However, all lake proper visibility or could provide testing before the applying the deployment, but nothing more.

Is there some simple tool(or any other way) to perform such tests? Such tool should provide the following apis:

  1. API to tell K8s to "Run periodically this deployment with some other command(otherwise same deployment spec)"
  2. API to inspect tests results - ui or even just kubectl get SOME_SIMPLE_TEST_RESULTS_CRD.

Other command is basically system init without actually running anything.

Additionally, looking for similar questions I have found this one to be the closest and yet it seems quiet different:
Running integration/e2e tests on top of a Kubernetes stack

Aucun commentaire:

Enregistrer un commentaire