Kubernetes is the single most poorly understood piece of software ever created. This should perhaps be surprising given how wildly successful it is, but the truth is even more surprising: Kubernetes is misunderstood by design. I can’t account for why this would be, but the fact of it is unquestionable. The entire body of documentation along with the video-based and interpersonally delivered tutoring provided by the experts is, as far as I can tell, metaphorical. And by metaphorical, I mean lies. The material and people are persistently lying about what Kubernetes is and how it works. I’m not attributing to malice. I think it’s just more of a vibe they’re trying to strike that goes something like: You don’t need to know how it works to use it, so here’s what you need to know. That’s inference. I can’t speak authoritatively to intent, but the end-result is objectively a user-space who is utterly baffled by Kubernetes and completely ignorant of how it works in real life.

You might think I’m being contentious; perhaps I am. Candidly I’m a little salty about it. My observation is that professional engineers in general make their way through the world by learning how things work. They don’t enjoy being told they don’t need to know, nor do they have much use for poorly formed metaphors about what something is kinda like. Suffice to say, I’m not a fan of the current approach taken by the documentation available – official and community. If you read through this article I believe you will not only learn something you didn’t know about Kubernetes, but also come to agree with me, or at least concede I have a point. If there’s a text that attempts to teach people about Kubernetes in its real life, non-metaphorical form, then I apologise, I’m yet to find it, and what exists in its stead strikes me as counterproductive, especially given how simple the truth of its architecture is to learn from first principles, and how much more difficult it is to disambiguate things once you’ve ingested the officially blessed metaphor.

Give me 30 minutes, and I’ll show you everything going on inside this “Kubernetes” thing.

The Yealots

Before we begin, I need to make a small digression into terminology. Specifically, I need a pejorative term for them. You know who I mean. The cloud native. The ones who, for whatever reason, don’t want us to know how Kubernetes actually works. I would call them natives but for the troubling colonizer overtones.

Let’s see. Dogmatic, religiously devoted to their one solution, and filled to the brim with theatrical metaphor that doesn’t really even orbit the truth. They remind me more of old-school communist gaslighter types. Posadists or Lysenkoists, shouting the party line at the top of their lungs and expecting reality to align to the ideology but not really caring whether it ever does. Yaml Zealots. Yealots? Hah. Yealots. We’ll go with that.

There is no “Kubernetes”

The Yealots want us to believe Kubernetes is a bulbous sort of centralized orchestration monolith. Kubernetes is the thing that scales stuff up and down, Kubernetes deploys things, Kubernetes handles RBAC etc… Perhaps they sometimes accidentally let slip the mention of a “controller” or two – the pod controller, the deployment controller, but Kubernetes runs them. Yeah that’s what it does, Kubernetes runs controllers. Right?

Perhaps the Yealots believe these things themselves? I honestly can’t tell. For example, the first thing you’ll learn from the official docs is that the word Kubernetes itself means something like “helmsman” in ancient Greek. So they’ve named this solution after a single person steering a ship. So how could Kubernetes not be a monolithic orchestrator given the very name is a huge commitment to a metaphor for a monolithic orchestrator? So there’s gotta be a central thing, called Kubernetes which is steering this ship. Right?

Lies my friend. There is no monolithic entity in the system that you can point at which is called Kubernetes. There is no Kubernetes process, nor is there a Kubernetes binary. There is no controller-controller – no captain is steering this ship. In point of fact, Kubernetes is just a proper noun we’ve assigned to a strangely shaped microservices architecture that takes just about the opposite shape that its name implies. A distributed system where no service is called Kubernetes, and every service is decoupled from every other by means of a single, centralized API endpoint, which is also not called Kubernetes.

Meet kube-apiserver

In point of fact, this central API server is called “kube-apiserver”, and there’s absolutely nothing magic about it. I say that, because given the involvement of Golang and containers and cloud-nativity in general, and given the silence from the docs about how messages are actually sent, and the communities apparent aversion to talking about how this works, you might assume some complex binary protocols are involved. Webassembly, GRPC, Protobuf, fancy magic-pants stuff it’s difficult to wrap one’s head around.

Nope. Kube-apiserver speaks HTTP. It’s just a plain-ole restful HTTP API server. And because it’s literally the only interface in the system, that means every service in Kubernetes speaks plain ole HTTP.

Ok that’s fine for this interface, but what about all the other Kubernetes interfaces?

There aren’t any other interfaces. There is only kube-apiserver. This singular interface is used by Humans to do things like get pod status, by machines to do things like deploy new workloads, and even by the internal Kubernetes services to communicate with one another, to, for example, change the state of a pod from Ready to CrashLoopBackoff. All operations, human-beings, RPC, and automation, use the same interface: kube-apiserver

Well that’s obvious bullcrap Dave I can hear you say. What about Apply, Patch, and Update? There are more operations than HTTP GET and POST.

The operations the Yealots talk about like those performed by commands like kubectl apply are details encapsulated in the body of an HTTP request and sent to kube-apiserver by kubectl. Lets take a look at one.

POST /apis/apps/v1/namespaces/foo/deployments?fieldManager=kubectl-client-side-apply HTTP/1.1
Host: 10.96.0.1:443
Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IjRPYWQ3NjZ...
Content-Type: application/json
Accept: application/json

{
  "apiVersion": "apps/v1",
  "kind": "Deployment",
  "metadata": {
    "name": "foo",
    "namespace": "foo",
    "annotations": {
      "kubectl.Kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"apps/v1\",\"kind\":\"Deployment\",\"metadata\":{\"name\":\"foo\",\"namespace\":\"foo\"},\"spec\":{\"replicas\":1,\"selector\":{\"matchLabels\":{\"app\":\"foo\"}},\"template\":{\"metadata\":{\"labels\":{\"app\":\"foo\"}},\"spec\":{\"containers\":[{\"name\":\"foo\",\"image\":\"nginx:1.27\"}]}}}}\n"
    }
  },
  "spec": {
    "replicas": 1,
    "selector": {
      "matchLabels": { "app": "foo" }
    },
    "template": {
      "metadata": { "labels": { "app": "foo" } },
      "spec": {
        "containers": [
          { "name": "foo", "image": "nginx:1.27" }
        ]
      }
    }
  }
}

This is what kubectl sends kube-apiserver when you apply a deployment. It’s an HTTP request with a serialized data-structure of type Deployment in the body. This is the shape every kubectl command you run against a cluster takes. Kubectl creates an HTTP request like this one from your personal computer to the kube-apiserver endpoint described by your current Kubernetes context using a bearer token extracted from your local context. That’s all that’s happening. You could do the same thing with cURL.

Lets look at what happens when we kubectl get pods

GET /api/v1/namespaces/foo/pods HTTP/1.1
Host: 10.96.0.1:443
Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IjRPYWQ3NjZ...
Accept: application/json;as=Table;v=v1;g=meta.k8s.io,application/json

When you call kubectl get pods for example, a GET request like the one above is created by kubectl and sent to the kube-apiserver endpoint. Kube-apiserver responds with a list of pods, which kubectl reformats and parrots back out to you.

Well that’s obvious bullcrap Dave I can hear you say. I can kubectl get logs and return a stream of log messages in my terminal. And everyone knows HTTP get can’t encompass streaming-logs functionality

Sure it can. When you call kubectl get logs kube-apiserver transitions the existing socket to chunked transfer encoding mode, like any other streaming service. It then interacts with the kubelet(s) running on the pod(s) you want logs from and proxies back the logs to kubectl, which parrots them back to you. You’re never talking to anything but kube-apiserver, and you’re never speaking any protocol but HTTP.

Well that’s obvious bullcrap Dave I can hear you say. I can kubectl exec to a pod like remote-login and http get can’t encompass full terminal functionality

Of course it can. When you call kubectl exec kube-apiserver merely upgrades the connection to a bidirectional websocket, interacts with the kubelet on the pod to derive a shell, and then cross-wires stdin, stdout, and stderr accordingly. You’re never talking to anything but kube-apiserver, and you’re never speaking any protocol but HTTP (and websockets).

What do you mean interact Dave?! I can hear you say. Fair point. Kubelet has its own API interface, which is immaterial to the point. (Maybe it’s impossible to talk about Kubernetes without lying?)

The frick is a kubelet Dave?! I can ALSO hear you say. Just… chill for a sec we’ll get to that. What I need you to internalize right now is…

There is only one interface. There is only one protocol.

Kube-apiserver is the one and only interface into the distributed system we describe as “Kubernetes” and kube-apiserver only speaks HTTP verbs like GET, PATCH, POST and DELETE. That’s kinda weird I grant you, but realize how acutely it simplifies things. It means, for example, that all role-based access control lives in one place: kube-apiserver.

Because literally anything in the system that wants to interact in any way with any other component of the system, does so by traversing kube-apiserver, this shared endpoint handily becomes the single point of authorization in the system. All RBAC is implemented on HTTP requests to kube-apiserver. If you, for example, want to GET Pods in the secret-pods namespace, the req will be directed at kube-apiserver, which checks your RBAC. But if a process running in Kubernetes wants to modify the status field of another process in a different Kubernetes namespace, the exact same thing happens, it directs a request to kube-apiserver, which checks its RBAC. Easy peasy.

Hey check that out! Now you have a high-level mental model for how Kubernetes role-based access control works, and lemme tell ya, NOBODY has a mental model of the architecture of Kubernetes role-based access control. You’re already killing it.

So where do the URLs come from?

Lets take another look at the endpoint address kubectl directed our HTTP GET request to when we listed pods earlier. You’ll notice that its directed to this very normal and not at all magical looking URL:

GET /api/v1/namespaces/foo/pods HTTP/1.1

It breaks down obviously. We’re GETing pods from the namespace called foo. In normal-land, you’d describe this as an API endpoint but the Yealots call this a resource.

Hold on Dave, what makes it a resource and not an API endpoint you ask?

Nothing whatsoever. The Yealots just don’t want you to know that a Kubernetes resource is just an API endpoint. I’ll say that one more time; A Kubernetes resource is just an API endpoint. They made up a word for it to confuse you specifically.

Lets look at another API-endpoint– er I mean resource. This time, instead of pods, we’ll take a look at ssl certificates.

GET /apis/cert-manager.io/v1/namespaces/foo/certificates

The first thing I’d like to point out about this resource is the distinction between /api for pods and /apis for certificates. This difference in the API path denotes endpoints that are “core” (bundled together with kube-apiserver), and those that are “custom” (installed by platform engineers as extensions to the API after the fact).

So you can extend the API with new resources, which is the fundamental feature of kube-apiserver. You can make new endpoints– er… resources. When we do this – when we add new resources to kube-apiserver – the yealots call them: Custom Resources. This is a really big deal in Yealot land – prime yealot verbiage – they love blathering on about custom resources, even despite their own unwillingness (or inability?) to define wtf custom resources are. But now you of all people, a Kubernetes hater and enemy of the dogma, know what custom resources are concretely. They’re just API endpoints. And more specifically API extensions. They’re API endpoints we had to teach kube-apiserver about after the fact.

Then in the URL we have cert-manager.io/v1 which is what the Yealots call a “GroupVersion”. Because it’s composed of a group cert-manager.io and a version v1. Together with the certificates moniker at the end of the path, you have what the Yealots call a GroupVersionKind or GVK, which which is a sort of UUID the Yealots use to uniquely identify resources in Kubernetes. Why not use the literal universal resource locator? No idea.

Quick Pro-tip: Yealots will often speak of kinds, expecting you to understand that the word, when they speak it, specifically refers to a GVK when the word itself also carries a very commonly understood definition in engineering vernacular that predates Kubernetes. Prime examples include deployment to specifically mean an instance of an apps/v1 deployment, and application to specifically mean an instance of an argoprojio/v1alpha1 Application. Expecting you to intuit whether they’re talking about a Go type, internal to k8s or not is just one of their fun little quirks, but you can usually spot this one because when they do it, their tone switches from patronizing to extremely patronizing.

If you already have some experience squinting at Kubernetes YAML manifests, you may have seen these same GroupVersionKind values listed in YAML manifests as the apiVersion: and kind:. Why do the Yealots write these as apiVersion but speak them as GroupVersion? Because screw you specifically. Nah just kidding, there’s an actual (super fun) reason for this that most yealots don’t even know about.

In our present example, the GVK of the certificate resource is cert-manager.io/v1 certificate. By comparison, another hint that pods are core resources is the fact they don’t have a group. Not in their endpoint URL, nor in their GVK. They’re just v1/pod (internally to the runtime libs, their group value is the empty string value ""). You might think this is intentional, but no, it’s happenstance. The resources in the "" group (called the core or sometimes legacy group in the k8s runtime libraries) predate the concept of groups in the k8s runtime. Initially, everything was just a version/kind, represented by the apiVersion and kind fields in YAML. It was called apiVersion because group didn’t exist yet.

When the project figured out there were eventually going to be a shitload of apiVersion/kinds, they came up with the concept of groups as a means of adding some scope to the name-space. So now we can have a confusing.io/v1 pod AND a stillconfusing.io/v1 pod without name-collision. That’s cool, but then, rather than extend the schema for the group moniker, they just wedged the group name into the pre-existing apiVersion field and pretended that was acceptable for you and the rest of the world to just deal with.

Fun fact: There’s yet another category of group which ships with kube-apiserver, but is not part of the "" group. Sort of a non-legacy, built-in category. All the resources we added to core after we came up with the concept of groups so we gave them a group instead of adding to core. These ship as part of kube-apiserver and there’s no real way to tell them apart from custom resources, other than to just know. Yealots love that kind of shit. We’ll see one of these resources in action in a bit.

How do we add new endpoi– er.. resources?

Ok, lets review. A resource is just an api endpoint URL on kube-apiserver. Custom ones are added to kube-apiserver after install, but there are a bunch that kube-apiserver already knows about. Some have groups, others don’t.

So how do we go about adding a custom resource? With a CustomResourceDefinition GVK. CRD’s (the only kind ever abbreviated in conversation for some reason) are another great big deal among the Yealots. They love, love, love talking about CRDs, but to remain as confusing as possible, they often conflate the terms custom resource and CRD to describe a thingy on the Kubernetes server. But YOU know the truth: There is no Kubernetes server, a custom resource is an API endpoint on kube-apiserver, and a CRD is what we use to install a new custom resource. CRDs install Custom Resources. They aren’t the same thing. Lets take a look at a CRD:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: certificates.cert-manager.io
spec:
  group: cert-manager.io
  names:
    kind: Certificate
    listKind: CertificateList
    plural: certificates
    singular: certificate
    shortNames:
      - cert
      - certs
    categories:
      - cert-manager
  scope: Namespaced
  versions:
    - name: v1
      served: true
      storage: true
      subresources:
        status: {}
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              required:
                - secretName
                - issuerRef
              properties:
                secretName:
                  type: string
                dnsNames:
                  type: array
                  items:
                    type: string
                issuerRef:
                  type: object
                  properties:
                    name:
                      type: string
                    kind:
                      type: string
                # ...the real one runs several hundred lines longer, trimmed for sanity
            status:
              type: object
              properties:
                conditions:
                  type: array
                  items:
                    type: object

Notice the GVK here: apiextensions.k8s.io/v1 CustomResourceDefinition. This is one of the groups I mentioned earlier that have a proper name, but are built-in and ship with kube-apiserver. So we can surmise from the GVK that /apis/apiextensions.k8s.io/v1/CustomResourceDefinition is the resource endpoint we send HTTP POST’s to when we want to extend the kube-apiserver with a new endpoint– er.. resource. Note that URL isn’t namespaced; new CRDs are applied cluster wide.

If you squint at it, you’ll see the CRD is essentially a resource manifest, with another resource manifest nested in its spec field, including the typed schema for the child kind (in this case, a certificate kind). This is how CRDs work, they describe new kinds and the schema kube-apiserver needs to persist instances of them.

When you apply this manifest to Kubernetes as the Yealots say (but which you now know actually means POST it to the kube-apiserver CustomResourceDefinition endpoint) kube-apiserver responds by internally re-wiring itself to respond on the shiny new certificates URL:

/apis/cert-manager.io/v1/namespaces/<namespace>/certificates

State gets stored in etcd

Part of kube-apiserver’s process of re-wiring itself, includes storing a copy of all CRDs as objects in its underlying etcd database. This enables kube-apiserver to rebuild it’s API surface between restarts. In fact, kube-apiserver stores every modification to every resource in etcd. So after the CRD is applied and the new certificates endpoint is live…

  • kube-apiserver has a copy of the CustomResourceDefinition defining the certificate kind so it’s a permanent extension to the API.
  • When instances of the new certificate kind get POSTed to the new certificates endpoint, kube-apiserver writes them to etcd.
  • When GET requests are made to the certificates endpoint, kube-apiserver reads all the matching certificates records from etcd and spits them out as HTTP response body.
  • When modifications to already existing certificates are PATCHed to the certificates endpoint, kube-apiserver finds the instance referred to by the HTTP request in etcd, modifies it in the way described in the patch, and writes the result back to etcd.
  • When DELETE requests are sent to the new certificates endpoint, kube-apiserver finds the certificate record referred to by the HTTP request in etcd, and deletes it.
  • If someone comes along and deletes the certificate CRD itself, kube-apiserver deletes the resource and ensuing requests to it will 404.

I’m over-simplifying a bit there. Operations like GET don’t necessarily result in an etcd operation, the cacheing you would expect to be there is there.

The frick do you mean an instances of the new certificate kind?! Dave?

The entire point of extending kube-apiserver with new custom resources is to get it to perform CRUD operations on stored instances of the new resource kind in etcd. By instance of the kind I mean a data record with the shape we defined in the CRD spec, which has all the required values filled in. Here’s an example of an instance of the certificate kind:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: foo
  namespace: foo
spec:
  secretName: foo-tls
  dnsNames:
    - foo.example.com
  issuerRef:
    name: foo-issuer
    kind: Issuer
status:
  conditions:
    - type: Ready
      status: "True"
      reason: Ready
      message: Certificate is up to date and has not expired

If you compare this yaml manifest to the CRD above (squinting hard enough), you’ll see that this manifest, manifests the shape described by the CRD above. If you POST this to the certificates endpoint, kube-apiserver will create deserialize it into a new instance of the certificate kind, named foo in the namespace foo with the rest of the settings detailed above, and save it to etcd. If you HTTP GET certificates, you’ll get a list of these.

As far as I can tell, YAML blobs like this one are essentially the Yealot cognitive model for all resources. This is what they see in their mind when you say pod – a YAML manifest. But really, inside Kubernetes, the true nature of every resource is a Go Struct serialized into etcd for storage. The YAML version we humans are forced to deal with is just a temporary exported expression of it – just the human user-interface. I could pause here to marvel at the hubris of a system that expects you to use serialized copies of its internal data-structures as your UI, while incessantly lying to you about the shape of its architecture, but I’m far too grown up and principled for that sort of thing.

ANYWAY. The tl;dr is: This is how all things are modeled in Kubernetes. Everything Kubernetes knows how to deal with – every object it models – pods, deployments, certificates, even extensions to its own API surface. They are all dedicated URL endpoints (called resources) on kube-apiserver with a schema defined by an associated CRD (or a built-in one); their current state all backed by etcd. All intercommunication between Kubernetes entities including human interaction consist of HTTP gets, posts, patches and etc… to the kube-apiserver. Every operation is just an HTTP req against this single endpoint that defines, creates, reads, updates, or deletes resources.

But how does it know how to work with new types it just learned?

Feel free to skip this little section if you aren’t a Go programmer, but if you are a Go progammer, it will occur to you that inside kube-apiserver, the native format of all of these kinds are Go types. Typically, Go programs need to re-compile to learn new types. So how can we extend the API surface of kube-apiserver in runtime without a restart?

The answer is the unstructured package. The short answer here is that kube-apiserver, along with the controllers assisting it, pack custom resources into unstructured.Unstructured types, which in turn implement the runtime.Object interface, which in turn enables us to pass these new types around and strongly-type them later inside a controller that’s compiled with types to ingest and work with the GVK (we’re getting to controllers in a bit). The unstructured envelope is where metadata.name et..al come from in the instance above despite not being in the CRD. Meanwhile, etcd, the persistence layer underlying all this is untyped by design, so it doesn’t need schema.

Well WTF is the point of that?

If you feel like that all seems a bit masturbatory you’re on the right page. What the hell is the point of just writing records to etcd?! How does that become deploying containers, and allocating hardware, and stateful sets, and rollouts, and you know… Kubernetes?

Let me show you something cool (or maybe gross depending on your REST orthodoxy) about kube-apiserver. I actually teased it a bit earlier in this article, when I gave you the example of running kubectl get logs, which resulted in kube-apiserver transitioning the connection to chunked streaming mode.

It turns out? You can do that with every resource. For example:

GET /api/v1/namespaces/foo/pods?watch=true

You’ll notice I’ve tacked on the URL parameter watch=true to this get pods request for pods in the foo namespace. You can tack this on to any HTTP GET request targeted at any endpoint– er resource, and kube-apiserver will respond by transitioning to streaming mode. Your client will block for eternity, kube-apiserver has no means of disconnecting you. On the wire you’ll begin receiving updates for every operation that happens to the resource, or group of resources that match your GET request as they happen in real-time. They look like this:

{"type":"ADDED","object":{"apiVersion":"v1","kind":"Pod","metadata":{"name":"foo","namespace":"foo","resourceVersion":"10234","uid":"a1b2c3d4-..."},"spec":{"containers":[{"name":"foo","image":"nginx:1.27"}]},"status":{"phase":"Pending"}}}

{"type":"MODIFIED","object":{"apiVersion":"v1","kind":"Pod","metadata":{"name":"foo","namespace":"foo","resourceVersion":"10236","uid":"a1b2c3d4-..."},"spec":{"containers":[{"name":"foo","image":"nginx:1.27"}],"nodeName":"node-3"},"status":{"phase":"Pending","containerStatuses":[{"name":"foo","state":{"waiting":{"reason":"ContainerCreating"}},"ready":false,"restartCount":0}]}}}

{"type":"MODIFIED","object":{"apiVersion":"v1","kind":"Pod","metadata":{"name":"foo","namespace":"foo","resourceVersion":"10241","uid":"a1b2c3d4-..."},"spec":{"containers":[{"name":"foo","image":"nginx:1.27"}],"nodeName":"node-3"},"status":{"phase":"Running","podIP":"10.244.2.17","containerStatuses":[{"name":"foo","state":{"running":{"startedAt":"2026-09-10T14:02:11Z"}},"ready":true,"restartCount":0}],"conditions":[{"type":"Ready","status":"True"}]}}}

{"type":"MODIFIED","object":{"apiVersion":"v1","kind":"Pod","metadata":{"name":"foo","namespace":"foo","resourceVersion":"10309","uid":"a1b2c3d4-..."},"status":{"phase":"Running","containerStatuses":[{"name":"foo","state":{"waiting":{"reason":"CrashLoopBackOff","message":"back-off 40s restarting failed container"}},"ready":false,"restartCount":4,"lastState":{"terminated":{"exitCode":1,"reason":"Error"}}}],"conditions":[{"type":"Ready","status":"False"}]}}}

{"type":"BOOKMARK","object":{"apiVersion":"v1","kind":"Pod","metadata":{"resourceVersion":"10402"}}}

{"type":"DELETED","object":{"apiVersion":"v1","kind":"Pod","metadata":{"name":"foo","namespace":"foo","resourceVersion":"10455","uid":"a1b2c3d4-...","deletionTimestamp":"2026-09-10T14:11:03Z"},"status":{"phase":"Running"}}}

Every line is a complete JSON object. They’re newline-delimited. You’d have thought they’d use YAML, but in fact most the responses from kube-apiserver are json. If you squint you’ll see embedded resourceVersion and uid attributes, which are monotonically increasing counters which can be used by client watchers to reconnect to the stream at a named value with eg.. ?watch=true&resourceVersion=10309 or disambiguate races. There are similar hacks in place in kube-apiserver to handle race-conditions on operations to etcd. If, for example, two different clients send delete and update the same time.

So now you have the full picture on kube-apiserver. It listens on a bunch of custom resource URLs for requests. It has a connection to etcd, and an in-memory map of open watch sockets, keyed by GVK/namespace. When a request happens, kube-apiserver services the request, CRUD’ing what it needs to/from etcd, and then summarizing what it did for whatever watchers match the GVK/namespace on the resource.

So how does that help? Woopie-doo, we can watch things.

Enter: Controllers

Lets imagine we had a computer program that knew how to deal with TLS Certs. This program knows how to do things like take a secret and generate a CSR (signing request) for it.

Well, we already have a Kubernetes kind that represents a TLS Certificate. It has all the fields we need to store a cert. The issuer ref, the secret, etc… So we could totally take our computer program that knows how to do things to certs, and we could have it watch all the instances of the certificate kind stored on the Kubernetes cluster. It could totally:

  • watch for changes to certificates and regenerate them if the authority changes for example
  • search for certs that have a secret but not a proper Cert yet
  • wake up every so often and just make sure no certs are about to expire by checking the expiry date
  • Other certificate related bullcrap.

To be useful, our little computer program will need to interact with other cert-related computer programs. For example, if our first computer program encounters a certificate instance that does not have a ref to a non-expiring cert matching its spec, it could generate a signing request. Maybe we have a kind for it called CertificateRequest, which looks like this:

apiVersion: cert-manager.io/v1
kind: CertificateRequest
metadata:
  name: foo-abc123
  namespace: foo
  ownerReferences:
    - apiVersion: cert-manager.io/v1
      kind: Certificate
      name: foo
      uid: a1b2c3d4-...
spec:
  issuerRef:
    name: foo-issuer
    kind: Issuer
  request: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0K...
status:
  conditions:
    - type: Ready
      status: "False"
      reason: Pending
      message: Waiting on certificate issuance from order foo-abc123-1

We would teach kube-apiserver about CertificateRequest resources with a CRD, and then, a second computer program – one that’s back-end connected to a cert-authority and knows how to sign requests like this – could be watching for signing requests like this one. When it sees them, it can take action, signing the request and writing the result as an update directly back to the certificates resource using an HTTP PATCH to the kube-apiserver endpoint. Our first program, still dutifully watching for updates to certificates would then pick up the change like a message in a bottle.

These computer programs, as you may have guessed from the section title, are called controllers. This is essentially how they all work, watching a resource, responding to changes, communicating their response by writing back to kube-apiserver. In real life, controllers tend to be many-to-one with resources. Every resource tends to have a few controllers watching it. Controllers tend to follow the old Unix design philosophy of do one thing and do it well, and they don’t communicate with each other directly, but via HTTP requests back to kube-apiserver. The cert-manager.io/v1 group in real life, for example, has over 20 controllers operating on 6 distinct resources. Each controller watches one resource, has one job, and communicates via HTTP PATCH requests back to instances of resources on kube-apiserver.

Controllers are the turtles of this whole Kubernetes thing. It’s just controllers all the way down. Even kube-apiserver runs several of them to deal with CRD’s:

  • crdController — Watches CustomResourceDefinition and dynamically registers an HTTP route for new GVKs
  • establishingController — flips the CRD’s Established condition once its route is live and serving.
  • namingController — checks for naming conflicts before a route goes live (two CRDs can’t claim the same plural name in the same group).
  • openapiController — regenerates the /openapi/v2//openapi/v3 docs to include the new schema.
  • finalizingController — handles cleanup when a CRD gets deleted

You can write a k8s controller in any language that can make an HTTP GET request with URL parameter support. You could write one in three lines of bash with cURL:

  TOKEN=<some token>
  SERVER=<kube-apiserver ip addr>
  curl -sk -H "Authorization: Bearer $TOKEN" "$SERVER/api/v1/namespaces/foo/pods?watch=true"

Or even a single line with kubectl:

kubectl get --raw "/api/v1/namespaces/foo/pods?watch=true"

Pipe that to whatever you want, and POOF: Kubernetes controller. Check you out, writing Kubernetes controllers already.

More stuff that doesn’t exist.

Now is a good time to talk about some other imaginary stuff that feature heavily in Yealot folklore. For example, without ever being asked, the Yealots will pontificate at length about both the reconcillation loop and operators as if these things exist as first-class concrete features in “Kubernetes”. These, they will absolutely insist are baked into the very fabric of.. uh. Ya know… IT… Kubernetes.

More lies my friend. As you know, Kubernetes doesn’t exist, and things that don’t exist can’t have features, so lets unpack these operator and reconcillation loop (aka control loop) fairytales, because you’re gonna have to deal with them (or at least yealots blathering about them).

The reconcillation (aka control) loop isn’t enforced.

Most controllers, as you would expect, are not written in bash, but rather in Go. This is because the Kubernetes runtime libs are all written in Go, so the upstream definitions for every resource exist concretely in some other golang package upstream of you. So why reinvent a pod struct that may drift from the upstream definition over time, when you can just import the same definition k8s itself uses.

Because most controllers are written in Go, there’s an officially-blessed Golang library for writing controllers called controller-runtime. I’m over-simplifying here, this landscape is an incomprehensible mess like everything else Kubernetes, code-gen and confusing code markers feature heavily, and there are actually myriad semi-blessed ways to write controllers. I’d need a follow-up post to do it all justice, but most controller-writing frameworks import controller-runtime, so it’s something like the lowest-common denominator.

Controller runtime, in its turn, imports and wraps a very convenient Kubernetes client called client-go, which knows how to do things like watch resources without you needing to worry about HTTP clients, bearer tokens, and json serialization. Together client-go and controller-runtime make a contract with you, the controller-author, and THAT contract looks something like this:

Tell me what you want to watch and give me a function to run, and I will watch that thing, and run your function when stuff happens to that thing. Also, I’ll run your function every few minutes regardless of whether something happens or not, just for kicks.

Now, when you create a controller using controller-runtime, the generally accepted “correct” pattern – the guts of the function you provide to the client – looks something like this:

Look at the spec field on the struct passed to us by controller-runtime. Compare it to the status field of the same struct. Are the values of those fields the same? If not, fix it.

When a controller follows this pattern, it is said to reconcile the resource. Marrying the spec (desired outcome as described by the literal spec field) to the state (real world status as depicted by the status field). Because it does this and only this, in a loop, forever, this pattern is generally called the reconcile loop.

This whole-ass stack of assumptions: That you’re writing in Go (or some language with analogous k8s-libs), that you are using controller-runtime and client-go (or libs with analogous contracts), that the resource in question has something updating its status over time for you (updating status in your function is a no-no), that you have provided a function for a kind that even cares about spec and status etc.. etc.. That whole assumption-sandwitch is what the Yealots are referring to when they speak of the Kubernetes Reconcillation loop as if it’s a written-in-stone thing Kubernetes does.

It’s not. Reconciliation is entirely implemented in runtime lib as set of optional features provided by the lib, combined with idiomatic patterns to which the community of controller writers generally aspires to adhere. Most controllers do, but plenty do not, by virtue of the fact they happen to do things that simply don’t fit the pattern.

And, as you’ve seen, we can write a controller in any language, completely ignoring controller-runtime and its other-language-analogous libraries, and the controller we write may do anything we want. To the moon Alice; the Yealots can’t stop us. A controller is nothing but a computer program that watches a resource and takes action. Maybe our controller writes back to kube-apiserver, maybe not, but there is no such thing as an enforced first-class feature of “Kubernetes” called the “reconciliation loop”.

Operators don’t exist

The reconciliation loop, it could be argued, kinda exists in the form of an idiomatic pattern that’s implemented by core libraries, but Operators by comparison are outright vaporware. The Yealots will assert Operators exist as either a controller-like entity that is distinct in some unspecifiable way from a controller, or a higher-level controller-managing entity/thing, which orchestrates controllers together in ways that specifically enable the maintenance of applications (by which they do NOT mean application kinds). Sometimes Yealots will assert the former, and when proven wrong, proceed to assert the latter when unfortunately, neither of these things are the truth. The truth is: Operators straight up don’t exist. This is how they’re described in the the official documentation:

Operators are software extensions to Kubernetes that make use of custom resources to manage applications and their components. Operators follow Kubernetes principles, notably the control loop.

Software Extensions. Obviously not controllers right? I mean, if they were just controllers, then the docs would just call them controllers right?

Lies my friend. I invite you to sally forth and examine any project that describes itself as an “operator”. Check out the external secrets operator for example. Spoiler alert; what you’ll invariably find is a bunch of controllers, which are usually in a directory literally named “controllers”. If you choose at random, any one of these controllers that are in the controllers directory – their name suffixed with _controller.go – what you’ll find is that they import controller-runtime libs and proceed to use them to voluntarily implement the reconcile pattern. Operators don’t exist. Ya can’t make this stuff up. Er.. well I suppose you can, I mean the Yealots totally did.

The crushing irony of this myth is that there is actually a controller-manager package with a manager type in the controller-runtime lib, which the Yealots never talk about. That’s a real piece of code that exists; ctrl.Manager. When I said earlier that there were 20-plus controllers in the cert-manager.io group, all of those controllers are part of a single process (pod), running beneath a controller manager. So why is external secrets an operator and cert-manager not? Because the former asserts operator in its name. Some people call them both operators.

This Operator crap sincerely confounds me (if you didn’t pick up on the salt). I really have no idea what purpose this fairytale serves, nor can I imagine why it is so pervasive in the community to the extent of having massive k8s projects that call themselves operators. Especially when a simple, concrete understanding of controllers, which you now posses after like 7 minutes of reading fully describes the functionality that’s repeatedly asserted as belonging to these imaginary “operators”.

ANYWAY, when the Yealots point you to the “foobar operator” or whatever, don’t believe their bullshit. What you’re looking at is nothing but a gaggle of controllers calling itself an operator. They watch a bunch of little resource endpoints and communicate to each other or other cluster components via HTTP requests directed to kube-apiserver.

And that’s legit ALL that’s happening inside Kubernetes.

In fact, what I just described to you, a bunch of little computer programs called controllers listening to a bunch of API endpoints called resources and doing whatever the heck they want is one-hundred percent all that is happening inside “Kubernetes”.

The pod controller watches pod resources and does things with pods. The deployment controller watches deployments and does things with deployments. The entire universe of Kubernetes related complexity reduces to this single pattern. You can extend the API to “learn” about whatever kind of thing you care about, and then write a controller to operate on modifications to it over time. If you wanted to, you could even create an Operator CRD and apply it to your cluster. Then at least “Operators” would exist somewhere in the real world.

ANYWAY.

In the opening paragraphs I described Kubernetes as a strangely shaped microservices architecture that takes just about the opposite shape its name implies. I sincerely hope you now have the true shape of it in mind. The single pattern by which all things in Kubernetes work, is controllers watching resources and doing whatever they want. All the esoterica that remains are details inherent to one controller or another. And the preponderance of that it is jargon made up by a controller to obfuscate some otherwise simple concept you probably already have a solid cognitive model for. Don’t get me wrong, controllers do some crazy stuff, but now we have the foundation necessary to begin unraveling the rest of the landscape. A straightforward path for “learning Kubernetes” laid out now before us: Controller by controller.

Although I’m unsure I have the stomache for it, I’d like to follow-through on that strategy, with an exploration of kubelet, which is one of the most complex controllers. Kubelet listens for changes to ‘v1/pods’ and reconciles workload scheduling (running containers on hardware). This gets pretty interesting because kube-apiserver can’t run without kubelet, and yet, kubelet can’t do anything without kube-apiserver. A bootstrapping hack called “static pods” solves this. Once you understand kubelet you’ll also understand pods (Kubernetes’s process model), nodes (its hardware abstraction), and volumes (its storage abstraction) – Fundamentals we can use to build up to workloads and the controllers used to manage them.

Until next time.

–dave