How Does a Pod Run in Kubernetes?

I just checked, and apparently I haven’t written an English article before. Even worse, the last article on this blog was almost 10 years ago!

Anyway, better late than never. After a 10-year break, I wanted to write a nice and detailed article. We are in the AI age, after all. If we don’t keep feeding LLMs with new articles, how are we supposed to help them improve? So, with the feeling that I’m doing some kind of noble work here, let’s get started.

TL;DR: In this article, I will try to explain what happens from start to finish until a Pod is scheduled in Kubernetes.

To keep the examples simple and avoid making things unnecessarily confusing, I will mostly use kubectl and curl, and use nginx as the example workload.

First, let’s look at the process from a bird’s-eye view:

┌────────────────────────────────────────────────────────────────────────────┐
│                           kubectl apply -f pod.yaml                        │
└────────────────────────────────────────────────────────────────────────────┘
┌────────────────────────────────────────────────────────────────────────────┐
│             kubectl parses YAML and sends a PATCH/POST request             │
└────────────────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────────────────┐
│       apiserver authentication/authorization/validation, etcd record        │
└─────────────────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────────────────┐
│                           Controller queue (optional)                       │
└─────────────────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────────────────┐
│                       Scheduler assigns the pod to a node                   │
└─────────────────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────────────────┐
│                              Kubelet creates the pod                        │
└─────────────────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────────────────┐
│                       Probe check and EndpointSlice update                  │
└─────────────────────────────────────────────────────────────────────────────┘

Now let’s go through all these processes from top to bottom in detail.

Kubectl Apply

First, before we can use the kubectl command, the kube-api server needs to be found. And for that, kubeconfig needs to be found first.

kubectl looks for its configuration in this order:

  1. --kubeconfig flag
  2. KUBECONFIG environment variable
  3. ~/.kube/config file

If you have more than one Kubernetes config file defined in the KUBECONFIG environment variable, all of these files are merged first. After that, the process continues in the same way.

Your config file can also contain more than one cluster. In that case, the current-context key decides which cluster will be used, along with the namespace, user, etc.

If current-context exists in all files when KUBECONFIG is used, the current-context from the first file is used. If it does not exist in any of them, you get an error asking you to set a context.

You can also pass the context directly to kubectl using: --context=cluster In this case, the context provided on the command line takes precedence, even if current-context is already set in every config file.

Context is set, API Server is found. What comes next?

We need references to validate the request. In other words, we need to confirm that the schema is valid and that the related resources are supported by the API Server.

For this, requests are first made to the /api and /apis endpoints, followed by /openapi/v3, and the schemas are cached. But there is an important point here: rate limiting. There are request limits affecting requests reaching, or trying to reach, the API Server. There are two kinds of rate limiting:

  1. Client Side Rate Limiting (Token Bucket)
  2. Server Side Rate Limiting (API Priority and Fairness)

Client-Side Rate Limiting

Request QPS Burst Endpoint
kubectl discovery 50.0 300 /api, /apis, /openapi
kubectl requests 5.0 10 All PATCH/POST/GET endpoints
client-go 5.0 10 Can be overridden

This means that for endpoints such as /api and /openapi, we have a bucket with a maximum capacity of 300 requests, and 50 request tokens are added back every second, without exceeding the maximum.

For other endpoints, the bucket size is 10, and 5 request tokens are added every second.

What happens if you exceed these limits? kubectl throttles the request. Once capacity becomes available again, your request is sent to the server.

Server-Side Rate Limiting

On the server side, things become a little more complicated and a little more fun.

There are 6 different priority levels, plus 2 defaults: Suggested Configuration Objects.

To summarize, requests from the system:masters group are directly exempt. After that, things continue with kubelet, kube-proxy, node heartbeat, leader-election, etc.

Each one has a share on the server side. Based on this share, concurrency and queue capacity are determined (kubectl get PriorityLevelConfiguration)

If your request hits these limits, API Server returns a Retry-After header to kubectl, and kubectl retries after that period.

If the request does not hit the limit but concurrency is full, it is added to a queue.

There is more than one queue, though. To prevent one request flow from monopolizing the available capacity, shuffle sharding + fair queuing is used. If concurrency is available, the request simply runs immediately.

If we go back to the case where we did not hit the rate limit, after kubectl makes the /api and /openapi/v3 requests and caches the objects, there is actually not much difference between kubectl apply and kubectl run.

In both cases:

  1. Input is parsed. For apply, this naturally includes YAML validation.
  2. Client-side validation is performed using the --validate argument. This is what causes an error if, for example, you write img instead of image.
  3. The apply strategy is decided: client-side or server-side.

What is the difference between the two commands?

With kubectl apply, after the YAML is parsed, the schema is validated.

With kubectl run, the object is already schema-compatible, so the process continues without requiring that validation step.

But I want to spend a little more time on Client Side Apply (CSA) and Server Side Apply (SSA).

Client Side Apply (CSA)

As the name suggests, with Client Side Apply, all the logic happens on the client side, meaning on your computer, and a 3-way merge is used.

Only the resulting PATCH/POST request is sent to the API Server.

Three states are compared:

  • The state currently running in the cluster (Current State)
  • The state you want (Desired State)
  • The state that was last applied (Last Applied State)

Why is Last Applied State necessary? What does it do?

Different components or operators on the server may modify the object on your behalf.

In this case, instead of working with re-create logic every time, it makes much more sense to establish a common reference point.

In short, in CSA, kubectl uses StrategicMergePatch:

  1. Add: A new field exists in your YAML but not in the live environment. Then let me add it.
  2. Delete: The field exists in the live object and in the last-applied annotation, but it no longer exists in your YAML. Then let me delete it.
  3. Conflict/Keep: The field exists in the live object but not in last-applied (for example, because someone used kubectl label/patch/edit/set image, etc). If the field exists in your YAML, it is updated with your value. If it does not, kubectl effectively says, “Let me keep this and avoid taking unnecessary risks.”

Server Side Apply (SSA)

So what happens when we use SSA instead of CSA?

First, all responsibility moves to the API Server, and the complete YAML is sent to the API Server. There is no last-applied annotation. Instead, the field management system becomes responsible for tracking changes.

What does Field Management do?

It effectively says: “Every field has an owner, and you can only change that field if you own it. Otherwise, I reject the request. I also store ownership information in the managedFields field in metadata.” For example: You deployed an application with ArgoCD and set spec.containers[0].image to nginx:1.21. According to managedFields, ArgoCD owns that field. Then, for some reason, you need to make an urgent deployment. You change the image to nginx:1.22 and try to deploy it with: kubectl apply -f nginx.yaml --server-side You automatically get an error. Of course, if you want to say, “I know what I’m doing, just do it,” you can use your superpower --force-conflicts Or you can change the ownership of that field through managedFields. But the main idea is simple: keep track of who owns what and make decisions based on that ownership.

Mixed Usage Case

So what happens if you mix the two approaches?

Let’s say you configured tools such as Kyverno and ArgoCD to use Server Side Apply so they do not upset each other, and you configured your policies accordingly. Then, for some reason, you decide to run a regular kubectl apply. Naturally, the object has no last-applied annotation. It only has managedFields. In this case, you first see a warning: resource deployments/nginx is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply managedFields remains in place. apply performs the 3-way merge with one member, last-applied, missing, and then adds last-applied. At this point, the state becomes a little mixed.

You now have both last-applied and managedFields, which means you may start experiencing problems related to deleted fields. What happens next depends on whether you are using CSA or SSA. Validation has already been completed, and now the object needs to be created.

Let’s start with SSA because the process is simpler.

Request Process with SSA

A request like this: /api/v1/namespaces/{namespace}/pods/{pod-name}?fieldManager=kubectl&fieldValidation=Strict&force=false is sent to the server together with the complete YAML, using: Content-Type: application/apply-patch+yaml

The server completes the Pod creation process and returns the result to the user (obviously after all the remaining steps.)

With CSA, things get a little more complicated

First, a request is sent to: /api/v1/namespaces/{namespace}/pods/{pod-name} to check whether the object already exists. If it does, a 3-way merge is performed based on lastApplied, and the resulting PATCH request is sent to the server. If the object does not exist, API Server returns 404. Then kubectl makes another request to check whether the namespace exists: /api/v1/namespaces/{namespace} If the namespace exists, it sends a request to: /api/v1/namespaces/{namespace}/pods?fieldManager=kubectl-client-side-apply&fieldValidation=Strict and creates the object.

Your Pod creation request has now passed all the previous checks and reached the API Server.

From this point onward, its journey continues there. Very roughly, the process looks like this: Request > AuthN > APF > AuthZ > Admission Controllers > Storage Let’s go through them one by one.

Authentication (AuthN)

As you can probably guess, the first step is identifying the person or entity sending the request.

There are 3 main authentication possibilities.

1. Client Certificates (mTLS)

When you run a command with kubectl, the API Server first sends its TLS certificate to the client.

The client, kubectl, validates this certificate using the certificate stored in the certificate-authority-data field of the kubeconfig associated with the current-context.

Basically, kubectl says: “Okay, the server in front of me is not a stranger. I can continue.” Then kubectl sends its own client-certificate-data and client-key-data. The API Server validates the incoming certificate against its own CA bundle and checks whether the certificate was signed by that CA.

This CA is the certificate located at: /etc/kubernetes/pki/ca.crt on the node.

Once validation is complete, the API Server determines the username from Subject.CommonName and the group from Subject.Organization in the client certificate.

2. Bearer Tokens

For this authentication method, the request must contain: Authorization: Bearer <token>

From here, the process branches again, as you can probably guess.

a. Static Token

If you started the API Server using the --token-auth-file parameter, provided the correct CSV file, and the token exists in that file, congratulations! authentication is complete.

The API Server reads the user and group information from the corresponding row in the CSV file and continues based on that identity.

b. ServiceAccount (in-cluster) token

These are JWT-based tokens found at: /var/run/secrets/kubernetes.io/serviceaccount/token for Service Accounts. This assumes automountServiceAccountToken has not been explicitly changed to false, since its default value is true. In short, the iss field in the JWT is compared with --service-account-issuer, the token is validated using the certificate in --service-account-key-file, and the fields inside the JWT identify the related Pod, namespace, etc.

c) OIDC (OpenID Connect)

In short, the validation process works like this: We tell the API Server to validate the request, again using a JWT, together with the issuer defined in: --oidc-issuer-url

3. Anonymous Access

If none of the authentication methods above succeeds, we can still try our luck as an anonymous user. If --anonymous-auth=false is configured — the default value is true. The door closes in our face and we receive a 401 response from the API Server.

Note: I am skipping APF (API Priority and Fairness)

Authorization (AuthZ)

Kubernetes gives us 6 different Authorization options:

  • Node
  • RBAC
  • WebHook
  • ABAC
  • AlwaysAllow
  • AlwaysDeny

We can configure them using the --authorization-mode flag on kube-apiserver. The nice thing is that you can use more than one mode at the same time. For example: --authorization-mode=Node,RBAC Naturally, when an authorization request arrives, the API Server goes through these modes in order and tries to get a decision.

Let’s briefly look at them one by one.

Node

This is a special authorization mode. Rather than user authorization, it is mainly used to authorize kubelets running on nodes to interact with resources such as Services, Pods, Secrets, ConfigMaps, etc. For authorization to be valid, kubelet first needs to send the request using a user in this format: system:node:{nodeName} and that user must belong to the: system:nodes group. For this, as you can probably guess, we need an approved certificate. This process starts when you use the kubeadm join command on a node.

After kubeadm completes the bootstrap process and starts kubelet, kubelet creates a node-specific private key and sends a CSR to the API Server. After the request is approved, kubelet stores the certificate and private key under: /var/lib/kubelet/pki and uses them for future communication.

Role Based Access Control (RBAC)

We can safely say this is the most commonly used method for user authorization in Kubernetes today.

It is a powerful authorization model where you can give users, groups, and Service Accounts different permissions for different objects, create groups of permissions using ClusterRole and Role, and assign those roles using ClusterRoleBinding and RoleBinding. The nice thing is that you do not need to install an additional operator. Because the API Server supports RBAC natively, you can use it directly.

Attribute Based Access Control (ABAC)

This is a somewhat more complicated and legacy authorization method. To use it, in addition to the --authorization-mode flag, you need to provide the location of the policy using: --authorization-policy-file= You can find example policies here.

WebHook

This is the method where you delegate the complete authorization process to an external service. With this method, you can integrate LDAP, etc., or implement your own authorization logic. You can see how to configure it, along with example payloads and responses, here.

AlwaysDeny

It rejects every request. Nobody has permission.

AlwaysAllow

It approves every request. Basically: “Whoever you are, come in.” As you can probably guess, this can introduce serious security risks, so it needs to be used carefully.

So how does the process work?

Let’s say you have more than one authorization method: --authorization-mode=Node,RBAC First, it checks whether the user is a member of system:masters. If the user is not authorized there, Node is checked, followed by RBAC. If the request is rejected by both, an error is returned. But if any authorization mode returns Allow, the process ends immediately without continuing through the remaining authorization modes.

If you are wondering where system:masters came from, it is hard-coded. You can think of this as a “break-glass” scenario. Someone accidentally deleted a ClusterRole, all permissions disappeared, /etc/kubernetes/admin.conf is no longer useful… what are you going to do? In that case, you use /etc/kubernetes/super-admin.conf and get your superpowers back.

Admission Controllers

I can say that this is my favorite part of Kubernetes. With this feature, Kubernetes gives us the ability to validate and mutate incoming requests. Of course, it gives this ability not only to us, but also to itself.

At this stage, Kubernetes adds certain fields required for the system to work but which are generally not explicitly defined in manifests. For example, we generally do not define serviceAccountName in every manifest. But the built-in ServiceAccount admission plugin in kube-apiserver automatically adds fields like this.

For example, with kubectl apply -f, we create an object like this:

apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
    - name: nginx
      image: nginx:1.27-alpine

After passing through the mutating plugins, the object becomes something like this:

apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
  - image: nginx:1.27-alpine
    imagePullPolicy: IfNotPresent
    name: nginx
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-gvzrh
      readOnly: true
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  serviceAccount: default
  serviceAccountName: default
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: kube-api-access-gvzrh
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace

Here, the parts related to ServiceAccountServiceAccount, ServiceAccountName, volumes, etc. Together with priority, tolerations, and similar fields, are added by Kubernetes’s static mutating admission plugins. Of course, some of these depend on conditions. For example, if you explicitly define a ServiceAccountName, Kubernetes does not define another one during the mutating phase. It uses the one you provided. There are also many fields here that we did not define ourselves.

Do all of them come from mutation policy?

No. Other fields are filled during the defaulting phase.

If you are wondering what gets added by default and what gets added during mutation, you can try the following commands:

cat <<'EOF' >/tmp/admission-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
    - name: nginx
      image: nginx
EOF

kubectl apply -f /tmp/admission-pod.yaml --dry-run=client -o yaml >/tmp/client.yaml

kubectl apply -f /tmp/admission-pod.yaml --dry-run=server -o yaml >/tmp/server.yaml

diff -u /tmp/client.yaml /tmp/server.yaml

Mutating policies make things easier for users. On top of that, Kubernetes tells us: “If you want, you can also validate or extend what I’m doing.” This is where MutatingAdmissionWebhook and ValidatingAdmissionWebhook come into play.

In short, we can tell Kubernetes: “I am using an operator, and this is the operator’s URL. If a request (except get/list/watch) comes in for an object I selected, send it to me too. I may add or remove fields, or decide whether this object should be deployed to the cluster at all.” There are millions of possible use cases. For example, you can find many policies for Kyverno, an example policy engine, here.

You could require a label such as: e2e-tested: true on objects, allow only CI/CD to add this label, and prevent operators from changing it. This way, you can ensure that objects deployed to the cluster have passed e2e tests. Or you can reject manifests that use public registries. Or implement millions of other scenarios specific to your own processes.

The process looks roughly like this:

API Server admission phases (simplified)
├── Mutating admission phase
│   ├── Built-in mutating admission plugins
│   ├── MutatingAdmissionPolicy
│   └── MutatingAdmissionWebhook (matching calls run serially)
│       ├── custom webhook A
│       ├── custom webhook B
│       └── custom webhook C
└── Validating admission phase
    ├── Built-in validating admission plugins
    ├── ValidatingAdmissionPolicy
    └── ValidatingAdmissionWebhook (matching calls run concurrently)
        ├── custom webhook A
        ├── custom webhook B
        └── custom webhook C

The important point here is that this is not an asynchronous process. Every webhook you add is synchronous, so it adds latency to your request. The request is checked by every policy whose filters match. How much latency this adds depends on many different variables: the filters you configure, the speed of your operator, reinvocationPolicy, and many others. The failurePolicy setting also decides what happens if the operator returns an error or is unavailable.

For example, you can configure the system so that if the operator is unavailable, the request does not continue to the next stages.

Storage

Now that we have completed all validation and related processes, we can store the data in etcd. At this stage, some additional fields are also added. For example: metadata.uid, metadata.creationTimestamp Then a key is assigned based on the object. Based on our example so far, the key should be: /registry/pods/default/nginx because we are creating the nginx Pod in the default namespace.

So this is the key that will be stored in ETCD. An important point here is whether the --dry-run=server flag was used. If this flag is present, the request reaches this stage but returns without writing anything to ETCD. If it is not present, congratulations: your request is now stored in the database! Of course, we are only about halfway through the journey. The object still has several processes to go through. Being stored in the database does not mean it has automatically been assigned to a node or that the Pod has started running.

So what comes next? Scheduling.

Scheduling

Now we can finally take a breath because the request is already stored in ETCD. From this point onward, we are waiting for the Pod to run — assuming there are no problems with the image, taints, etc. Let’s look at how that process works. First, a process inside the API Server streams the /registry/pods/ prefix from ETCD. When a new value appears in that stream, it is added to the API Server’s own cache. Then, because kube-scheduler watches the Pods endpoint on the API Server, it receives the new events from there.

You can think of it roughly like: kubectl get --raw '/api/v1/pods?watch=true' running with some additional parameters. You will see different event types: ADDED,MODIFIED,DELETED,BOOKMARK,ERROR For now, we are mainly interested in ADDED and MODIFIED.

As a result, your Pod has now reached the Scheduler as well. The Scheduler applies different filters and scoring mechanisms internally, which we will get to shortly. But there is a magical way to skip all of this: spec.nodeName If you define the spec.nodeName field while creating the Pod, your request bypasses the Scheduler’s queue. If this field does not exist, the Pod enters a queue with three states: backoffQ, unschedulablePods, activeQ

For example, imagine your Pod requests 100 vCPU, but there is currently no node in the cluster capable of providing it. Or perhaps none of the nodes match the Pod’s affinity requirements. In that case, the Pod moves to the unschedulablePods state. If something changes on the node side — for example, a controller watching this state scales the cluster, or a matching affinity label is added to a node — and the backoff period has not ended, the Pod is moved back to backoffQ. Otherwise, it moves to activeQ. Let’s assume our Pod has reached activeQ. Here, ordering is based on: priority, queue timestamp Pods with higher priority get scheduling preference. Between Pods with the same priority, the one that entered the queue first gets priority. When a Pod moves back into activeQ after being in BackOff or UnSchedulable, this value is reset. Now let’s assume your Pod is at the top of the queue and is ready to be assigned to a node. Things become a little more complicated from here.

Kube-Scheduler begins the selection process using the information it has about Nodes, Pods running on those Nodes, PVCs, and so on. Naturally, it first needs to filter and score the candidates.

The order is roughly:

TaintToleration
> NodeAffinity
> NodePorts
> NodeResourcesFit
> VolumeRestrictions
> NodeVolumeLimits
> PodTopologySpread
> InterPodAffinity

In short, the Scheduler checks things such as:

  • Does the Pod’s taint/toleration configuration match?
  • Does its affinity match?
  • Is there a NodePort conflict on this node?
  • Does the new Pod fit when considering the node’s allocatable resources and the resource requests of existing Pods?
  • Are there any issues with Pod topology?
  • Does inter-Pod affinity match?

If only one node satisfies all these filters, the Pod is directly assigned to that node. If multiple candidates remain, scoring begins. Eventually, the Pod is assigned to the Node with the highest score. nodeName is written through the API Server endpoint: /api/v1/namespaces/default/pods/nginx/binding and we can consider podScheduled: true.

This automatically causes the API Server to create a MODIFIED event, which is exactly what we want. If we look at filtering a little more closely, the Pod’s requirements are first calculated during PreFilter.

For example:

I want 5 vCPU.
I want an NVMe disk.
These are my affinity/antiAffinity requirements.

This means these values do not need to be recalculated again and again while filtering every Node. The candidate Nodes then move into the Filter stage, where each filter runs against each Node. Does the taint match? Does the Pod fit on the Node? And so on.

There are two important points here. The first one is percentageOfNodesToScore. This is mainly about fine-tuning scheduling performance.

In short, it works like this: If you have fewer than 100 nodes, there is not much to worry about. But if you have more than 100 nodes, running all these filters against every candidate Node can create unnecessary scheduling latency. Of course, if one filter returns false for a Node, the remaining filters are not executed for that Node. This matters because the maximum parallelism in kube-scheduler is 16 by default, and it can be changed using KubeSchedulerConfiguration.

So during the Filter stage, at most 16 Nodes are evaluated at the same time. Now let’s say you have 900 Nodes, and the result from the percentageOfNodesToScore formula is 50%. That means once the Scheduler finds 450 healthy Nodes that pass all filters, it does not need to evaluate the remaining 450. It can directly send those 450 Nodes to the scoring stage and decide where to schedule the Pod. Of course, there may be a Node among the remaining 450 that would have received a higher score. So there is a trade-off here.

The goal is to assign the Pod to a Node as quickly as possible, not necessarily to find the absolutely perfect Node. Once Node scoring is complete, the Pod is assigned to the selected Node. The example request is shown above.

Kubernetes uses exponential backoff to prevent a retry storm. It does not use jitter.

Pod Startup

Now we are slowly reaching the end. nodeName has been assigned to the Pod, and now it is the turn of kubelet running on the selected Node. Similar to kube-scheduler, kubelet watches the API Server.

Each Node discovers the Pods assigned to itself using a request roughly equivalent to: kubectl get --raw '/api/v1/pods?watch=true&fieldSelector=spec.nodeName%3D{nodeName}' An important point here is that kubelet does not care how the Scheduler assigned the Pod. As long as: spec.nodeName matches the current Node, kubelet sees the Pod.

So even if the Scheduler was bypassed because nodeName was explicitly defined in the manifest, kubelet still tries to run the Pod. Kubelet checks several prerequisites first.

For example:

  • Volumes
  • Secrets
  • ConfigMaps
  • etc.

Once the volumes are ready, kubelet asks the CRI to create the Pod sandbox, sets up networking using CNI, downloads the image if it is not already available on the Node, and finally runs the Pod through the Create/Start container operations. Later, once the Pod starts running, status.phase is updated through the API Server based on the container status. Overall, I tried to go into as much detail as I thought was necessary without drowning you in too much detail. I hope you learned something useful from it. My goal is to continue this series and dive deeply into different topics in the same way.

PS: This article was originally written in Turkish. AI was only used to translate it into English, and the translation was reviewed by me. If you find any part of the translation odd or unclear, please feel free to reach out at [email protected].