changing replicas create/delete pods immediately during paused deployment, but It is not specified in the kubernetes official docs

Question

while i was reading Pausing and Resuming a Deployment from kubernetes official docs, here i found that i can apply multiple fixes in between pausing and resuming without triggering unnecessary rollouts.
Now, when i only changed the image of the deployment then it works fine, but when i changed the number of replicas in the deployment manifest and applied that then the pod created/deleted immediately according to the replicas number even the deployment was in paused state.

Now, my question is, why it is happening? doesn't it allowed to change the replicas during paused deployment? or only image changing is allowed? the doc did not specify it.

Another thing is if i change number of replicas and also the image then only the replicas get applied during paused deployments and created pods with the previous image not the current one.

the manifest file of the deployment that is used is:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.16.1
        ports:
        - containerPort: 80

Updated Question (Gave full details)

What happened: While I was reading Pausing and Resuming a Deployment portion of kubernetes official docs. I paused a deployment and during deployment I changed the number of replicas in the deployment manifest file. After that when I applied that changes using kubectl apply -f deployment.yaml then according to the new number of replicas the pods immediately created/deleted even during the paused deployment state, those pods came with the previous image.

First I created the deployment with this manifest using the command kubectl apply -f deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.1
        ports:
        - containerPort: 80

after that I opened two terminal for monitoring the pods and replicaset by using the commands:

kubectl get pods -w
kubectl get rs -w

after that I paused the deployment using kubectl rollout pause deployment/nginx-deployment command.

then I change the manifest and it like below:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 5 # <------ here changed the number of replicas
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.16.1 # <---- here changed the image
        ports:
        - containerPort: 80

after applying this manifest using kubectl apply -f deployment.yaml the things that I saw is: two new pods get created with the previous image image: nginx:1.14.1

What I expected to happen: According to the pausing concepts, any changes into the PodTemplateSpec of the paused Deployment will not trigger new rollouts as long as it is paused. Now why did the changing replicas implemented during the paused deployment.

Anything else?: Also if I use horizontalautoscaler the same thing happens.

Environment:

• Kubernetes version (use kubectl version):

Client Version: v1.20.1
Server Version: v1.19.1

question from:https://stackoverflow.com/questions/65904072/changing-replicas-create-delete-pods-immediately-during-paused-deployment-but-i

Answer 1

I don't think this is an issue. I belive this is expected behaviuor and I will try to explain why I think so.

Pause functionality is part of kubectl rollout command.

kubectl rollout pause ...

and it is used to pause a specific deployment from rolling update.

But what does rolling update actuall is? Let's try to define it. In k8s docs we can read:

Rolling updates allow Deployments' update to take place with zero downtime by incrementally updating Pods instances with new ones

So we can see that rolling update is being used to apply changes with zero downtime. This includes changes to image version, container limits etc. Basically anything related to how application runs and behaves.

Now, does changing number of replicas impact the running application? Does it change the way it behaves? No! It only adds/deletes replicas of the same application. Users still can use the same version that was available before pausing the deployment. No changes are applied.

After you resume the deployment, all the changes submited after the pause will be applied.

So to summarize, I belive this is expected behaviour, since the change of number of replicas does not change the actual application and in the meanwhile the actuall update is indeed paused.

If you are still not convinced, I'd recommend you to open an issue on k8s github repo and ask developers directly.