google kubernetes engine - Is it possible to delete and re-create GKE nodepool with new machine type that is managed through terraform?

Question

I want to change the machine type on my gke nodepool to better match my cpu vs memory usage, but I am having a lot of trouble with getting terraform to delete this nodepool and re-create it. I am locked into having a default nodepool because of the module that has been used in the past to create the cluster. The cluster is in a shared module with the nodepool, so I cannot permanently delete the nodepool through terraform without also deleting the cluster which would affect everybody that is expecting the cluster to stay available.

So my solution was to create an additional and temporary nodepool, migrate all pods to it, cordon and drain the default nodepool, then through terraform, change the nodepool's machine type so that it would recreate without affecting any running deployments, pods, etc. However, terraform did not attempt to delete the nodepool, only to re-create it. Therefore it failed with a 409, nodepool already exists.

My question is, can I delete a nodepool manually - through gcloud commands or other such methods - and then re-run terraform and hopefully not experience the 409 (nodepool already exists) error? Are there any consequences this could have on the terraform state file? Would terraform fail completely if I deleted a resource (the nodepool) that it was expecting to exist?

Note - I did my best to include all information, but if there's more info needed please let me know and I will attempt to edit this and add more info. Thanks.

question from:https://stackoverflow.com/questions/66050047/is-it-possible-to-delete-and-re-create-gke-nodepool-with-new-machine-type-that-i

Answer 1

If your issue is the default terraform behaviour of destroying resources before creating new resources you could try to use the 'lifecycle' meta-argument in your terraform configuration.

resource "google_container_node_pool" "example" {
  # ...

  lifecycle {
    create_before_destroy = true
  }
}

This would ensure that your node pool which you want to replace stays up until the replacement pool with new machine type has been created.