python - Airflow + Kubernetes Executor too old resource version

Question

I have a strange behaviour of Airflow with Kubernetes executor. In my config tasks run in dynamically created kubernetes pods, and i have a number of tasks that runs once or twice a day. Taks itself is python operators that runs some ETL routine, dag files is syncing via separate pod with git repo inside. For some time all working ok, but not so long ago in scheduler pod i begin to see error

kubernetes.client.exceptions.ApiException: (410)
Reason: Gone: too old resource version: 51445975 (51489631)

After that error is appear, old pods from tasks won't be deleted and after some time new pods can't be created and tasks won't run(or to be more precise it freezes in "scheduled" state). In this situation only deleting the scheduler pod with

kubectl delete -n SERVICE_NAME pod scheduler 

and waiting for kubernetes to recreate it helps, but after some time error appears again and situation repeats. Another strange thing, that this error seems only appear after scheduled tasks run. If i trigger any task any number of time it via UI no error appears and pods are created and deleted normally. Airflow version is 1.10.12 Any help will be appreciated, thanks!

question from:https://stackoverflow.com/questions/65643712/airflow-kubernetes-executor-too-old-resource-version

Answer 1

This is because of Kubernetes Python client version 12.0

Restrict the version to <12

pip install -U 'kubernetes<12'