I'm currently trying to run a minimal Airflow ennvironment to test a few DAGs on my local machine. The setup is quite basic: I am using docker-compose
to spin an Airflow webserver and a scheduler after initialising the local sqlite db.
My docker-compose.yml
file looks like the following:
version: '3'
services:
webserver:
image: apache/airflow
container_name: airflow_webserver
restart: on-failure
volumes:
- ./dags:/opt/airflow/dags
- ./plugins:/opt/airflow/plugins
- ./scripts:/scripts
ports:
- 8080:8080
environment:
AIRFLOW_RUNAS_WEBSERVER: 1
depends_on:
- scheduler
entrypoint: /scripts/wait-for-it.sh scheduler:8793 -t 40 -- /scripts/docker-entrypoint.sh
scheduler:
image: apache/airflow
container_name: airflow_scheduler
restart: on-failure
ports:
- 8793:8793
volumes:
- ./dags:/opt/airflow/dags
- ./plugins:/opt/airflow/plugins
- ./scripts:/scripts
environment:
AIRFLOW_RUNAS_SCHEDULER: 1
entrypoint: /scripts/docker-entrypoint.sh
Where my docker-entrypoint.sh
is like:
#!/bin/bash
AIRFLOW_USERNAME=test
AIRFLOW_PASSWORD=test
if [ "$AIRFLOW_RUNAS_SCHEDULER" = "1" ]; then
echo "initializing airflow database"
airflow db init
echo "running airflow scheduler"
while true
do
airflow scheduler
echo "restarting airflow scheduler"
sleep 1
done
elif [ "$AIRFLOW_RUNAS_WEBSERVER" = "1" ]; then
echo "running airflow webserver"
airflow db upgrade
airflow users create
--username=$AIRFLOW_USERNAME
--password=$AIRFLOW_PASSWORD
--firstname=test
--lastname=test
[email protected]
--role=Admin
exec airflow webserver -p 8080
elif [ "$AIRFLOW_RUNAS_WORKER" = "1" ]; then
echo "running airflow worker"
exec airflow worker
elif [ "$AIRFLOW_RUNAS_FLOWER" = "1" ]; then
echo "running airflow flower"
exec airflow flower
else
echo "ERROR: no AIRFLOW_RUNAS_* variable set"
exit 1
fi
Now, from the logs I can clearly see that running docker-compose up
both airflow webserver
and airflow scheduler
spin up their services correctly even though it seems Airflow webserver is having an hard time finding the scheduler. On the UI I am greeted with a warm:
The scheduler does not appear to be running.
The DAGs list may not update, and new tasks will not be scheduled.
How should I go at finding out why the two services don't talk to each other?
Update:
Strangely enough everything works well if I adapt my docker-entrypoint.sh
to run all the steps within the same service with a trivial
exec airflow webserver -p 8080 & exec airflow scheduler
This makes following complicated as everything gets logged by the same container.
question from:
https://stackoverflow.com/questions/65881570/airflow-webserver-doesnt-find-scheduler-in-docker-compose 与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…