Removing Airflow task logs

Question

I'm running 5 DAG's which have generated a total of about 6GB of log data in the base_log_folder over a months period. I just added a remote_base_log_folder but it seems it does not exclude logging to the base_log_folder.

Is there anyway to automatically remove old log files, rotate them or force airflow to not log on disk (base_log_folder) only in remote storage?

question from:https://stackoverflow.com/questions/43548744/removing-airflow-task-logs

Answer 1

Please refer https://github.com/teamclairvoyant/airflow-maintenance-dags

This plugin has DAGs that can kill halted tasks and log-cleanups. You can grab the concepts and can come up with a new DAG that can cleanup as per your requirement.