How to Set Up Persistent Logging for Apache Airflow on GKE using the GCS Fuse CSI Driver

When running Airflow on GKE, worker pod logs are lost when the pods are deleted. The standard log persistence feature in the Helm chart doesn't work out-of-the-box because it requires a shared volume (ReadWriteMany), which GKE's default storage doesn't provide. This article shows how to solve this by using the GCS Fuse CSI Driver to mount a Google Cloud Storage bucket as the shared volume for logs.

If you use the official Apache Airflow Helm chart to deploy and manage Airflow in Google Kubernetes Engine (GKE) with the Kubernetes Executor, you’ll notice that the execution logs disappear when the worker pods are deleted. One way to solve this is to enable log persistence in the values.yaml file:

# Example section for the values.yaml file
logs:
    persistence:
        enabled: true
        size: 1Gi

However, this configuration requires a PersistentVolumeClaim (PVC) with ReadWriteMany capabilities. In GKE, the standard provided StorageClass doesn’t support this, which will cause an error. You need to provide a storage solution that can be mounted by multiple pods simultaneously.

Using the Google Cloud Storage FUSE CSI Driver for GKE

One of the best options for providing the shared PVC required by multiple Airflow pods is the GCS Fuse CSI driver. This functionality is natively supported by GKE. You can enable this feature in the GKE console by navigating to Cluster details > Features and toggling the Cloud Storage FUSE CSI Driver.

gcloud container clusters update <CLUSTER_NAME> \
--update-addons GcsFuseCsiDriver=ENABLED \
--location=<LOCATION>

Next, you need to configure the Google Cloud service account that the pods will use.

You can do this by creating a new service account or updating an existing one, adding the Storage Object Admin role, and then adding your Kubernetes service accounts as principals with the Workload Identity User role.

The official Helm chart creates multiple Kubernetes service accounts that you need to grant permission to impersonate the Google Cloud service account.

The principal entries should look like this:

project-name.svc.id.goog[airflow/airflow-webserver]
project-name.svc.id.goog[airflow/airflow-scheduler]
project-name.svc.id.goog[airflow/airflow-triggerer]
project-name.svc.id.goog[airflow/airflow-worker]

Next, add the necessary annotations to the Kubernetes Service Accounts by editing the Helm values.yaml file.

webserver:
    podAnnotations:
        gke-gcsfuse/volumes: "true"
    serviceAccount:
        annotations:
            iam.gke.io/gcp-service-account: <service-account-name>@<project-name>.iam.gserviceaccount.com


scheduler:
    podAnnotations:
        gke-gcsfuse/volumes: "true"
    serviceAccount:
        annotations:
            iam.gke.io/gcp-service-account: <service-account-name>@<project-name>.iam.gserviceaccount.com


workers:
    podAnnotations:
        gke-gcsfuse/volumes: "true"
    serviceAccount:
        annotations:
            iam.gke.io/gcp-service-account: <service-account-name>@<project-name>.iam.gserviceaccount.com


triggerer:
    podAnnotations:
        gke-gcsfuse/volumes: "true"
    serviceAccount:
        annotations:
            iam.gke.io/gcp-service-account: <service-account-name>@<project-name>.iam.gserviceaccount.com

The podAnnotations will tell Google to inject the GCS Fuse sidecar container into your pods, and the serviceAccount annotations will allow the pods to authenticate with and access your Cloud Storage buckets.

Now, you can create the PersistentVolume (PV) and PersistentVolumeClaim (PVC) that will be used for logging:

apiVersion: v1
kind: PersistentVolume
metadata:
    name: airflow-logs-storage
spec:
    accessModes:
        - ReadWriteMany
    capacity:
        storage: 1Gi
    storageClassName: ""
    persistentVolumeReclaimPolicy: Retain
    claimRef:
        namespace: airflow
        name: airflow-logs-storage
    csi:
        driver: gcsfuse.csi.storage.gke.io
        volumeHandle: your-gcs-bucket-name # Replace with your GCS bucket name
        volumeAttributes:
            bucketName: your-gcs-bucket-name # Replace with your GCS bucket name
            mountOptions: "implicit-dirs"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
    name: airflow-logs-storage
    namespace: airflow
spec:
    accessModes:
        - ReadWriteMany
    resources:
        requests:
            storage: 1Gi
    storageClassName: ""
    volumeName: airflow-logs-storage

Finally, edit the logs section in your values.yaml file to use the newly created PVC:

# --- LOGS Storage ---
logs:
    persistence:
        enabled: true
        existingClaim: airflow-logs-storage # The name of the PVC you created


# If you are using gitSync for your DAGs, your dags section might look like this:
dags:
    persistence:
        enabled: false # This should be false if you use gitSync
    gitSync:
        enabled: true
        repo: <your-git-repo>
        branch: master
        ref: HEAD
        subPath: "dags/" # Optional: if your DAGs are in a subfolder
        credentialsSecret: <your-secret-name>

And that’s how you configure persistent log storage for Apache Airflow in GKE using the GCS Fuse CSI driver.

References