DevOps teams have spent years adding testing, logging, secrets management and auditability to software delivery pipelines. ML pipelines deserve the same treatment. They may not deploy a classical application, but they collect data, create artifacts and influence production decisions.

Figure 1: The Airflow DAG Completes Only After the Final Security Audit Task Succeeds

This article presents a practical SecMLOps experiment based on Apache Airflow. A simple weather ML pipeline was extended with security controls that map naturally to DevSecOps practices: Secret separation, runtime validation, external boundary checks, artifact integrity and audit evidence.

From Workflow Success to Workflow Trust

A green DAG is not enough. A task can succeed while still producing an untrusted artifact. The API key may have been mishandled. Runtime values may have been malformed. The external API may have returned incomplete data. The model may have been saved without metadata. The pipeline may have no evidence except a success state.

Figure 2: SecMLOps Architecture for the Airflow Weather Pipeline

Pipeline Overview

The workflow collects weather data from OpenWeatherMap, stores raw JSON files, generates CSV datasets, trains regression models, selects the best one and saves a model artifact. The SecMLOps version adds a security layer to those operations.

Control	Purpose
Encrypted Airflow Variables	Keep API keys and runtime values outside the source code.
Input Validation	Reject invalid city names before making external API calls.
HTTP and JSON Checks	Fail fast when the external boundary returns an unexpected result.
Empty Dataset Protection	Prevent training on unusable generated CSV files.
Model Metadata	Record the selected model, score, features, row count and timestamp.
SHA256 Model Hash	Give the model artifact a verifiable identity.
Final Security Audit Task	Generate evidence that the expected outputs exist and have passed checks.

Secrets Belong to the Runtime, not the Repository

The first control is simple: The API key is loaded from Airflow Variables. It is not written in Python source code, and it is not committed to GitHub. In the validated run, Airflow Variables are encrypted with Fernet.

Code 1: Runtime Secret Loading From Airflow Variables

from airflow.models import Variableapi_key = Variable.get(“api_key”, default_var=None)if api_key is None or len(api_key.strip()) == 0: raise ValueError(“Missing Airflow variable: api_key”)

Source: Experiment 13 SecMLOps Airflow Weather Pipeline

Figure 3: Airflow Variables Stored With Encryption Enabled

Runtime Validation Before Automation

The city list is also a runtime variable. It may look harmless, but it controls external API calls and file generation. The DAG checks that city values match an expected pattern before using them.

Code 2: Runtime Variable Validation

CITY_REGEX = re.compile(r”^[A-Za-zÀ-ÿ .’]{1,50}$”)for city in cities: city = city.strip().lower() if not CITY_REGEX.match(city): raise ValueError(f”Invalid city name: {city}”)

Source: Experiment 13 SecMLOps Airflow Weather Pipeline

The external API boundary is then controlled with a timeout and explicit status checks. A failing API call must stop the task instead of silently creating corrupted inputs for the rest of the pipeline.

Code 3: External API Boundary Control

response = requests.get( “https://api.openweathermap.org/data/2.5/weather”, params={“q”: city, “appid”: api_key, “units”: “metric”}, timeout=10)if response.status_code != 200: raise RuntimeError(f”API error for {city}: {response.status_code}”)

Source: Experiment 13 SecMLOps Airflow Weather Pipeline

Figure 4: API Validation Confirming That the Runtime Secret Works

Generated Data Must be Checked

A generated file is not necessarily a useful dataset. The transformation step rejects empty CSV files. This protects scheduled executions from training models on meaningless data after an upstream issue.

Code 4: Empty Dataset Protection

if df.empty: raise ValueError(f”{filename} is empty”)return output_path

Source: Experiment 13 SecMLOps Airflow Weather Pipeline

Artifact Integrity and Model Metadata

The selected model is hashed with SHA-256 and described in a metadata report. The metadata records the model name, score, features, training row count and timestamp. This is not a full model registry, but it gives the artifact a verifiable identity.

Code 5: SHA-256 Model Artifact Hashing

def get_file_sha256(path): sha256 = hashlib.sha256() with open(path, “rb”) as file: for block in iter(lambda: file.read(4096), b””): sha256.update(block) return sha256.hexdigest()

Source: Experiment 13 SecMLOps Airflow Weather Pipeline

Figure 5: Generated Model Metadata and Security Audit Evidence

Audit as a Pipeline Task

The final security audit is not an external checklist. It is part of the DAG. The pipeline only reaches its final success state after the audit task verifies the expected files and produces a security_audit.json report.

Code 6: Final Audit Task

@taskdef security_audit(model_path): checks = { “data_csv”: check_file_exists_and_not_empty(DATA_PATH), “fulldata_csv”: check_file_exists_and_not_empty(FULLDATA_PATH), “model”: check_file_exists_and_not_empty(model_path), “model_metadata”: check_file_exists_and_not_empty(MODEL_METADATA_PATH), “model_sha256”: get_file_sha256(model_path), “status”: “passed” } write_json_report(SECURITY_AUDIT_PATH, checks) return SECURITY_AUDIT_PATH

Source: Experiment 13 SecMLOps Airflow Weather Pipeline

Figure 6: Security Audit Task Logs Inside Airflow

Figure 7: Airflow Cluster Activity Used as Operational Evidence

Operational Lesson for DevOps Teams

The lesson is straightforward: MLOps pipelines should inherit DevSecOps discipline. Secrets should be externalized. Inputs should be validated. External services should be treated as trust boundaries. Artifacts should be identifiable. Successful runs should leave evidence.

This is especially important because ML pipelines often evolve from notebooks and experiments into scheduled automation. Security controls should be added before the pipeline becomes part of production decision-making.

Conclusion

A secure ML pipeline is not only a pipeline that produces a model. It is a pipeline that can explain what it did, what it used and what it generated. In the Airflow experiment, that explanation is implemented through runtime validation, artifact hashing, metadata and a final audit report.

Reproduce the Experiment

The full experiment is available in the public AI Security repository. The repository contains the Airflow DAG, documentation, screenshots, sample data and the security notes used to reproduce the workflow.