DevOps teams have spent years adding testing, logging, secrets management and auditability to software delivery pipelines. ML pipelines deserve the same treatment. They may not deploy a classical application, but they collect data, create artifacts and influence production decisions.
Figure 1: The Airflow DAG Completes Only After the Final Security Audit Task Succeeds
This article presents a practical SecMLOps experiment based on Apache Airflow. A simple weather ML pipeline was extended with security controls that map naturally to DevSecOps practices: Secret separation, runtime validation, external boundary checks, artifact integrity and audit evidence.
From Workflow Success to Workflow Trust
A green DAG is not enough. A task can succeed while still producing an untrusted artifact. The API key may have been mishandled. Runtime values may have been malformed. The external API may have returned incomplete data. The model may have been saved without metadata. The pipeline may have no evidence except a success state.
Figure 2: SecMLOps Architecture for the Airflow Weather Pipeline
Pipeline Overview
The workflow collects weather data from OpenWeatherMap, stores raw JSON files, generates CSV datasets, trains regression models, selects the best one and saves a model artifact. The SecMLOps version adds a security layer to those operations.
| Control | Purpose |
| Encrypted Airflow Variables | Keep API keys and runtime values outside the source code. |
| Input Validation | Reject invalid city names before making external API calls. |
| HTTP and JSON Checks | Fail fast when the external boundary returns an unexpected result. |
| Empty Dataset Protection | Prevent training on unusable generated CSV files. |
| Model Metadata | Record the selected model, score, features, row count and timestamp. |
| SHA256 Model Hash | Give the model artifact a verifiable identity. |
| Final Security Audit Task | Generate evidence that the expected outputs exist and have passed checks. |
Secrets Belong to the Runtime, not the Repository
The first control is simple: The API key is loaded from Airflow Variables. It is not written in Python source code, and it is not committed to GitHub. In the validated run, Airflow Variables are encrypted with Fernet.
Code 1: Runtime Secret Loading From Airflow Variables
| from airflow.models import Variableapi_key = Variable.get(“api_key”, default_var=None)if api_key is None or len(api_key.strip()) == 0: raise ValueError(“Missing Airflow variable: api_key”) |
Source: Experiment 13 SecMLOps Airflow Weather Pipeline
Figure 3: Airflow Variables Stored With Encryption Enabled
Runtime Validation Before Automation
The city list is also a runtime variable. It may look harmless, but it controls external API calls and file generation. The DAG checks that city values match an expected pattern before using them.
Code 2: Runtime Variable Validation
| CITY_REGEX = re.compile(r”^[A-Za-zÀ-ÿ .’]{1,50}$”)for city in cities: city = city.strip().lower() if not CITY_REGEX.match(city): raise ValueError(f”Invalid city name: {city}”) |
Source: Experiment 13 SecMLOps Airflow Weather Pipeline
The external API boundary is then controlled with a timeout and explicit status checks. A failing API call must stop the task instead of silently creating corrupted inputs for the rest of the pipeline.
Code 3: External API Boundary Control
| response = requests.get( “https://api.openweathermap.org/data/2.5/weather”, params={“q”: city, “appid”: api_key, “units”: “metric”}, timeout=10)if response.status_code != 200: raise RuntimeError(f”API error for {city}: {response.status_code}”) |
Source: Experiment 13 SecMLOps Airflow Weather Pipeline
Figure 4: API Validation Confirming That the Runtime Secret Works
Generated Data Must be Checked
A generated file is not necessarily a useful dataset. The transformation step rejects empty CSV files. This protects scheduled executions from training models on meaningless data after an upstream issue.
Code 4: Empty Dataset Protection
| if df.empty: raise ValueError(f”{filename} is empty”)return output_path |
Source: Experiment 13 SecMLOps Airflow Weather Pipeline
Artifact Integrity and Model Metadata
The selected model is hashed with SHA-256 and described in a metadata report. The metadata records the model name, score, features, training row count and timestamp. This is not a full model registry, but it gives the artifact a verifiable identity.
Code 5: SHA-256 Model Artifact Hashing
| def get_file_sha256(path): sha256 = hashlib.sha256() with open(path, “rb”) as file: for block in iter(lambda: file.read(4096), b””): sha256.update(block) return sha256.hexdigest() |
Source: Experiment 13 SecMLOps Airflow Weather Pipeline
Figure 5: Generated Model Metadata and Security Audit Evidence
Audit as a Pipeline Task
The final security audit is not an external checklist. It is part of the DAG. The pipeline only reaches its final success state after the audit task verifies the expected files and produces a security_audit.json report.
Code 6: Final Audit Task
| @taskdef security_audit(model_path): checks = { “data_csv”: check_file_exists_and_not_empty(DATA_PATH), “fulldata_csv”: check_file_exists_and_not_empty(FULLDATA_PATH), “model”: check_file_exists_and_not_empty(model_path), “model_metadata”: check_file_exists_and_not_empty(MODEL_METADATA_PATH), “model_sha256”: get_file_sha256(model_path), “status”: “passed” } write_json_report(SECURITY_AUDIT_PATH, checks) return SECURITY_AUDIT_PATH |
Source: Experiment 13 SecMLOps Airflow Weather Pipeline
Figure 6: Security Audit Task Logs Inside Airflow
Figure 7: Airflow Cluster Activity Used as Operational Evidence
Operational Lesson for DevOps Teams
The lesson is straightforward: MLOps pipelines should inherit DevSecOps discipline. Secrets should be externalized. Inputs should be validated. External services should be treated as trust boundaries. Artifacts should be identifiable. Successful runs should leave evidence.
This is especially important because ML pipelines often evolve from notebooks and experiments into scheduled automation. Security controls should be added before the pipeline becomes part of production decision-making.
Conclusion
A secure ML pipeline is not only a pipeline that produces a model. It is a pipeline that can explain what it did, what it used and what it generated. In the Airflow experiment, that explanation is implemented through runtime validation, artifact hashing, metadata and a final audit report.
Reproduce the Experiment
The full experiment is available in the public AI Security repository. The repository contains the Airflow DAG, documentation, screenshots, sample data and the security notes used to reproduce the workflow.
