Tag: pipeline security

GitHub Separates Who Writes Code From Who Runs Your CI

Anyone with write access to a repository can trigger a GitHub Actions workflow. That has been the default since Actions launched. It’s convenient. It’s also a problem when a compromised account or a malicious pull request can turn your CI into an attack path.

Attackers have noticed. GitHub’s own 2026 Actions security roadmap points to incidents involving tj-actions/changed-files, Nx, and trivy-action as examples of the same pattern. Go after the automation, not the application. Steal a token, and you own the pipeline.

Mitch Ashley, vice president and practice lead for CIO & technology buyers and software lifecycle engineering at The Futurum Group, puts the stakes plainly. “A single compromised developer account can mean the difference between a well-running pipeline and compromised software.”

GitHub’s latest answer is now generally available. On September 17, the company made workflow execution protections GA for GitHub Enterprise, organizations, and repositories. The feature first appeared in public preview on June 18.

The idea is simple. Administrators build an allowlist. GitHub checks the rules before a workflow run begins. If the actor or the event isn’t on the list, the run doesn’t start.

There are two kinds of rules. Actor rules control who can trigger workflows. These can be individual users, repository roles such as Read, Maintain, and Admin, GitHub Apps, Copilot, and Dependabot. Event rules control which events are allowed, including push, pull_request, pull_request_target, and workflow_dispatch.

GitHub’s documentation explains the point of actor rules this way: “Actor rules let you separate who contributes code from who runs your CI, so you can grant a contributor write access without granting them the ability to execute workflows.”

Write access and execution rights used to arrive as a bundle. Three features in the GA release weren’t in the preview.

First, rules can now target specific workflow files instead of a whole repository. A team can limit deploy.yml to a named group while leaving CI workflows open to every contributor.

Second, there are new insights. They show how rules are evaluated and enforced at the enterprise, organization, and repository level. Admins can use them to tune a policy before and after it goes live.

Third, there’s a REST API. It supports create, read, update, and delete at all three levels, including workflow path conditions. Teams that manage governance as code can keep these policies in the same review process.

Evaluate mode carries over from the preview. It lets admins “run your rules in shadow, so you can see exactly what a rule would block before you enforce it,” according to the June announcement. GitHub’s docs list evaluate mode as an Enterprise Cloud capability. Rules live under Settings, then Actions, then Policies. They build on GitHub’s rulesets, so enterprise-wide policies and custom-property scoping work as they do elsewhere. The protections cover all public repositories, plus private repositories on GitHub Team or GitHub Enterprise.

One caution from the docs: bots such as dependabot[bot] must be added as allowed actors if your workflows depend on them.

Ashley sees the value in the simplicity. “Often the simplest solutions can be the most effective. Tighter controls over how CI or deployment to test or production is executed greatly reduce the attack surface. With these improvements, dev teams will no longer have to rely upon bespoke built solutions to secure this aspect of their development pipelines.”

The part with a deadline is the new default rule for pull_request_target.

This trigger runs with the base repository’s token and secrets. If a workflow checks out code from an untrusted fork, the attacker’s code can run with those privileges. GitHub already tightened actions/checkout in June so that version 7 refuses to fetch fork pull request code in these workflows unless a developer sets allow-unsafe-pr-checkout.

A default rule disables pull_request_target in public repositories that don’t already have an event policy. It’s running in evaluate mode today. Enforcement starts November 2, 2026. It doesn’t apply to private or internal repositories.

That’s about six weeks. Here’s how to use them.

Search your .github/workflows directories for pull_request_target. Then check the policy insights page to see which runs the default would have blocked. Some workflows only post a comment using trusted code from the base branch. Those are low risk. Others check out pull request code while holding secrets. Fix those first.

For workflows that truly need the trigger, you have two options. Add an event policy that allows pull_request_target only for named workflow files. Or restructure. Run the untrusted analysis in a read-only pull_request job, save the results as an artifact, and post comments from a separate workflow_run job that has write access. The second path takes more work, but it removes the risk.

One analysis from Digital Applied flags AI code-review bots and labelers as the likeliest to be caught. They read pull request text from strangers while holding write tokens.

Policy-driven execution is one item on GitHub’s Actions security roadmap. Workflow dependency locking, scoped secrets, and a native egress firewall are still coming. This is the first piece most teams can use today.

The larger point is about how teams think of CI. Pipelines hold deployment credentials, signing keys, and cloud access. They deserve the same scrutiny as production.

Start in evaluate mode. Read the insights. Then turn rules on for the workflows that matter most.

September 21, 2026
Why Your CI/CD Pipeline Is Your Most Unprotected Attack Surface

Your team probably spends considerable effort securing your application. WAFs, rate limiting, input validation, penetration testing. Then code passes through a CI/CD pipeline that runs with privileged credentials, executes arbitrary code from third-party actions, has broad network access to internal systems, and logs secrets in plain text if someone makes one configuration mistake.

The pipeline that builds and deploys your secure application is itself the least secure system in your infrastructure.

Think about what a CI/CD pipeline actually is from an attacker’s perspective. It is a system that automatically executes code, often with administrative access to your cloud accounts, container registries, databases, and production environments. It pulls dependencies from the public internet. It runs plugins and actions written by strangers. It often has access to every secret needed to deploy to every environment. If you designed a system specifically to be an attractive target for supply chain attacks, you would design a CI/CD pipeline.

The XZ Utils backdoor was a masterclass in exploiting this exact attack surface. A patient attacker spent years building trust in an open-source project, then introduced a backdoor that would have propagated through build systems worldwide. That attack was caught by accident. The next one might not be. And your pipeline is running builds right now that pull in hundreds of dependencies, each of which is a potential vector.

Here is what a hardened pipeline actually looks like. First, pipeline credentials should be scoped to the minimum required for each stage, not a single service account that can do everything. Second, third-party actions and plugins should be pinned to specific commit SHAs, not tags that can be silently updated. Third, pipeline execution environments should be ephemeral and isolated, with no network access beyond what is explicitly required for that build step. Fourth, every pipeline run should produce a signed provenance attestation that can be verified independently.

Most organizations are nowhere near this. They are running Jenkins instances that have not been patched in months, GitHub Actions workflows that use community actions pinned to “latest,” and build agents with permanent credentials stored in environment variables. The gap between how seriously we take application security and how casually we treat pipeline security is the biggest unaddressed risk in modern DevOps. And it will take one high-profile breach through a CI/CD vector to make everyone suddenly care.

September 18, 2026

Why MLOps Pipelines Need Security Audits

DevOps teams have spent years adding testing, logging, secrets management and auditability to software delivery pipelines. ML pipelines deserve the same treatment. They may not deploy a classical application, but they collect data, create artifacts and influence production decisions.

Figure 1: The Airflow DAG Completes Only After the Final Security Audit Task Succeeds

This article presents a practical SecMLOps experiment based on Apache Airflow. A simple weather ML pipeline was extended with security controls that map naturally to DevSecOps practices: Secret separation, runtime validation, external boundary checks, artifact integrity and audit evidence.

From Workflow Success to Workflow Trust

A green DAG is not enough. A task can succeed while still producing an untrusted artifact. The API key may have been mishandled. Runtime values may have been malformed. The external API may have returned incomplete data. The model may have been saved without metadata. The pipeline may have no evidence except a success state.

Figure 2: SecMLOps Architecture for the Airflow Weather Pipeline

Pipeline Overview

The workflow collects weather data from OpenWeatherMap, stores raw JSON files, generates CSV datasets, trains regression models, selects the best one and saves a model artifact. The SecMLOps version adds a security layer to those operations.

Control	Purpose
Encrypted Airflow Variables	Keep API keys and runtime values outside the source code.
Input Validation	Reject invalid city names before making external API calls.
HTTP and JSON Checks	Fail fast when the external boundary returns an unexpected result.
Empty Dataset Protection	Prevent training on unusable generated CSV files.
Model Metadata	Record the selected model, score, features, row count and timestamp.
SHA256 Model Hash	Give the model artifact a verifiable identity.
Final Security Audit Task	Generate evidence that the expected outputs exist and have passed checks.

Secrets Belong to the Runtime, not the Repository

The first control is simple: The API key is loaded from Airflow Variables. It is not written in Python source code, and it is not committed to GitHub. In the validated run, Airflow Variables are encrypted with Fernet.

Code 1: Runtime Secret Loading From Airflow Variables

from airflow.models import Variableapi_key = Variable.get(“api_key”, default_var=None)if api_key is None or len(api_key.strip()) == 0: raise ValueError(“Missing Airflow variable: api_key”)

Source: Experiment 13 SecMLOps Airflow Weather Pipeline

Figure 3: Airflow Variables Stored With Encryption Enabled

Runtime Validation Before Automation

The city list is also a runtime variable. It may look harmless, but it controls external API calls and file generation. The DAG checks that city values match an expected pattern before using them.

Code 2: Runtime Variable Validation

CITY_REGEX = re.compile(r”^[A-Za-zÀ-ÿ .’]{1,50}$”)for city in cities: city = city.strip().lower() if not CITY_REGEX.match(city): raise ValueError(f”Invalid city name: {city}”)

Source: Experiment 13 SecMLOps Airflow Weather Pipeline

The external API boundary is then controlled with a timeout and explicit status checks. A failing API call must stop the task instead of silently creating corrupted inputs for the rest of the pipeline.

Code 3: External API Boundary Control

response = requests.get( “https://api.openweathermap.org/data/2.5/weather”, params={“q”: city, “appid”: api_key, “units”: “metric”}, timeout=10)if response.status_code != 200: raise RuntimeError(f”API error for {city}: {response.status_code}”)

Source: Experiment 13 SecMLOps Airflow Weather Pipeline

Figure 4: API Validation Confirming That the Runtime Secret Works

Generated Data Must be Checked

A generated file is not necessarily a useful dataset. The transformation step rejects empty CSV files. This protects scheduled executions from training models on meaningless data after an upstream issue.

Code 4: Empty Dataset Protection

if df.empty: raise ValueError(f”{filename} is empty”)return output_path

Source: Experiment 13 SecMLOps Airflow Weather Pipeline

Artifact Integrity and Model Metadata

The selected model is hashed with SHA-256 and described in a metadata report. The metadata records the model name, score, features, training row count and timestamp. This is not a full model registry, but it gives the artifact a verifiable identity.

Code 5: SHA-256 Model Artifact Hashing

def get_file_sha256(path): sha256 = hashlib.sha256() with open(path, “rb”) as file: for block in iter(lambda: file.read(4096), b””): sha256.update(block) return sha256.hexdigest()

Source: Experiment 13 SecMLOps Airflow Weather Pipeline

Figure 5: Generated Model Metadata and Security Audit Evidence

Audit as a Pipeline Task

The final security audit is not an external checklist. It is part of the DAG. The pipeline only reaches its final success state after the audit task verifies the expected files and produces a security_audit.json report.

Code 6: Final Audit Task

@taskdef security_audit(model_path): checks = { “data_csv”: check_file_exists_and_not_empty(DATA_PATH), “fulldata_csv”: check_file_exists_and_not_empty(FULLDATA_PATH), “model”: check_file_exists_and_not_empty(model_path), “model_metadata”: check_file_exists_and_not_empty(MODEL_METADATA_PATH), “model_sha256”: get_file_sha256(model_path), “status”: “passed” } write_json_report(SECURITY_AUDIT_PATH, checks) return SECURITY_AUDIT_PATH

Source: Experiment 13 SecMLOps Airflow Weather Pipeline

Figure 6: Security Audit Task Logs Inside Airflow

Figure 7: Airflow Cluster Activity Used as Operational Evidence

Operational Lesson for DevOps Teams

The lesson is straightforward: MLOps pipelines should inherit DevSecOps discipline. Secrets should be externalized. Inputs should be validated. External services should be treated as trust boundaries. Artifacts should be identifiable. Successful runs should leave evidence.

This is especially important because ML pipelines often evolve from notebooks and experiments into scheduled automation. Security controls should be added before the pipeline becomes part of production decision-making.

Conclusion

A secure ML pipeline is not only a pipeline that produces a model. It is a pipeline that can explain what it did, what it used and what it generated. In the Airflow experiment, that explanation is implemented through runtime validation, artifact hashing, metadata and a final audit report.

Reproduce the Experiment

The full experiment is available in the public AI Security repository. The repository contains the Airflow DAG, documentation, screenshots, sample data and the security notes used to reproduce the workflow.

September 16, 2026

The Silent Risk of AI-Written DevOps Pipelines

These days, when a developer needs a CI/CD pipeline, they don’t always dive into GitHub Actions docs or spin up Jenkins from scratch. Instead, they pull up an AI assistant and type out something like:

“Create a deployment pipeline for a containerized application.”

Seconds later, the AI spits out a complete workflow. It looks polished. It builds, tests, packages, and deploys, clean syntax, logical steps, just what you’d expect. The developer copies the code, tweaks a few bits, throws it in the repo.

The pipeline works.

The deployment goes through.

The team checks it off and moves along.

This is just standard practice now. More and more, teams are using AI to pump out infrastructure code, deployment workflows, Kubernetes configs, and all sorts of automation scripts. Stuff that used to take specialized know-how gets built almost instantly.

Productivity shoots up. That’s obvious.

But the risks aren’t so easy to spot.

And that’s what makes AI-written DevOps pipelines dangerous. They look right, they even work as intended. But sometimes, they hide big problems that don’t come out until much later.

Why Are AI Generated Pipelines So Popular?

Part of it is the pressure DevOps teams face. Speed matters, a lot. Every new project needs a workflow or deployment setup. Infrastructure needs scripts and automation right away. The push for shorter delivery cycles, fewer manual steps, and more reliability isn’t going anywhere.

So, AI assistants fit the bill. Instead of losing time sifting through documentation, developers just describe what they need. The AI gives them something that works immediately.

For small teams, it feels like hitting the jackpot.

You end up spending less time wrestling with configs and more time building features. Those repetitive deployment patterns? They’re just a prompt away now. Suddenly, you’re delivering faster than ever.

The kicker isn’t that AI writes these pipelines. The problem is people tend to trust what they get, even if they don’t really understand it.

Just Because It Works Doesn’t Mean It’s Right

Here’s where things get tricky. Most of the time, AI produced pipelines seem perfect.

Build passes.

Tests succeed.

Deployment finishes without a hitch.

Everything looks fine at a glance.

But pipelines do more than move code. They manage secrets, access production, spin up infrastructure, touch cloud resources, serious stuff.

A pipeline can do its job and still leak secrets, have permissions set way too wide, or bake in security holes.

Most of these issues don’t stop things from working. They usually just lurk unnoticed, sometimes for months, until something breaks or someone finds them.

The Permissions Trap

This comes up a lot. AI generated workflows often grant broad permissions. It’s safer, from the AI’s perspective, to give the workflow access to everything it might need, just to make sure nothing fails.

From a user’s angle, success means no errors. From a security angle, success means only granting what’s absolutely necessary. Those aren’t the same thing.

Now, you’ve got pipelines that can do way too much. Maybe they can write to areas of your repo they shouldn’t, or reach into sensitive environments no script should ever touch.

Nothing seems wrong until those permissions get abused or leaked.

And most of the time, nobody even realizes it’s a risk.

Hidden Security Gaps

AI just predicts code patterns. It doesn’t know your organization’s security standards, compliance rules, or governance policies. It doesn’t actually understand what safe means to you.

That means it’ll happily generate pipelines with questionable practices that look harmless on the surface.

Secrets might get handled sloppily.

You might end up pulling in third-party actions without checking them out.

Dependencies come from external sources, maybe unverified.

Deployment steps run commands nobody’s really reviewed.

None of these are guaranteed to spark a breach. But they definitely raise the odds.

Supply Chain Problems

Today’s delivery pipelines are built on tons of external stuff, GitHub Actions, container images, plugins, scripts, you name it. And because AI is trained on all sorts of public code, it tends to pull in whatever’s commonly used out in the wild.

That’s what makes the workflow work, familiar ingredients.

But the AI has no clue if you actually trust those components in your environment.

Something that looks safe now gets compromised later. Some dependency gets a silent update that changes its behavior. Or maybe a container image carries vulnerabilities only a deep review would catch.

Teams that copy and paste AI output without checking every piece inherit whatever risks those pieces carry without knowing it.

Speed Kills Review

Building a pipeline used to take effort. Engineers read docs, figured out integrations, and picked every step intentionally. That process forced you to review everything as you went.

AI blew that up.

Now you can generate pipelines much faster than you can realistically review them.

So, teams start to focus on “does it work?” and forget about “is it safe?”

The quicker these pipelines get created, the easier it is to skip a real review.

Fast tools are great until they become a shortcut past critical thinking.

Blind Trust Creeps In

As AI gets better, people start trusting its output. It’s human nature.

But here’s the downside. When the AI nails it, teams stop double checking what it gave them. There are pipelines running right now that went live six months ago, and nobody knows exactly what they do. New hires inherit them, assuming someone else reviewed everything. The people who set them up assume the previous team made careful choices. Over time, trust quietly replaces inspection.

And risks pile up in the background.

So, How Should Teams Use AI?

The answer isn’t to ditch AI. Far from it. AI assistants save time, kill boring work, and help you move faster.

But you have to treat generated pipelines as just a starting point, not a finished product.

Every AI generated workflow deserves real scrutiny, the same way you review core app code.

Check the permissions.

Validate how secrets get handled.

Review all third party pieces.

Don’t let anything run in production you don’t fully understand.

It’s not about slowing down innovation. It’s about not letting automation erase your awareness.

Looking Ahead

AI will keep getting better at DevOps work. Systems will generate smarter workflows, design deployment strategies, optimize infrastructure, and maybe even decide when to automate crucial operations.

That’s exciting stuff. But it makes governance and review even more critical.

The winners will be the teams who automate fearlessly but never check their brains at the door. Automation is only as good as your willingness to keep paying attention.

Final Thoughts

AI generated pipelines aren’t some distant future, they’re already part of everyday work for a lot of engineers. They’re fast, simple, and they boost productivity.

But just because a pipeline runs smoothly doesn’t mean it’s secure.

The real danger isn’t that AI makes broken workflows. It’s that it makes seemingly perfect workflows that quietly carry hidden risks.

Automation has always been DevOps’ superpower, but it only works when you still know what your systems are actually doing behind the curtain.

AI shakes up how teams build, but it doesn’t get you off the hook for visibility, accountability, or good judgment.

You can generate a pipeline in seconds. The fallout from a bad one could stick with you for years.

June 4, 2026
Worms in the Supply Chain: Shai-Hulud and the Next DevOps Reckoning
DevOps was supposed to make software delivery faster, safer and more reliable. For the most part, it has. But every so often, something nasty crawls out of the shadows and reminds us how fragile the system really is.

It wasn’t a zero-day in Kubernetes or a cloud misconfiguration that caught my eye. It was a worm. Not a metaphorical one, not a cute “bug.” A self-replicating, credential-stealing worm slithering through npm, GitHub, and CI/CD pipelines. Its name: Shai-Hulud, after the sandworms of Dune. And like those great worms of Arrakis, this one tunneled through the ground beneath our feet — our supply chain.

How Shai-Hulud Struck

Security researchers at Wiz, Zscaler and StepSecurity described how the worm infected 200+ npm packages and over 500 versions in just a few days. Some of those packages had millions of weekly downloads.

Once installed, the malware pulled in TruffleHog to scan for secrets in environment variables and config files. It stole GitHub tokens, dumped them into a public repo called Shai-Hulud, and then planted malicious GitHub Actions workflows for persistence. In some cases, it even migrated private repos to public forks — broadcasting secrets to the world.

Charlie Eriksen from Aikido summed it up: “At least 187 code packages … have been infected with a self-replicating worm that steals credentials from developers and publishes those secrets on GitHub.” That’s not a nuisance. That’s a supply chain wildfire.

Why This is a DevOps Problem

You could argue this is just another npm mess. “Don’t use random packages,” “lock dependencies,” “scan for secrets.” We’ve heard it all before. But if you’re in DevOps, you can’t shrug this one off.

Why? Because Shai-Hulud didn’t just target developers at their laptops. It went straight after the DevOps machinery: the pipelines, the tokens, the workflows. That’s our house. That’s our responsibility.

Think about what DevOps has become. CI/CD pipelines aren’t side projects — they’re the factories. Every commit goes in, every release comes out. If attackers compromise that factory, they don’t just own one app, they own everything that factory produces.

This worm was tailor-made for that reality.

Lessons for the DevOps Community

So what do we do about it? A few takeaways hit me square in the face:
1. Long-lived tokens are suicide notes. If your pipelines still rely on credentials that never expire, you’re inviting attackers to take up permanent residence. Shift to OIDC or short-lived tokens.
2. Pipelines must be locked down. Too many orgs let builds run with full internet access, install scripts from who-knows-where, and treat CI environments as disposable. They’re not disposable—they’re crown jewels. Lock them down.
3. Provenance is not optional. If you can’t prove where an artifact came from, you shouldn’t be shipping it. Signed builds, SBOMs, attestations — call it bureaucracy if you want, but it’s the only way to stop worms from tunneling in.
4. Golden paths need teeth. DevOps teams love paved roads, but paved roads without guardrails are just fast lanes to compromise. Harden your templates. Don’t make security an optional checkbox.
5. Continuous verification beats once-a-year audits. Secret scanning, anomaly detection, policy enforcement — these need to be continuous. A worm doesn’t wait for your quarterly review.
A Pattern We’ve Seen Before

If all this feels eerily familiar, it should. Every era of computing has had its worm moment. Networks had SQL Slammer. Email had ILOVEYOU. Mobile had Stagefright. And now, the DevOps era has Shai-Hulud.

The pattern is always the same: systems grow fast, guardrails lag, and attackers exploit the gap. Worms are the blunt instruments that expose just how thin our defenses really are.

Shimmy’s Take

For me, Shai-Hulud is more than a clever name in the npm registry. It’s a warning shot across the bow of DevOps. We’ve built incredible factories of automation, but too many of them are made of cardboard and duct tape.

The bad news? Worms thrive in that environment. The good news? We know what to do. Kill the zombie tokens. Lock down the build rooms. Demand provenance. Make golden paths truly golden.

DevOps was always supposed to be about culture and discipline, not just tooling. Shai-Hulud is reminding us of that in the harshest way possible. If we treat pipelines as disposable plumbing, they’ll fail us. If we treat them as infrastructure, they can withstand worms.

The spice — the code — must flow. But it’s up to us in DevOps to make sure it flows safely, not straight into the desert for the worms to feast on.
October 1, 2025
Shift Left With DAST: Dynamic Testing in the CI/CD Pipeline

Dynamic application security testing, or DAST, is a process used to evaluate an application’s security while it’s running rather than while it’s in a static state. This method involves simulating attacks on an application, identifying vulnerabilities and analyzing the responses. DAST focuses on the exposed HTTP and HTML interfaces of web-enabled applications, making it an excellent tool for spotting security vulnerabilities that may occur during an application’s active runtime.

DAST is a ‘black-box’ testing method, meaning it tests the application from the outside in—much like a potential attacker might. It doesn’t have access to the source code and instead focuses on the application as a whole rather than its individual components. This external perspective enables DAST to discover potential security threats that static testing methods might miss.

While DAST is an effective security testing tool, it becomes even more potent when paired with a shift left approach.

Understanding the Concept of ‘Shift Left’ in Software Testing

Shift left is a practice in software development where testing is conducted earlier in the life cycle—that is, it’s ‘shifted left’ on the project timeline. Traditionally, testing was seen as the last phase before the release of a software product. However, with the shift left approach, testing becomes an integral part of every stage of the development process from planning to deployment.

This approach has several advantages. It allows for the identification and resolution of issues earlier in the development process, reducing the cost and effort required to fix them later. It also encourages collaboration between developers and testers, fostering a more cohesive, efficient development environment.

Shift left is not just a testing strategy but a philosophy that promotes preventive measures over reactive solutions. It’s about building quality and security into software from the get-go, rather than tacking it on at the end.

DAST in the Context of Shift Left

Importance of DAST in Early Stages of Development

The integration of DAST in the early stages of development is crucial for several reasons. First, by conducting dynamic security testing from the onset, teams can identify vulnerabilities earlier, making them easier and less costly to fix. This proactive approach helps to prevent security issues from becoming ingrained in the code, which can lead to significant problems down the line.

Second, early integration of DAST encourages a security-focused mindset from the beginning of the project, promoting a culture of security within the team. This cultural shift is crucial in today’s cybersecurity climate, where threats are increasingly sophisticated, and the stakes are higher than ever.

How DAST Complements Other Testing Methods

DAST doesn’t replace other testing methods; rather, it complements them. While DAST focuses on external vulnerabilities, other testing methods like static application security testing (SAST) can target internal issues. By combining these methods, teams can achieve a more comprehensive view of their application’s security.

In a shift left approach, this combination of testing methods can be very powerful. By conducting these tests early and often, teams can ensure that both the external and internal aspects of their application are secure. This layered approach to security testing can help to catch any vulnerabilities that might otherwise slip through the cracks.

Incorporating DAST Into the CI/CD Pipeline

1. Understanding the Application Structure

Before we can integrate DAST into the CI/CD pipeline, it’s crucial to understand the structure of the application we’re working with. This involves understanding the layers of the application, the technologies used and the data flows. By analyzing the application’s architecture, you can identify the critical components that need more attention during the security testing phase.

Understanding the application structure is not just about knowing the technology stack. It’s also crucial to understand the business logic and the functionality of the application. This will allow you to create better test cases during the DAST process. Additionally, understanding the application structure will enable you to make better decisions when configuring the DAST tool, leading to more efficient and effective security testing.

Finally, understanding the application structure will help you prioritize the vulnerabilities found during the DAST process. Not all vulnerabilities are created equal; some might be more critical due to the data they expose or the potential damage they could cause to the application.

2. Configure the DAST Tool

The next step is to configure the DAST tool. This involves setting up the scanning parameters, defining the test cases, and tuning the tool to reduce false positives.

When configuring the DAST tool, it’s important to strike a balance between thoroughness and speed. You want the tool to be able to identify all potential vulnerabilities, but you also want the scanning process to be fast enough not to slow down the development process. This balance can be achieved by carefully selecting the scanning parameters and continuously tuning the tool based on the results of the scans.

Additionally, when configuring the DAST tool, you should consider the unique characteristics of your application. For example, if your application uses a specific technology or framework that is known to have certain vulnerabilities, you should ensure that the DAST tool is configured to test for these vulnerabilities. Similarly, if your application handles sensitive data, you should prioritize testing the components that process and store this data.

3. Integrating DAST Into the CI Pipeline

The next step is to integrate DAST into the CI pipeline. The CI pipeline is a sequence of steps that developers follow to merge their changes into the main codebase. By integrating DAST into this pipeline, we can ensure that every code change is automatically tested for security vulnerabilities.

Integrating DAST into the CI pipeline can be achieved using various tools and plugins. Most DAST tools provide APIs that allow you to automate the testing process. By using these APIs, you can trigger a DAST scan every time a new code change is pushed to the repository. This allows you to catch vulnerabilities early in the development process, reducing the cost and time needed to fix them.

However, integrating DAST into the CI pipeline is not a one-time task. It requires continuous monitoring and tuning to ensure that the testing process remains effective. You will need to update your DAST configurations as the application evolves, add new test cases as new features are developed, and fine-tune the testing process to reduce false positives.

4. Scan for Vulnerabilities

After integrating the DAST tool, you can start scanning for vulnerabilities. The scanning process involves executing the test cases defined during the configuration phase and identifying potential security issues in the application. Depending on the size and complexity of your application, this process can take from a few minutes to several hours.

During the scanning process, the DAST tool will interact with the application just like a real user would. It will send inputs to the application, observe the responses and analyze them for signs of security vulnerabilities. This dynamic approach allows the tool to identify vulnerabilities that static analysis tools might miss, such as runtime vulnerabilities and issues related to the application’s business logic.

5. Analyze and Prioritize Findings

Once the scanning process is complete, the next step is to analyze and prioritize the findings. The DAST tool will provide a report of the identified vulnerabilities, including their type, location and potential impact. This information should be used to prioritize the vulnerabilities based on their risk to the application and the business.

Analyzing and prioritizing the findings is not an easy task. It requires a deep understanding of the application, the business, and the threat landscape. You need to consider various factors, such as the potential impact of the vulnerability, the likelihood of exploitation, the sensitivity of the affected data, and the complexity of the remediation process. By considering these factors, you can prioritize the vulnerabilities and focus your efforts on those that pose the highest risk.

In addition, analyzing and prioritizing the findings should involve all relevant stakeholders. Developers, security teams, and business leaders should all participate in this process, as they each bring unique perspectives and insights that can help make better prioritization decisions.

6. Remediate and Retest

After analyzing and prioritizing the findings, the next step is to remediate the vulnerabilities and retest the application. Remediation involves modifying the application’s code or configuration to eliminate the identified vulnerabilities. Depending on the complexity of the vulnerabilities, this process can involve simple code changes or significant architectural modifications.

After remediation, it’s crucial to retest the application to ensure that the vulnerabilities have been effectively addressed. This involves running the DAST tool again and verifying that the previously identified vulnerabilities are no longer present. Additionally, retesting allows you to check for new vulnerabilities that might have been introduced during the remediation process.

Remediation and retesting is a continuous process. As new vulnerabilities are discovered and new features are developed, you will need to repeat this process to ensure the ongoing security of your application. This continuous effort is a critical component of the shift left with DAST strategy.

7. Continuous Monitoring and Updates

The final step in incorporating DAST into the CI/CD pipeline is continuous monitoring and updates. This involves regularly checking the effectiveness of the DAST tool, updating its configurations as the application evolves, and tuning it to reduce false positives.

Continuous monitoring and updates are crucial to maintaining the effectiveness of your shift left with DAST strategy. As the application evolves and the threat landscape changes, the DAST tool needs to be updated to remain effective.

Conclusion

In conclusion, incorporating DAST into the CI/CD pipeline is a powerful strategy to enhance the security of your software. By understanding the application structure, integrating DAST into the CI pipeline, configuring the DAST tool, scanning for vulnerabilities, analyzing and prioritizing findings, remediating and re-testing, and continuously monitoring and updating, you can unleash the power of shift left with DAST. This comprehensive approach will allow you to identify and address security vulnerabilities early in the development process, leading to more secure and reliable software.

July 18, 2023
3 Must-Haves When Implementing DevSecOps

The term DevSecOps is already more than a dozen years old. DevOps—the practice of combining software development with IT operations to deploy applications faster—was first coined in 2008 and refined in a historic conference presentation in 2009. DevSecOps—the concept of baking in security at every stage of Agile development, rather than tackling it at the end—followed soon after.

Despite a huge amount of industry conversation and activity around DevSecOps, it has been hard to tell just how much traction the movement has been gaining.

In a 2020 survey of 1,500 IT professionals around the world by the Synopsys Cybersecurity Research Center, 63% reported that they are incorporating at least some DevSecOps activities into their development pipelines. “At least some” indicates, however, that a portion of the adopters have yet to fully embrace DevSecOps. And what about the other 37% who haven’t incorporated it at all?

Nevertheless, two factors suggest that DevSecOps is starting to pass a tipping point on the way to mainstream adoption.

One, the prodigious number of cyberattacks that have been hitting companies and government agencies demand aggressive and proactive approaches to address security, which is exactly what DevSecOps does.

Two, the pandemic has been a powerful accelerant for digital initiatives. For example, global spending on public cloud services was predicted to grow slightly more than 23% in 2021, to $332.3 billion from $270 billion in 2020, according to a forecast from Gartner. In addition to the greater reliance on technology driven by COVID-19, increasing use of emerging technologies such as containerization and edge computing are fueling higher cloud spending. All of this points to the need for organizations to take security risk into account throughout the modern software development life cycle.

The logic behind DevSecOps is undeniable. Viewing security as a shared, collaborative responsibility among development, operations and security functions starting at the earlier possible stage—the methodology known as shifting left—rather than in silos enables the deployment of secure software at the speed and responsiveness that today’s business conditions require.

Companies that have successfully meshed the three functions under the broader DevOps paradigm enjoy radically reduced friction in software development and production. These organizations can deploy software faster and more safely with fewer design flaws, misconfigurations and other missteps that can lead to security issues.

Companies not as far along in their DevSecOps journeys tend to have espoused some of its practices but not consistently. Clunky, linear waterfall approaches persist here and there. Vulnerability management remains a reactive box-checking effort later in the life cycle, forcing developers to backtrack and fix the code rather than a proactive, preventative process seamlessly integrated from design to production.

The more advanced organizations typically focus on three key ingredients to assure success.

1. Education. DevSecOps is a cultural shift as much as a technological one. It requires a new mindset that empowers developers to take ownership for creating secure code at the beginning of the software development life cycle. Thus, IT leaders can’t simply snap their fingers and expect DevSecOps to take hold without educating team members on why it is so critical and how it drives rather than disrupts great work.

“Organizational learning and change are key to allowing DevOps to flourish,” senior Gartner analyst George Spafford wrote. “In other words, people-related factors tend to be the greatest challenges, not technology.”

This means that before introducing DevSecOps into an organization, it is important to connect the endeavor with the value it will bring and explain why it will allow the business to deliver more value to customers and improve its competitiveness.

2. Process. The right DevSecOps policies and processes can spell the difference between a nimble software development machine and one still bogged down by inefficiency.

For instance, many organizations that rely heavily on scanning tools to detect vulnerabilities assign nonsensical policies to their use. Let’s say that vulnerabilities of a particular score or type need to be eliminated before code is deployed to production. But what if the flaw resides in an unimportant section of the code and is unexploitable?

The best DevSecOps initiatives include policies and processes that seek to understand the entire context of a project, from design to production and handle everything in a way that reflects the context of true risk to security and compliance requirements rather than merely running all code through the same pipeline.

3. Tooling. Fact: Without the right automated tools that provide deep visibility into code, a DevSecOps implementation doesn’t stand a chance.

To carry out the qualities of collaboration and agility that DevSecOps promises, organizations should gravitate toward tools that give developers all the relevant information they need to make more informed decisions.

These tools must truly simplify the addition of security into DevOps methodologies by automating various manual processes that traditionally have slowed everything down—threat models, security and compliance reviews, pentests and risk assessments and the like. The tools can’t just aggregate data from silos — which fails to provide enough contextual information and generates too many false positives —they need to provide holistic relief to overburdened teams in securing applications at the feature request, commit, pull request and CI/CD stage.

By keeping these three elements in mind, organizations can fulfill DevSecOps’ potential as a true game-changer. As DevSecOps continues to cross the chasm to more universal adoption, organizations can look to earlier adopters for guidance on how to do DevSecOps right.

March 24, 2022
What to Expect When Transitioning to DevSecOps
How do you ensure your DevOps pipeline is secure? Does DevSecOps protect you against serious breaches or is it just a way to allay the concerns of stakeholders about security in DevOps?

A data breach can cost an average of $3.92 million USD, as per IBM’s study Cost of a Data Breach. In this report, health care was found to be the most expensive industry, with breach fines averaging $6.45 million USD.

While DevOps delivers on the promise to increase the rate of delivery, the traditional CI/CD framework does not emphasize security. For the sake of speed, misconfigurations and known vulnerabilities may be allowed to pass through the development pipeline to production.

Security and testing are generally a low-priority activity, which means these vulnerabilities are fixed retroactively. This takes more time, effort and money. With rampant, unverified integration of open source tools, you open up the software to attackersthat could compromise the security of your apps, systems and users’ data.

DevSecOps is an upgrade to DevOps, allowing you to deliver security and compliance testing continuously. These checks get integrated within the CI/CD pipeline to enable the delivery of more secure code faster.

From DevOps to DecSecOps: What to Expect

The automated nature of DevOps delivers several benefits that enable teams to realize the true power of DevOps as silos break and delivery rates skyrocket.

As DevOps practices span the entire delivery pipeline, it provides the necessary mechanism to deliver audit, compliance and security requirements along with speed and efficiency.

There are many benefits of upgrading from DevOps to a dedicated DevSecOps practice, and they are important to understand before you embark on that journey.

1. No Restrictive Access Control Policies for Developers

Access to infrastructure assets must be strictly controlled and monitored to enforce requisite security measures. This means approval gates, secure configuration parameters and other security measures.

With DevSecOps built into the pipeline, these access controls can be automatically applied. This enables on demand access to infrastructure resources for the team and reduces their dependence on factors outside the team.

For example, many organizations have started deploying internal DevOps teams that enable automated access to a shared pool of deployment processes, tools, repositories, workflows, etc.

This team is usually separate from the IT team but enforces security mechanisms with the same rigor. It ensures that access to all the infrastructure instances required by development, QA or production is automatically tracked, monitored and controlled as per the guidelines. Additionally, inconsistent processes or misconfigurations are also controlled.

Often called the shadow IT team, the idea of this team is not to circumvent the IT department but to create an additional team that matches the agility of the DevOps team.

2. Seamless Security and Compliance Testing

Traditionally, security and compliance testing happened all at once at the end of the release cycle close to the release date. This typically meant that a large spreadsheet of issues is sent to the development team to be fixed immediately, since maintaining a consistent release velocity is the key to DevOps success.

This requires the development team have at least one dedicated resource to work on those issues. When metrics are tracked, to no one’s surprise, this method resulted in negative impacts on the velocity of the next release cycle.

DevSecOps applies all the security and compliance checks in batches. So, testing for security and compliance is shifted left and spread out across the pipeline.

This makes the security feedback loops shorter and more frequent, thus aligning security and compliance testing with the DevOps philosophy. As more secure code is committed every time, the possibility of security and compliance errors reported by the QA team is minimized.

A second, more personalized approach is using the integrated development environment (IDE) itself to take security and compliance checks to the developers. With regular and automated security and compliance baked right into the development process, this process is sped up considerably and the development team becomes self-reliant.

3. Overcoming the Risks Associated With Open Source Software

The 2019 State of The Software Supply Chain report from Sonatype showed that, on average, 85% of the codebase among modern applications uses open source components. While open source components enable innovation, they also come with considerable security risks. As more of these open source tools are added to the software supply chain, it increases the risks on the cybersecurity front.

The same report showed evidence to back up the need for greater open source software security checks and processes. Companies that managed their open source components released 55% less vulnerable components than those who had no verifiable open source management process.

While many of these open source components released frequent patches, it is also important for the DevSecOps team to step in and regulate the open source components. Some of the ways DevSecOps teams can do this are:
- Create a formal component management plan which makes sure that when employees leave, the practice doesn’t go away with them. Having a documented plan lets you look at the bigger picture of your pipeline and find vulnerable components that may be skipped without a formalized process. Incremental improvements to this plan are also possible if it’s well-documented.
- Use scanning tools like Gemnasium and Snyk that help regularly check for vulnerabilities.
- If vulnerabilities are detected or reported in a component, look at either updating or replacing the component.
- Enable quick approval for new open source software. Alternatively, prepare developers to test new components using security and compliance checklists.
4. Overall Cost Reduction and Acceleration of the Delivery Rate

An integrated security and compliance framework in DevOps prevents breaches and guarantees faster recovery when breaches do happen. As developers don’t have to set aside a large chunk of time to work on security and compliance violations, this shortens release cycles and ensures more secure code at the same time.

The fewer breaches you identify and fix, the more money you save on overcoming security disasters.

5. Automated Compliance Reporting

DevSecOps automates your security and compliance tests and it also logs and documents all the tests. DevOps automation platforms collect large amounts of information for the build, test, integration and deployment phases. This data can be used to create an end-to-end audit trail without any manual intervention or extra time spent scouring through various tools to prepare compliance reports.

A fantastic advantage of DevSecOps is the complete traceability from day one and automated compliance reporting. DevSecOps also focuses on the automation of testing, allowing you to deliver a secure development framework on a continuous basis.

DevSecOps is the Future of DevOps

The rapid release velocity DevOps enables is not without its disadvantages. In particular, the unchecked use of open source tools in the pipeline can expose your applications and systems to many known vulnerabilities, increasing the probability of a breach. Combined with a general lack of focus on security, this is a recipe for an expensive security incident.

To counter this, DevSecOps radically transforms the pipeline by shifting security and compliance testing left. Developers check the code before every commit. With fast feedback loops, they fix any reported vulnerabilities along with other errors in the app.

With DevSecOps, compliance and security checks are not done all at once; instead, developers chip away at vulnerabilities and security flaws with every release.
February 28, 2022
A Blueprint for Securing Software Development
Software development has changed dramatically in recent years, as technologies like DevOps, application containers, and cloud-native transform how software is built and distributed. Unfortunately, attackers have been paying close attention to these changes, and have retooled their attack strategies to take advantage of relatively weak security controls in software development and build environments. Attackers recognize that security teams are more focused on protecting infrastructure and less focused on software development and supply chains. This is how damaging attacks like SolarWinds happen – and it’s likely that attackers will strike again to yield even more successful outcomes.

A new approach is needed to secure the software development pipeline. What’s more, this approach needs to be put in place by the engineers developing the software—and not by information security teams. Engineering teams need to be responsible for design and implementation while giving security teams visibility and assurance that controls are operational and that policies are being enforced.

A new blueprint for securing the pipeline centers on a flexible set of controls that secure the software build pipeline by ensuring that authentication and authorization are properly managed, that the integrity of software artifacts are tested at appropriate stages and that controls are placed on third-party and open source solutions incorporated into the software. The result: Software innovation and delivery at lower risk of supply chain tampering, and reduction of attacks during development.

A Pragmatic Design Philosophy

This modern software security blueprint is based on the following design philosophy:
- Least-privilege: Only grant the access and permissions required to accomplish a job, and no more.
- Immutability: Artifacts and infrastructure are not modified after deployment to an environment. If a change is required, it’s done in the image or script in the development environment and then promoted through the higher environments.
- Everything-as-code: Infrastructure, security policies and other parts of the pipeline are also implemented as code and subject to the same controls as the software artifacts.
- Traceability: All changes, whether to infrastructure or business code, are version-controlled.
Controls Blueprint for Software Development Pipelines

The basis for this software security blueprint centers around a set of 15 controls that are built for modern development pipelines where developers, pipelines and execution can happen anywhere in the world.

Control one: Restrict administrative access to CI/CD tools. Ensure that only authorized people can make administrative changes to the CI/CD system, which prevents unauthorized changes to pipeline definitions.

Control two: Only accept commits signed with a developer GPG key. Unsigned code commits are challenging, if not impossible, to trace and pose a risk to the integrity of the codebase.

Control three: Automation access keys expire automatically. By ensuring that access keys used by automation expire periodically, developers create a shorter window of attack when keys are compromised.

Control four: Reduce automation access to read-only. CI systems should have read-only access to source code repositories following the principle of least-privilege access.

Control five: Only dependencies from trusted registries can be used. By configuring the dependency manager to only allow connections to an authorized list of registries, attacks can be blunted by keeping malicious packages in public registries from entering the pipeline.

Control six: Any critical or high-severity vulnerability breaks the build. Supply chain attacks may introduce code with vulnerabilities into the software pipeline. Static application security testing (SAST) helps to identify serious security issues, including poor cryptographic practices, hard-coded credentials and injection vulnerabilities.

Control seven: Artifacts are stored in a repository in development, staging and production. This control helps to enforce the immutability of the artifacts so they can be compared in the development, staging and production repositories.

Control eight: Validate artifact digest. Before an artifact is deployed in any environment, its digest should be validated against the artifact in the repository to ensure that it has not been compromised.

Control nine: Pull requests require two reviewers (including one default reviewer) and a passing build to be merged. In addition to supporting good coding practices, this control also helps ensure that no commits are made without competent human oversight.

Control 10: Artifacts in higher repositories are signed. Requiring artifacts to be signed in a repository throughout the process ensures visibility and traceability for whatever is deployed to production.

Control 11: Available container images don’t have any high or critical vulnerabilities. Just as applications need to be tested for vulnerabilities before being placed in production, so do the container images into which they are packaged for deployment. Container images could have open source vulnerabilities if they contain open source software.

Control 12: Validate artifact signatures and digests. Validating the signature of the artifact against the digest ensures that the artifact was not tampered with in the repository and that the artifact being deployed is the same one that was tested.

Control 13: Scan deployed images in production. This helps to ensure that the prior controls were followed for all software in production.

Control 14: Validate Kubernetes resource manifests. If the resource manifests are tampered with, they could be tricked into deploying a container of the attacker’s choice.

Control 15: Ensure build environments are ephemeral and immutable. Build environments should be defined in code with automated creation and teardown and a fresh environment created for every build. Build hosts should not be accessible using interactive logins.

We believe this blueprint provides engineering teams with an actionable architecture that security teams can rally behind and support. By adopting these standards, engineering teams are promised the delivery of assured software at lower risk of supply chain tampering, reduction of attacks during development and manipulation in staging and production.

Here’s what it looks like:

To learn more about the suggested blueprint, read our GitHub post about the project. Our goal is to prioritize speed and agility for software development while incorporating a security-always mindset.
October 19, 2021

Tag: pipeline security

From Workflow Success to Workflow Trust

Pipeline Overview

Secrets Belong to the Runtime, not the Repository

Runtime Validation Before Automation

Generated Data Must be Checked

Artifact Integrity and Model Metadata

Audit as a Pipeline Task

Operational Lesson for DevOps Teams

Conclusion

Reproduce the Experiment

Why Are AI Generated Pipelines So Popular?

Just Because It Works Doesn’t Mean It’s Right

The Permissions Trap

Hidden Security Gaps

Supply Chain Problems

Speed Kills Review

Blind Trust Creeps In

So, How Should Teams Use AI?

Looking Ahead

Final Thoughts

How Shai-Hulud Struck

Why This is a DevOps Problem

Lessons for the DevOps Community

A Pattern We’ve Seen Before

Shimmy’s Take

Understanding the Concept of ‘Shift Left’ in Software Testing

DAST in the Context of Shift Left

Importance of DAST in Early Stages of Development

How DAST Complements Other Testing Methods

Incorporating DAST Into the CI/CD Pipeline

1. Understanding the Application Structure

2. Configure the DAST Tool

3. Integrating DAST Into the CI Pipeline

4. Scan for Vulnerabilities

5. Analyze and Prioritize Findings

6. Remediate and Retest

7. Continuous Monitoring and Updates

Conclusion

From DevOps to DecSecOps: What to Expect

1. No Restrictive Access Control Policies for Developers

2. Seamless Security and Compliance Testing

3. Overcoming the Risks Associated With Open Source Software

4. Overall Cost Reduction and Acceleration of the Delivery Rate

5. Automated Compliance Reporting

DevSecOps is the Future of DevOps

A Pragmatic Design Philosophy

Controls Blueprint for Software Development Pipelines