devopsy.kinsta.cloud

Tag: SAST

Are LLMs Equally Good (or Bad) at Building Secure Software?

In the tricky business of choosing the right artificial intelligence coding models for security and cost, development teams need to keep in mind two important factors.

First, even the best and best-known AI coding assistants vary widely in performance, both between and within individual frameworks. Each has strengths and weaknesses, and there is no across-the-board best of the bunch; a model that leads the pack in one framework may be in last place in another.

And second, when it comes to generating secure code, you don’t necessarily get what you pay for. The real costs of using AI agents can mount quickly depending on an agent’s use of tokens, and some models are more inclined than others to make extensive use of them, but there’s no correlation between cost and the level of security provided.

Those are among the findings of an in-depth study conducted by Secure Code Warrior and the Royal Melbourne Institute of Technology (RMIT) to evaluate the security behavior of leading AI coding models. With many software engineering teams moving from AI coding assistants into full agentic AI code generation and increasing the amount of code they produce exponentially, ensuring the security of that code must be a top priority. The study produces practical guidance for organizations that are getting on board the AI-assisted or agentic code development train.

Clear Results, but No Clear Winner

For the initial study, we tested six frontier large language models (LLMs), producing 660 complete application codebases across eleven language/framework combinations. Each codebase was scanned by three independent, open-source static application security testing (SAST) tools, triaged by an AI-powered agentic false-positive verification pipeline, and scored using a multi-factor CWE Risk Model to normalize results across frameworks.

The LLMs tested were Anthropic’s Claude Sonnet 4.5 and Haiku 4.5, OpenAI’s GPT 5.1 and GPT 5 mini, and Google’s Gemini 2.5 Pro and Gemini 2.5 Flash.

Some of the more surprising results include the fact that no model gained a universal advantage in security when measured against OWASP’s Top Ten vulnerabilities, and results varied widely depending on the frameworks used. In some cases, a specific LLM produced twice as much secure code as others while working on identical tasks. But no matter which you choose, you’ll have to accept that a model has weak spots as well as strengths—and you should be aware of both.

That said, GPT 5.1 scored above average across all the OWASP categories, performing strongest on Software Integrity (number 8 of the OWASP list), and Security Logging and Monitoring Failures. Its weakest area was Insecure Design.

By contrast, Gemini 2.5 Pro scored highest on Insecure Design and Identification and Authentication Failures. And like GPT 5.1, it was strong in most categories, but did fall short when tested against Integrity Failures and Logging & Monitoring. Meanwhile, Sonnet 4.5 had the most balanced profile, with no dramatic strengths or weaknesses, and Haiku 4.5 was moderately above or below average in most areas. Gemini 2.5 Flash was below average in almost all areas, but it did show elevated results in Server-Side Request Forgery (SSRF). GPT 5 mini, an efficient tool with the lowest per-token costs among the models tested, brought up the rear, scoring below average in all categories.

The overall security scores:

GPT 5.1: 79.6

Gemini 2.5 Pro: 73.5

Sonnet 4.5: 71.2

Haiku 4.5: 49.5

Gemini 2.5 Flash: 36.4

GPT 5 mini: 10.0

Overall performance is one thing, but how each model performs in different frameworks can be just as important, depending on the programming environment.

Tests showed, for instance, that Sonnet 4.5 dominates the Java ecosystem and React (performing best in C# Basic, Java EE JSP, Java Spring, Java Spring API, and JavaScript React). Gemini 2.5 Pro shone in Python and Swift (Python Basic, Python Django and Swift iOS SDK), and GPT 5.1 dominated in low-level and API-centric frameworks (performing best in C Basic, Java EE API and JavaScript Basic).

Security First, but The Costs Can’t Be Ignored

The other side of the coin when choosing an AI coding model is cost, the biggest portion of which may have been overlooked in the nascent stage of agentic AI.

After the release of the Chinese LLM DeepSeek, a bargain-basement model that compared pretty favorably to the leading LLMs at the time, prices for LLMs fell steeper and faster than they had for any technology in history. That’s because the only costs most people considered were how much each model charged per token. But the arrival of agentic AI—autonomous and highly active—quickly changed the equation.

LLMs charge AI agents for processing requests. The number of token calls can vary dramatically among models, even when performing the same task, and they add up quickly. Because agentic AI workflows involve multiple steps, including interacting with other agents and applications, an agentic action could involve myriad tokens—up to about a thousand times more than an interaction with a chatbot.

Input tokens (sent to an AI) typically cost from 15 cents to $5 per million; output tokens (what the AI sends back) require more processing and have per-million costs ranging from about 60 cents to $25. At a glance, token costs can look like small potatoes in the scheme of things, but they add up quickly, especially with active AI agents that make avid use of token calls. With token use growing as more organizations use AI agents, token costs have increased IT budgets by about 20%, according to recent estimates.

Our tests revealed wide cost variation among the models, driven by each model’s reliance on tokens. The Claude Sonnet models made the heaviest use of tool-calling turns, using considerably more tokens per run than the other models. Sonnet 4.5 coder produced 176 million output tokens across 110 runs, while GPT 5.1 produced 42 million and Gemini 2.5 Flash used 20 million.

To a much greater extent than what each model charges per token, the number of tool calls created the wide range of costs. For example, GPT 5.1’s highest overall normalized security score was achieved with a cost per run of $5.60. Claude Sonnet wasn’t far behind on security but wound up costing $44 per run.

Sonnet had the highest per-token costs as well, at $3 per million inputs and $15 per million outputs, but the number of API calls was the biggest reason for the disparity in per-run costs. (GPT 5 mini charges 25 cents per million inputs and $2 for a million outputs. Gemini 2.5 Flash was 30 cents for inputs and $2.50 for outputs. Haiku charged $1 per million inputs and $5 for outputs. And both GPT 5.1 and Gemini 2.5 Pro had the same costs of $1.25 per million inputs and $10 for every million output tokens.)

Preparing Developers for the Future of Software Development

LLMs are like developers in that each has unique strengths you should play to and weaknesses you should avoid when striving to produce secure code. The frameworks in which AI agents operate will also play a big part in deciding which tools to use for which purposes, as will unavoidable cost considerations.

But organizations also need to pay attention to future growth in developers’ skills, providing the upskilling they need to move upstream from code creation and review to orchestrating AI architecture and secure design. When combined, a clear understanding of the security and cost factors of LLMs across different frameworks, alongside a team of developers with the skills to ensure secure code production, can enable organizations to ride the front of the wave in the new world of software development.

We have since expanded the study to include 16 LLMs, including Anthropic’s Fable 5, as well as Chinese models like Alibaba’s Qwen3 and European models such as Mistral Devstral2, with over 1,700 AI-generated codebases and additional cost analysis. These numbers will likely shift as the technology evolves, and it will be imperative for security leaders to monitor changes and continue honing their security programs to reduce security risk, friction, and errors as these tools become more autonomous and embedded alongside human orchestrators.

August 19, 2026
Is Your New DevSecOps Tooling Reducing Work Or Just Adding to It?
Security belongs in the software delivery pipeline. The harder question is where, how often and at what cost.

Many pipeline teams eventually add security scanning to CI/CD, and relatively few go back afterward to measure what it actually cost the delivery process. Security coverage went up. Something else changed too, and it is rarely measured with the same rigor.

“Shift left” gets treated as a free upgrade: catch problems earlier, at lower cost, with no real downside. That is true for the cost of fixing a vulnerability. It is not automatically true for the cost of running your pipeline. A security control can be worth adding and still change your delivery economics in ways worth naming honestly, rather than assuming they net out to zero.

What Changes When Security Enters the Pipeline

SAST, SCA, container scanning, secret scanning, and dependency analysis all do real work, and none of them are free to run. Each one is a pipeline stage with its own execution time, and many scale with codebase size or dependency count rather than staying flat. A scan that runs quickly on a small service can take meaningfully longer on a large monorepo, and image scanning in particular can grow with the number and size of layers in a container, not just the size of the application code itself.

Findings from new scanners also generate new work: triage, false-positive review, and in some cases blocked builds while a team decides whether a flagged issue is real. Blocked builds mean retries. Retries mean re-running earlier pipeline stages that had already passed, which is easy to overlook if you are only looking at how long the new scanning stage itself takes rather than its effect on the whole pipeline run.

None of this is a flaw in the tooling. It is the actual mechanics of adding a new, non-trivial stage to a pipeline that used to run without it. Running new stages in parallel where the pipeline architecture allows it can offset some of this, but parallelization has its own compute cost, and it does not eliminate the added triage and retry work downstream.

What I’ve Actually Seen

I’ve personally observed this pattern across pipelines I’ve worked on: adding security tooling increased overall pipeline workload, delayed artifact and binary production, and produced a noticeable increase in CI/CD infrastructure cost.

I want to be precise about what that is and is not. It is a practitioner observation from real pipelines, not a benchmark, not a percentage, and not a claim that every team will see the same magnitude of effect. The pattern is consistent enough to be worth naming. The number is not something I would generalize.

Ask What It Replaces

Before adding a control, it is worth asking a question teams skip: does this replace something we already do, or does it sit alongside it?

A new SCA tool that catches known-vulnerable dependencies does not automatically make an existing manual dependency review obsolete. Sometimes it should, sometimes the two catch genuinely different things, and sometimes they overlap almost entirely. The same question applies to container scanning layered on top of an existing manual image-review checklist, or secret scanning added alongside a code-review step that was already supposed to catch hardcoded credentials.

Overlap without a corresponding removal is duplicated engineering effort dressed up as added security. Nobody put it there on purpose. It accumulates one well-intentioned addition at a time, and it is rarely anyone’s job to go back and remove the process a new tool has quietly made redundant.

The Cost Most Teams Don’t Model

A more complete way to think about a new security control’s real cost looks something like this:

incremental DevSecOps cost = tool or service cost + additional CI/CD compute + additional execution time + retry and rebuild cost + engineering triage effort + delivery and release impact

Most cost conversations stop at the first term. The rest are real and they compound: more compute per run, more minutes per run, more reruns when a build fails a new gate, more engineer-hours spent triaging what the gate flags, and a measurable effect on how quickly artifacts actually ship.

For high-frequency pipelines, the useful comparison is not simply scan duration but the cumulative effect across builds: execution minutes, compute consumption, reruns, and time-to-artifact over a representative period. None of these need a specific number attached to be worth putting in the model. The point is that the model should include them at all.

Five Questions Worth Asking Before the Next Tool
1. What security gap does this control actually close?
2. What existing control, check, or manual process does it replace fully or partially?
3. How much pipeline execution time does it add, end to end?
4. How much additional compute does it consume, and how does that scale with the codebase?
5. What effect does it have on artifact and release throughput?
None of these questions argue against adding the control. They argue for knowing the answer before you do.

Where This Leaves You

This is not an argument for fewer security controls in CI/CD, and it is not an argument that security tooling is a poor investment. Security coverage and delivery speed are not opposing goals; they are both outputs of the same system, and mature DevSecOps practice treats them that way instead of optimizing one while quietly absorbing the cost on the other.

The lesson is not simply “measure security ROI.” It is simpler: measure the whole delivery system, not just the part of it you were already tracking. A security control that closes a real gap can be worth the pipeline cost it introduces, but you need to understand that cost before making the trade-off.
August 17, 2026
Shift Left Security: 4 Automated Security Gates in GitHub Actions
A security researcher filed a report against our Node.js API two years ago. The vulnerability was prototype pollution in an npm package we had been shipping for eight months. The fix took about 20 minutes. The internal review to figure out how it got through took two days.

The package had a known CVE. It had been sitting in the NVD for months before we found it. Our CI never checked. Code review didn’t catch it. The researcher did.

There’s a stat from IBM’s Cost of a Data Breach 2024 report that I keep referencing when this comes up: The average breach costs $4.88 million. Teams running DevSecOps practices save $1.68 million from that cost on average. The number I actually find more useful is smaller. A vulnerability caught in CI costs about $80 to fix. Caught in production, the same costs $7,600.

That gap is why this article exists. Four automated security gates wired into GitHub Actions, running on every push and pull request — none of them require an enterprise license:
- Gate 1: Dependency scanning — finds CVEs in your npm packages before they ship.
- Gate 2: Container image scanning — finds CVEs in your Docker base image OS packages.
- Gate 3: Static analysis (SAST) — finds code-level vulnerabilities such as SQL injection and hardcoded secrets.
- Gate 4: Dynamic scanning (DAST) — finds runtime vulnerabilities in a running instance of your app.
Wired together, the pipeline looks like this:

Pull Request / Push to main

│

▼

┌──────────────────────────────────────────────────────┐

│ GitHub Actions Pipeline │

│ │

│ Gate 1: Dependency Scan Gate 2: Image Scan │

│ (npm audit + Snyk) (Trivy) │

│ Finds: vulnerable packages Finds: OS + pkg CVEs│

│ ❌ CRITICAL → fail build ❌ CRITICAL → fail │

│ │

│ Gate 3: Static Analysis Gate 4: DAST │

│ (CodeQL) (OWASP ZAP Baseline)│

│ Finds: SQL injection, XSS, Finds: runtime vulns│

│ hardcoded secrets in running app │

│ ❌ High severity → fail ❌ Alerts → report │

│ │

│ ✅ All gates pass → merge allowed / deploy │

└──────────────────────────────────────────────────────┘

Gates 1 through 3 run in parallel on every push. Gate 4 only starts after all three pass. It boots the app and probes it live.

Prerequisites

Docker needs to be installed locally before you start. Most of the local testing this article highlights relies on it. The Snyk step needs a free account at snyk.io, but npm audit alone covers the basics if you’d rather not sign up. One other thing upfront: CodeQL runs free on public repositories; no paid GitHub plan is needed.

The sample app used throughout this article is an Express API with a Dockerfile and docker-compose.yml. The full code is on GitHub at github-actions-security-gates.

Gate 1: Dependency Scanning With Npm Audit and Snyk

Recently, I ran npm install on a fresh Express app. It pulled in 847 packages. I explicitly chose maybe 12 of them. The rest came along for the ride. Each one was a potential CVE.

Both tools scan the same dependency tree but do different things with what they find. npm audit catches what Node knows about and reports severity. Snyk goes further. The last time I ran it on a project, it surfaced issues npm audit missed, and gave me the exact upgrade version for each one, flagging which jumps would break the API. That detail cuts debug time significantly.

Drop this in .github/workflows/gate-1-dependency-scan.yml:

name: Gate 1 — Dependency Scan

on:

push:

branches: [main]

pull_request:

branches: [main]

jobs:

dependency-scan:

runs-on: ubuntu-latest

steps:

- name: Checkout code

uses: actions/checkout@v4

- name: Set up Node.js

uses: actions/setup-node@v4

with:

node-version: '20'

cache: 'npm'

- name: Install dependencies

run: npm ci

- name: npm audit

run: npm audit --audit-level=high

- name: Snyk dependency scan

uses: snyk/actions/node@v1.4.0

continue-on-error: false

env:

SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }}

with:

args: --severity-threshold=high

The --audit-level=high flag in npm audit means the job only fails on high and critical vulnerabilities, not on every minor advisory. Without this flag, most real codebases will fail immediately on low-severity issues that have no fix and no meaningful risk.

Snyk adds upgrade paths. When something gets flagged, the report shows which version resolves it and whether the jump is safe. I’ve had engineers close a failing PR in five minutes with that information. Without it, they’re opening changelogs and guessing.

Getting SNYK_TOKEN Into GitHub

Sign up at snyk.io, then go to your personal account settings at https://app.snyk.io/account. The Auth Token is there — not on the Organization Settings page, which only has the org API key. Copy it, then add it to your repo at Settings → Secrets and variables → Actions → New repository secret. Name it SNYK_TOKEN. That’s the exact name the workflow expects.

Handling False Positives With a `.snyk` Ignore File

Do not turn off a gate because one advisory is a false positive. Document the exception instead:

# .snyk

version: v1.25.0

ignore:

SNYK-JS-INFLIGHT-6095116:

- '*':

reason: No upstream fix available. Package used only in dev dependencies, not shipped to production.

expires: '2027-01-01T00:00:00.000Z'

created: '2026-07-01T00:00:00.000Z'

Three fields matter: reason explains the decision, expires forces a review date and created creates an audit trail. If a SOC 2 auditor asks why you suppressed a CVE, this file is your answer.

Gate 2: Container Image Scanning With Trivy

Dependency scanning only looks at your npm packages. Your Docker base image carries its own OS-level packages: glibc, curl, openssl. Those have CVEs too.

Trivy scans the entire container image: OS packages, language packages and configuration issues. It runs in CI before the image ever touches a registry.

Create .github/workflows/gate-2-image-scan.yml:

name: Gate 2 — Container Image Scan

on:

push:

branches: [main]

pull_request:

branches: [main]

jobs:

image-scan:

runs-on: ubuntu-latest

steps:

- name: Checkout code

uses: actions/checkout@v4

- name: Build Docker image

run: docker build -t myapp:${{ github.sha }} .

- name: Scan image with Trivy

uses: aquasecurity/trivy-action@0.28.0

with:

image-ref: myapp:${{ github.sha }}

format: table

exit-code: '1'

severity: 'CRITICAL,HIGH'

ignore-unfixed: true

Two decisions here are worth explaining.

First, ignore-unfixed: true. Some CVEs have no patch available. If you leave this out, your build fails on vulnerabilities that have no fix. It blocks your team with no action they can take. This flag filters those out. You still see them in the report; they just don’t block the build.

Second, scan before pushing to the registry. A lot of pipelines build the image, push it, then scan. That means a vulnerable image is sitting in your registry for however long the scan takes. Build it, scan it, push only if clean.

Suppressing Known False Positives With `.trivyignore`

# .trivyignore

# CVE-2023-44487 HTTP/2 Rapid Reset - handled at load balancer (nginx rate limits + conn caps)

# owner: zia.ullah@valueadd.se | next review: 2027-01-01

CVE-2023-44487

Same principle as the Snyk ignore file: Document the reason, add a review date.

Gate 3: Static Analysis With CodeQL

Dependency scanning and image scanning look at what you install. CodeQL looks at what you write.

It traces data flows through your code. SQL injection, where input from a route param reaches a database call without sanitization; XSS, where a user-controlled string ends up in an HTML response; hardcoded secrets buried in config files; path traversal — these aren’t hypothetical patterns; they’re the vulnerability classes in most breach reports.

Create .github/workflows/gate-3-codeql.yml:

name: Gate 3 — Static Analysis (CodeQL)

on:

push:

branches: [main]

pull_request:

branches: [main]

schedule:

- cron: '0 6 * * 1' # weekly scan on Monday at 6am

jobs:

static-analysis:

runs-on: ubuntu-latest

permissions:

actions: read

contents: read

security-events: write

steps:

- name: Checkout code

uses: actions/checkout@v4

- name: Initialize CodeQL

uses: github/codeql-action/init@v3

with:

languages: javascript, typescript

queries: security-extended

- name: Autobuild

uses: github/codeql-action/autobuild@v3

- name: Perform CodeQL Analysis

uses: github/codeql-action/analyze@v3

with:

category: "/language:javascript"

The queries: security-extended line matters. The default query set catches the most obvious issues. The extended set adds more SQL injection patterns, SSRF detection and stricter taint tracking. For a production application, use extended.

The security-events: write permission lets CodeQL post results to GitHub’s Security tab. You’ll see findings there even when the build passes.

CodeQL is free for public repositories. For private repositories, it requires GitHub Advanced Security. If you’re on a private repo without GHAS, Semgrep OSS is a free alternative:

- name: Semgrep scan

uses: returntocorp/semgrep-action@v1

with:

config: p/owasp-top-ten

Gate 4: DAST With OWASP ZAP

The first three gates analyze artifacts: Package manifests, container layers, source code. Gate 4 does something different. It actually starts your application and attacks it.

Gate 4 was the one I was most curious about when I first set this up. It actually boots the app and fires real HTTP requests at it. What comes back tells you things the source code never could. A 500 where the app should return a 400. Missing headers on endpoints that aren’t configured. Open redirects. None of that shows up in a static scan.

The OWASP ZAP baseline scan is fast enough for CI. It runs passive checks. No brute-forcing, no fuzzing. It observes and probes. Most pipelines complete it in under three minutes.

Create .github/workflows/gate-4-dast.yml:

name: Gate 4 — DAST (OWASP ZAP)

on:

push:

branches: [main]

pull_request:

branches: [main]

jobs:

dast:

runs-on: ubuntu-latest

steps:

- name: Checkout code

uses: actions/checkout@v4

- name: Start application

run: |

docker compose up -d

sleep 15

- name: Wait for app to be ready

run: |

for i in {1..10}; do

curl -s http://localhost:3000/health && break

echo "Waiting for app..."

sleep 3

done

- name: ZAP Baseline Scan

uses: zaproxy/action-baseline@v0.10.0

with:

target: 'http://localhost:3000'

rules_file_name: '.zap/rules.tsv'

fail_action: warn

- name: Stop application

if: always()

run: docker compose down

fail_action: warn is the right default for DAST. ZAP has a higher false-positive rate than the other gates. On the first run against a JSON API, you’ll see alerts for missing CSP headers, missing HSTS and permissions policy. Most of those are intentional. Setting it to warn keeps the findings visible in the job output without blocking the build. Review the ZAP report on a schedule; don’t let warnings pile up unreviewed.

Cutting ZAP Noise With `.zap/rules.tsv`

10021IGNORE(Strict-Transport-Security not set on HTTP - intentional on local non-TLS endpoint)

10038IGNORE(Content Security Policy not set - handled at reverse proxy level)

The rule IDs are in the ZAP docs at zaproxy.org/docs/alerts. I kept that page open during the first few scans. 10038 is the CSP one. 10035 is HSTS. After looking them up a couple of times, you stop needing to.

Running All Four Gates in One Workflow

Four separate workflow files work fine for learning, but in practice, I merge them into one. The first three jobs run in parallel. Gate 4 holds until all three finish: No point scanning a live app if the code already has a known SQL injection in it.

Create .github/workflows/security-gates.yml:

name: Security Gates

on:

push:

branches: [main]

pull_request:

branches: [main]

jobs:

dependency-scan:

runs-on: ubuntu-latest

steps:

- uses: actions/checkout@v4

- uses: actions/setup-node@v4

with:

node-version: '20'

cache: 'npm'

- run: npm ci

- run: npm audit --audit-level=high

- uses: snyk/actions/node@v1.4.0

continue-on-error: false

env:

SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }}

with:

args: --severity-threshold=high

image-scan:

runs-on: ubuntu-latest

steps:

- uses: actions/checkout@v4

- run: docker build -t myapp:${{ github.sha }} .

- uses: aquasecurity/trivy-action@0.28.0

with:

image-ref: myapp:${{ github.sha }}

format: table

exit-code: '1'

severity: 'CRITICAL,HIGH'

ignore-unfixed: true

static-analysis:

runs-on: ubuntu-latest

permissions:

actions: read

contents: read

security-events: write

steps:

- uses: actions/checkout@v4

- uses: github/codeql-action/init@v3

with:

languages: javascript, typescript

queries: security-extended

- uses: github/codeql-action/autobuild@v3

- uses: github/codeql-action/analyze@v3

with:

category: "/language:javascript"

dast:

runs-on: ubuntu-latest

needs: [dependency-scan, image-scan, static-analysis]

steps:

- uses: actions/checkout@v4

- name: Start application

run: |

docker compose up -d

sleep 15

- name: Wait for health check

run: |

for i in {1..10}; do

curl -s http://localhost:3000/health && break

sleep 3

done

- uses: zaproxy/action-baseline@v0.10.0

with:

target: 'http://localhost:3000'

rules_file_name: '.zap/rules.tsv'

fail_action: warn

- name: Stop application

if: always()

run: docker compose down

The needs: [dependency-scan, image-scan, static-analysis] line is the key. Gate 4 only starts if the first three complete without error.

Handling False Positives Without Turning Off Gates

Every team that installs security tooling eventually hits a false positive and considers disabling the rule. I’ve seen it happen within the first week on most teams. The right response is to suppress it with documentation, not remove it.

Each tool has its own mechanism.

Snyk uses .snyk with expiry dates. A suppression that never expires is forgotten. Set an expiry. When the date passes, the build fails again and forces a review.

Trivy uses .trivyignore with CVE IDs and comments. One CVE per line, one comment explaining why.

ZAP uses .zap/rules.tsv with rule IDs. The ZAP documentation lists every rule ID and what it detects. Rule 10038 is CSP. Rule 10035 is HSTS. Once you know the IDs, suppression is a one-liner.

CodeQL suppressions go in the code itself:

// codeql[js/sql-injection]

const result = await db.query(query);

The codeql comment tells CodeQL to ignore the finding on that line. Use it rarely, document why.

If a SOC 2 auditor asks about your suppression decisions, these files are your evidence. They show that you evaluated each finding, made a documented decision and set a review date. That’s more defensible than having no suppressions and no findings because the tool is turned off.

Troubleshooting

These are real issues that surfaced while building and testing this pipeline, not edge cases. Each one will stop your build until you know the fix.

ZAP Docker Image Not Found: `owasp/zap2docker-stable`

This image was deprecated and removed from Docker Hub. Any article or Stack Overflow post referencing it is out of date. The correct image is:

ghcr.io/zaproxy/zaproxy:stable

Update your local test command and any workflow files that reference the old image name. The GitHub Container Registry image is maintained by the ZAP project team and receives regular updates.

Healthcheck Shows `(unhealthy)` — Curl: Command Not Found

node:20-slim is a minimal Debian image. It does not include curl. A healthcheck that uses curl will silently fail and mark the container as unhealthy, which blocks ZAP from scanning a ready application.

Switch the healthcheck to use Node’s built-in http module instead:

healthcheck:

test: ["CMD", "node", "-e", "require('http').get('http://localhost:3000/health', r => process.exit(r.statusCode === 200 ? 0 : 1))"]

interval: 10s

timeout: 5s

retries: 3

start_period: 10s

Node’s http module ships with every Node image. No extra packages, no install step. Works on alpine, slim and full Debian variants.

Docker Compose Fails With “port is Already Allocated”

This one caught me the first time I ran the local setup. I had npm start running in another terminal, so docker compose couldn’t bind port 3000. It also happens when a container from a previous run is still sitting there — docker ps will show it.

Before running docker compose, stop anything using the port:

npx kill-port 3000

docker ps -a # check for running containers from other projects

docker stop <container-name>

If you see docker compose up succeed but the container exits immediately, the port conflict is the first thing to check.

ZAP Scan Fails With “Connection Refused” or Downloads With EOF Error

Two separate issues that look the same.

I hit this on the first local test. The container looked like it started fine, but ZAP couldn’t reach it. docker ps still showed (starting). The healthcheck needs to pass before ZAP can connect. If it’s stuck there, add more time to start_period in docker-compose.yml and try again.

The EOF error during the ZAP image download means the pull got cut off partway. Pull it separately first:

docker pull ghcr.io/zaproxy/zaproxy:stable

Then re-run your scan command. The cached layers will be used on the next attempt.

Snyk Returns 401 Unauthorized in GitHub Actions

It shows up as SNYK-0005 in the workflow logs. Usually, one of two things caused it.

First, the token location. Snyk has two types of keys: An Organization API key and a personal Auth Token. The GitHub Actions integration requires the personal Auth Token, not the org key. Get it from https://app.snyk.io/account under “Auth Token” — not from the Organization Settings page.

Second, the secret scope. The secret must be added to the specific repository where the workflow runs. Organization-level secrets are not automatically shared with all repositories. Go to the repository, then Settings → Secrets and variables → Actions → New repository secret. Name it SNYK_TOKEN.

Trivy Reports CVEs That Aren’t in Your Code

The first time I ran Trivy against this project, it flagged 18 findings. My application code had zero issues. Everything else came from two places: The Debian base image packages and npm’s own bundled toolchain. Trivy scans the whole container, not just what you wrote.

Debian packages such as libgnutls30 and libcap2 come bundled with node:20-slim. In our case, most of those CVEs were DTLS vulnerabilities in a library our HTTP-only API never touches. They flag, but there’s no exploitable path.

The npm ones took me a minute to understand. The path gave it away — /usr/local/lib/node_modules/npm/node_modules/. That’s not your app. That’s npm’s own private dependencies: tar, minimatch, cross-spawn. That directory is npm eating its own dog food — packages npm uses internally when it installs things. Once the build is done and the container starts, nothing calls that code again.

Handle both with .trivyignore. Document each suppression:

# .trivyignore

# libgnutls30 — DTLS vulnerability, not exploitable in this HTTP-only API

# Fixed version available in Debian but not yet in node:20-slim

# Approved: your.email@company.com | Review: 2027-01-01

CVE-2026-33845

# tar (npm internal) — npm's own toolchain, not in runtime dependencies

# Only used during npm install, not exposed in the running container

CVE-2026-23745

The pattern is the same as .snyk: Reason, owner, review date.

Node_modules Committed to Git — Missing .gitignore

If you initialize a repository and run npm install before creating a .gitignore, Git will track node_modules and .env. A push with node_modules sends thousands of files to GitHub and can expose a committed .env file.

Add this before you push anything:

node_modules/

.env

*.log

If you pushed before adding .gitignore, node_modules is already in Git’s history. You need to untrack it without deleting it from the disk:

git rm -r --cached node_modules

git rm --cached .env

git commit -m "Remove tracked files that should be ignored"

CodeQL Fails With “No Source Code Was Seen During the Build”

I hit this on a project where the build script wasn’t the default. CodeQL’s autobuild guesses your build command — it gets it right most of the time, but not always. When it misses, swap it out for an explicit run step:

- name: Manual build

run: npm ci && npm run build

ZAP Scan Fails Because the Application Won’t Start

The sleep timer in the workflow might not be long enough for your app to initialize. Increase sleep 15 to sleep 30 or use the healthcheck loop shown in the full workflow above. If the app needs environment variables to start, pass them with a .env.ci file in the repository (with non-production values).

Wrapping Up

Four gates; none requiring an enterprise license — each one catches a different class of vulnerability at the point where it’s cheapest to fix.

The math isn’t complicated. A critical CVE found in CI takes minutes to fix. The same CVE found after a breach starts the clock on $4.88 million in average costs. The tooling here is free. The GitHub Actions minutes are cheap. The only thing the pipeline costs is the time required to set it up. Once.

One thing teams usually don’t anticipate: False-positive management is ongoing work. You will suppress some findings. You will review them when the expiry dates pass. That’s the point. Security tooling that runs without maintenance is security tooling that nobody trusts.

The complete working example — including the Node.js app, Dockerfile, docker compose and all four workflow files — is available at github-actions-security-gates.
August 4, 2026
Checkmarx Adds Hybrid SAST Engine to Improve AppSec in AI Era

Checkmarx this week revealed it has re-engineered the core engines embedded within its static application security testing (SAST) tools for the agentic artificial intelligence (AI) era.

At the core of that effort is a next-generation SAST hybrid scanning engine that combines three distinct capabilities within the Checkmarx One platform. An existing deterministic rules-based foundation is now being extended using a purpose-built large language model (LLM) and a Finding Analysis Engine (FAE) that suppresses false positives.

Frank Emery, director of product management for Checkmarx, said the SAST tools embedded in the Checkmarx One platform, as a result, now combine existing deterministic results for specific programming languages with the probabilistic insight generated by large language models (LLMs) that have been shown to be effective at discovering vulnerabilities. The challenge is that LLMs tend to generate a lot of false positives, which can now be sharply reduced using the FAE, he added.

The overall goal is to provide DevSecOps teams and application security professionals with the best of what traditional SAST tools and LLMs provide via a single platform that determines the degree to which a vulnerability is actually exploitable, said Emery.

Historically, DevSecOps teams have relied on SAST tools that are able to surface security issues that are specific to a programming language. Armed with those insights, application development teams were able to address vulnerabilities and weaknesses in source code before it was compiled. LLMs add an ability to discover additional issues after code has been compiled no matter what programming language has been used. Those two capabilities are now being combined in a way that promises to make it simpler for DevSecOps teams to improve the quality of the code that ultimately gets deployed in a production environment.

Additionally, those insights can also be shared with the dynamic application security testing (DAST) tools that Checkmarx provides to make it easier to identify the root cause of any security issue that arises after software is deployed in a production environment, noted Emery.

Those capabilities, collectively, are more crucial than ever in an era where AI coding tools are generating more vulnerabilities than ever. In fact, a recent Checkmarx survey finds 70% of respondents reporting they are also now discovering more vulnerabilities, with 31% describing that increase as being significant.

Mitch Ashley, vice president and practice lead for software lifecycle engineering at the Futurum Group, said layering a reasoning engine over deterministic scanning concedes that rule-based pattern-matching alone has reached its ceiling against AI-generated code. That structural shift moves the decisive layer from detection to reasoning about which findings actually warrant a developer’s attention, he added.

DevSecOps teams will now judge tools based on how well they reason about exploitability and remediation at machine speed, noted Ashley.

It’s not clear to what degree software engineering teams are revisiting DevSecOps workflows in the age of AI, but the one thing that is apparent is that existing processes are not up to the challenge, so it’s now more a question of how much change can be absorbed before there is a major incident that could have easily been avoided with a little additional foresight and planning.

June 16, 2026
Checkmarx Acquisition of Tromzo Accelerates Plan to Apply AI to Application Security

Checkmarx has revamped its strategy for infusing artificial intelligence (AI) agents into DevSecOps workflows following its acquisition of Tromzo, a provider of an application security posture management platform (ASPM).

Earlier this year, Checkmarx committed to building three classes of AI agents within a Checkmarx Assist portfolio, starting with a Developer Assist tool that it developed. The other two Policy Assist Agent and Insight Assist Agent will now be based on AI agents developed by Tromzo to apply AI.

Ori Bendet, vice president of product management for Checkmarx, said in effect the acquisition of Tromzo accelerates the timeline the company had set for building a suite of AI agents that automate application security workflows across the entire software development lifecycle (SDLC). Checkmarx also gains a cybersecurity team that has already developed a lot of expertise when it comes to applying AI to application security, he added.

Tromzo CEO Harshil Parikh said that as a provider of an ASPM, Tromzo is already using AI to understand the unique attributes of a codebase. Whenever an issue is discovered, its AI agents will then surface code recommendations for fixing it. That approach eliminates the need for a developer to spend time researching the vulnerability and then building a patch, said Parikh.

That capability will soon be extended using data collected by the static application security testing (SAST), dynamic application security testing (DAST) and software composition analysis (SCA) tools developed by Checkmarx, noted Parikh.

The acquisition of Tromzo comes at a time when DevSecOps teams continue to struggle. A recent Checkmarx survey finds nearly all (98%) organizations have experienced a security breach attributable to vulnerable code, with 81% acknowledging their organization has shipped code with known vulnerabilities into production environments. More than a quarter of organizations (27%) experienced four or more breaches due to these vulnerabilities.

Roughly a third of respondents are resigned to additional incidents occurring in the next 18 months, with software supply chain compromise (35%) topping the list followed by a third-party vendor/partner security incident (35%), cloud infrastructure misconfiguration (34%), insider threat or privileged access misuse (33%) and application programming interface (API) security breach or business logic attack (32%).

As the pace at which applications are being developed in the age of AI continues to accelerate, many of these issues will only be further exacerbated. The only way DevSecOps teams will be able to keep pace is to rely more on AI to discover and remediate issues, many of which will be created by developers relying on AI coding tools that generate more flawed code than ever.

Hopefully, there will come a day soon when AI coding tools generate higher-quality code, but in the meantime, DevSecOps teams should prepare for the worst. Cybercriminals are now also adopting AI tools with an eye toward not only discovering more vulnerabilities but also creating the code needed to exploit them in a matter of minutes. As such, the number of zero-day vulnerabilities that DevSecOps teams will be required to remediate as quickly as possible is about to dramatically increase.

December 18, 2025
Qwiet AI Extends Microsoft Support in Platform for Fixing Vulnerabilities

Qwiet AI today extended the reach of its application security platform that uses artificial intelligence (AI) agents to discover and remediate vulnerabilities in code to now provide deeper integrations with Azure DevOps, Azure Boards and GitHub platforms from Microsoft.

The company has also enhanced its support for webhook notifications, automated data export processes, expanded secrets configuration and made minor user interface (UI) enhancements.

Additionally, Quiet AI is also beta testing observability graphs and reports to provide deeper vulnerability trend analysis.

Finally, the AutoFix AI agents developed by Qwiet AI to remediate vulnerabilities have also been extended to add an ability to analyze data stored in a Static Analysis Results Interchange Format (SARIF) for non-Code Property Graph projects and policy support for code built using the Swift programming language used to build macOS applications.

Quiet AI CEO Stuart McClure said integration with the DevOps platforms from Microsoft comes at a time when they are gaining increased traction. Many application developers are increasingly relying on artificial intelligence (AI) coding tools from Microsoft to build software, he added.

The challenge is that much of that code can contain vulnerabilities because the AI models that were relied on by AI coding tools were trained using examples of flawed code. Unfortunately, too many developers put too much faith in the output of AI coding tools. In some cases, the code created by an AI coding tool might be more secure than the code an inexperienced developer might create, but there is no way to tell without scanning it. The only way to address that issue at scale is to make use of an application security platform such as Qwiet AI that has trained AI agents to both discover vulnerabilities and automatically remediate them, said McClure.

It’s not likely that concerns about vulnerabilities might be holding back adoption of AI coding tools so it’s probable the number of them that will find their way into production environments is only going to increase. The best way to proactively address that issue is to use AI to identify and remediate vulnerabilities as code is being written, added McClure.

In the case of Qwiet AI, that means using a platform that makes use of AI agents to run both static assessment security test (SAST) and software composition analysis to discover vulnerabilities that are then remediated using a series of AI agents that create, test and review changes that will be made to the code. In effect, Qwiet AI is making a case for using AI agents to compensate for the security flaws in AI coding tools.

Hopefully, the code being generated by AI coding tools will steadily improve as they are trained using examples of code that have been vetted for security flaws. In the meantime, however, DevSecOps teams should view any code generated by AI coding tools with at least the same level of suspicion as they would code written by human developers.

August 27, 2025
Legit Security Adds Dashboard to ASPM Platform to Improve DevSecOps

Legit Security this week added a dashboard to its application security posture management (ASPM) platform that makes it simpler to correlate the creation of a vulnerability to a specific application development team.

Company CTO Liav Caspi said the risk prevention dashboard also makes it possible to identify missing guardrails, such as a static application security testing (SAST) tool, that might not have been turned on as code moved through a DevSecOps pipeline.

These insights create a teaching opportunity using, for example, gamification capabilities built into a software-as-a-service (SaaS) platform that can be used to further adoption of best DevSecOps practices, he added.

Finally, the dashboard also makes it possible to keep track of all the vulnerabilities that a DevSecOps team has prevented, to better surface the actual return on investment (ROI) being provided by remediation efforts, noted Caspi.

ASPM platforms have emerged as centralized platforms that employ large language models (LLMs) and heuristics to identify vulnerabilities in code before they wind up finding their way into a production environment. The overall goal is to correctly identify access keys, passwords, application programming interface (API) keys and other personally identifiable information (PII) that hopefully will no longer be exposed to cybercriminals after an application is deployed, he noted.

That capability is critical at a time when the volume of code being generated is dramatically increasing because of the rise of AI. Unfortunately, the AI platforms that create this code were trained using often flawed examples of code scraped from across the web. As a result, it’s not uncommon for AI platforms to generate code that has known vulnerabilities.

Legit Security, in effect, is making a case for using AI to better identify vulnerabilities created by AI tools and humans alike in an era where cybercriminals are increasingly using the same capabilities to scan applications for those same vulnerabilities.

While a lot of progress has been made in terms of the adoption of best DevSecOps practices, there is clearly still much work to be done. A recent Futurum Research survey finds that over the next 12-18 months, organizations specifically plan to significantly increase software security spend on application programming interfaces (42%), DevOps toolchains (35%) incident response (34%) open source software (32%), software bill of materials (30%) and software composition analysis (27%) tools.

No application developer sets out to write insecure code, but expecting them to never make a mistake is unreasonable. It’s always possible through training to teach developers to write better code, but in the rush to build software, there will inevitably be times when code may not have been properly reviewed. The simple truth of the matter is that no developer has the cognitive ability to identify every vulnerability that might find its way into a code base, noted Caspi.

The one thing that is certain, however, is that it’s only a matter of time before a vulnerability is discovered in a production environment that is ultimately going to be more challenging and expensive to fix, as regulations continue to become more stringent.

March 29, 2025
Security Across the SDLC

We’ve seen some interesting movements in the security space over the last couple of years. My focus is on application security, but one of the movements we’ve seen is the blurring of what is application and what is infrastructure security, so I do a fair amount of that also. And then, with the advent of WAAP, we have network security playing a role in AppSec… It’s a growing snowball.

I will start with the mandatory happy declaration that DevOps no longer treats security as an annoyance. While we’re well past that, I think the shock of the whole “security is a problem, not protection” mentality that DevOps started with is still hanging with us, and some still actively believe that. But at this point, we’re well on the “Security is part of the process” trail we should be on.

If you look at the top-level changes, static application security testing (SAST) vendors have grown to be development security, covering SAST, DAST, IAST, RASP and even a bit of data security. Meanwhile, load balancers and WAF slowly evolved into full-on web applications and API security that covers the production side with active protections for things like DDoS and DLP.

Likewise, we’ve seen this with the growth of cloud workload protection and how the entire digital architecture is being pulled into CSPM.

I’ve said it before – we’re in a consolidation phase. It looks like it will go even further than I thought, though. Now, we are starting to see an interesting development: WAAP vendors are starting to include some or all of DAST, SAST and RASP. Arguably, they always did some IAST because their focus was on runtime protection, but their emphasis is more on protection than test, so it will be interesting to see that bit expand also.

And all of this includes an inconvenient fact. We don’t have any more people to do security than we ever did; in fact, we have less. That’s part of the reason that we haven’t shoved security everywhere – there aren’t people to do it. In true DevOps fashion, we have distributed the work across other teams – Dev is right there where code security and API security start, so we’ve implemented SAST and DAST bits right into the IDE, for example. Or Ops has always done some amount of security, and we’ve just upped the amount. But they don’t magically have more time, either. I’m sure you are feeling it if you are a practitioner; almost all of us are. AI promises to help with that, and in security, it will likely deliver. Another thing I’ve pointed out before is that security is one space that really does stand to gain from generative AI – in helping make more secure code, in helping make all the permutations of tests, in making rules based upon observed data and more. The time-saving benefits will deliver, in my opinion. But it’s still a lot of work, and some of it is straight-up additional work because we weren’t doing things like container security a few years ago.

In the short term, if full SDLC security is applied, there will be more work. In the long term, the mega platform for security may well reduce this work as it limits notifications across tools. Maybe we should name the coming mega platform SecArch. Also in the long term, AI will help alleviate some of the load. Until then, buckle in. You’re not likely to get staff to cover new work. But it will be worth it. Just keep plugging along, and know when to say you just can’t add one more thing to the pile.

January 31, 2024
2024: The Year of Testing

Caveat: I currently cover DevOps – including test – and security – including the *AST. This does give me a viewpoint that others may not have, and also potentially gives me blinders that others may not have. I am not generally a predictor but a reader of direction. Proclamations like this are rare for me and are shorter-term/more tactical than the grab bag of predictions usually seen this time of year.

I was thinking about this one this morning; the direction of travel of traditional and security testing is very similar at the moment. We are seeing the vast complexity of the overall space and the rapid growth of automation come together to once again make all testing a topic worthy of consideration.

I loved test-driven development (TDD). I can’t speak for others, but RPC/API is just a cool way to implement centralized functionality, and TDD is almost mandatory to develop APIs and the clients that call them. There is no UI to distract or have separate requirements in API development; there is a stated function and correct or incorrect behavior. To a computer scientist, this is predictable beauty. TDD made it more than predictable—it made it a trail through the woods. “We know it needs to do X, Y and Z, with the only side effects being A,” can be detailed with tests to prove that is what it does, and then look to prove the “It doesn’t have other side effects” part. And you can do it all while developing—no need for dev/test iterations. Like any IT solution, it isn’t a great fit for everything, but even if it isn’t called TDD, it is nearly impossible to develop API-based solutions without some level of TDD.

The thing is, TDD is absolutely a developer’s tool. Developers and DevOps pretty much must use it to turn out usable APIs. The test team only has a use for a subset of those tests. Because if the dev team did it right, they passed all the TDD tests, and the test team’s role in that scenario is to look closely at what the developers might have missed. Integration is often beyond the control of an individual dev and/or API, for example, because many different clients may be integrating with it. Test teams, in a true TDD shop, fill the gaps with meaningful tests that evaluate the overall application quality.

Security testing—at least static application security testing (SAST) and increasingly dynamic application security testing (DAST)—are also becoming a core part of the development process. This is as it always should have been. It is far easier to fix a security flaw (or any flaw) at the design/early code stage than in a testing process after deployment. The quality of security testing inputs is also increasing, making it far more viable. And finally, the tools are being effectively integrated. Being a developer who did a lot of security work because we often needed it and had no one to do it, I will say I get a little rush about being told right in my IDE about the security risks in my code. I write, I see, I revise where necessary, and I turn out better code. All without having to pause for security or bang heads over policy.

To date, all this testing is separate, but I suspect, having vaguely the same location in the process of development, that testing will all occur together and the markets may even start to merge a bit—though merger of security testing and functional/performance testing tools is a bit of a stretch, it is more possible than in the past. In most orgs, even though developers have an increasing stake in both sets of testing, the target purchasers are still two different groups—security versus test—with different priorities. That’s why I think they aren’t likely to merge any time soon.

While AI in most technology spaces is currently experiencing the same level of FUD as XML, Java, JavaScript or low-code was in the past, all super-hyped tools in tech do have a sweet spot. Security and testing are in that sweet spot for AI. Both in test generation and in results evaluation, it does it better, faster and with more coverage while giving staff a concrete set of results to pore over and improve security posture and/or code quality. I cannot say loudly enough that in terms of enabling the steps in the SDLC that have suffered from a lack of resources and timeline to write, execute and evaluate tests, AI is a game-changer.

So, all of these things together mean that 2024 is a good time to take a hard look at your testing environment(s) and architecture, both traditional testing and security testing. Now that AI has made advanced automation a fact, it is time to consider implementing the level of testing we always knew we should have but never had the resources. Build it right into the build process and make it part of the organizational DNA—just like Agile is.

If you could hire a specialist who would paw through your code and give you a list of the 25 things you need to fix first, you would do it. That is, essentially, what modern testing tools are. They can do more tests, evaluate more issues and side effects and filter down to the “No—this is really important!” list without costing the team a ton of time.

And keep rocking it. We’re all sitting on bugs and vulnerabilities. I do hope we are way past denying that fact. Apps are big old complex monsters; these days, sometimes built from generations of technologies cobbled together. Of course they have issues. Even yours. So let’s take steps to clean it up, just a little bit. And keep clipping along.

December 6, 2023
The Security Pipeline

Over the last few years, the ability to secure our applications has grown, and deep integration into the DevOps toolchain has, too. There are more tools doing more security checks protecting more of the infrastructure and source than there have ever been. The key is putting them to use intelligently.

We now have the ability to secure the image an app is based upon, the images that are derived from that original—complete with supporting software. We can get feeds to tell us which bits of our included open source components are insecure and how to protect them; we can scan the source, the assembled application, the APIs and the network. We can put runtime protections in place in front of web apps and in front of APIs. We can even track the long-tail attacks like slow data extraction that are becoming more common as security tools master more obvious attacks.

That is a great set of security capabilities. And automation puts every bit of it within reach. Your job is to pick the bits you need to increase the security posture of your environment and get them worked into your specific toolchain. This is not all that difficult, but nothing is free, so you will need at least some implementation time and some staff time to work through results. After all, no matter how automated, current solutions do end with a list of changes that need to be made to increase security. Some tools will put a runtime fix in place immediately, but honestly, that is a patch—it only stays in place until you can fix the underlying issue.

Integrating all of the great security tooling with each other is another aspect that will require some time. Some vendors cover a massive amount of the security testing needed across markets and technologies. These companies show us that even if an organization would rather use best-of-breed, there is a lot of benefit in increasing integrations. For an easy example, consider this: If the software bill of materials (SBOM) generated during the build steps shows that a given framework is included in an application, then active testing via dynamic or interactive application security testing can include extra tests specific to that library.

Integrations will take effort, but once they’re in place they will enhance application security. There is no perfectly secure application, but the tooling is available to make a given environment less of a target because it is too much work to attack. And that is what you should be shooting for.

I’ve said before that making security part and parcel of your DevOps practices is mandatory at this point, but that was in reference to the great DevSecOps toolsets currently out there. I’m suggesting now that we include it all–from design to GitOps runtime drift management. Because we can, and the result is reduced risk to the organization.

And keep rocking it. Honestly, you’re maintaining something worth protecting–that alone means you are successful, even if you aren’t aware of every issue that crops up. Issues are why you’re there, and the systems keep running because you are.

June 21, 2023