A dependency scan warns of a vulnerable library several hours before the release. The developer finds a solution, but it might affect some working features, and security and operations teams wait for approval to take any action. The end-of-line race against the clock to finalize releases is well known to teams that postpone security decisions to the final hours of a release. DevSecOps teams can streamline such processes by establishing protocols and providing feedback as the work progresses. This article discusses the location and responsibilities involved in making such determinations and decisions and examines approaches to negotiating releases in the presence of an unresolved security problem.
Why the Final Security Gate Slows Everyone Down
It is a difficult situation, however, to discover that a design choice needs to be changed. Late findings also raise new questions that should have been asked earlier. Should the developer be expected to investigate every scanner finding? Can the change be approved by a security team in lieu of a code change? Who decides if an unresolved issue is acceptable?
Formal approval is still required for a number of changes, especially those relating to security-sensitive systems or significant risks. The process of release security reviews is most effective when the criteria are known ahead of time and problems are identified with enough lead time to address them.
What a Security Partner Does Differently
Working with security as a partner starts before your code is reviewed. Instead of waiting to look through polished code, security engineers can help your team make decisions at three critical points during the development life cycle:
- Before Development: Decide what and where requires protection, and which proposed changes need extra attention.
- During Development: Bring relevant and actionable guidance and controls into the tools that developers use daily.
- At the Point of Finding: Discuss the risk and determine how to remediate and mitigate it.
This requires an agreed process for getting help when a decision has security consequences.
Put Clear Guardrails in Place
Security guardrails provide teams with a clear set of behaviors that help govern common, known risks, such as approved sources for dependencies, secret scanning on code commits, restricted access to production credentials and required reviews for all changes to authentication.
Make Failed Checks Understandable
A guardrail should describe the result of a failed check. A message that identifies the exposed file and the action needed is better than a red build and a scanner code.
Keep Ownership With the Team Building the Software
Developers tend to know what to do to make changes to their application in a safe way. There may need to be an owner of a service or a leader who takes ownership of any risk that remains after analysis. Creating this documentation helps to share the security responsibility when it comes to making a release decision if there is no time to wait.
Build Security Into the Delivery Pipeline
Enhancing a CI/CD pipeline with security means introducing appropriate checkpoints at which the team can take action.
| Stage | Useful Check | What the Team Needs From the Result |
| Pull Request | Secret, dependency and configuration checks | The affected file, the concern and a possible next step |
| Build | Checks on the application or container image | A result tied to the version being prepared for release |
| Before Deployment | Review of critical findings and open exceptions | A clear decision, owner and recorded reason |
At the Pull Request
A pull request is a useful point to detect committed secrets, inspect new dependencies and flag risky configuration changes. This keeps developer security workflows close to the change that produced the finding.
Automated security testing helps find potential issues early, but a scanner cannot always determine whether a vulnerable feature is used or exposed in a particular application.
Before Deployment
A later stage may examine the built application, the image of the application or the deployment configuration. Within the framework application, the team must discuss significant results and agree on the process for approving deviations, assigning responsibilities and setting time constraints. Even if the team detects a low-priority finding and has a corrective action plan, such a result cannot be classified as an incident at the same level as a production credential.
Example of a Vulnerable Dependency
Consider the vulnerable library from the initial example. The developer determines the version that is currently installed and whether the application uses the potentially harmful method. Security tools can help investigate the issue further. If an update is available, it is applied, and the tests are rerun.
If it is not possible to update the package immediately due to the risk of breaking the code, the preferred mitigation and its approval date, based on the company’s standards, can be chosen. The solution should be included with the release, and it should state the problem and how it was addressed.
Make Findings Easier to act On
A long list of alerts leaves developers to work out which ones matter. Vulnerability triage should provide more than a severity label. A useful finding answers four questions:
- Which components and systems are affected?
- Can the vulnerable behavior be reached?
- What could happen if the issue is exploited?
- Is there a fix or practical mitigation?
Teams can reduce noise by removing duplicate alerts and reviewing results that consistently prove irrelevant. They should also provide a way to question a finding without silently dismissing it. Short security feedback loops let developers supply application context and let security revise an assessment when the evidence changes.
Define When a Release Must Stop
A newly discovered production secret may have to be kept on ice until the credential is revoked and replaced. An actively exploitable issue in an internet-facing capability may justify stopping deployment.
A lower-impact issue in code that is currently inaccessible might be triaged as something with an owner and a due date instead.
Work Together Before an Incident
The phrase ‘shift left’ in security refers to addressing security concerns earlier in development. In practice, that can be a short design conversation rather than another lengthy approval meeting.
Imagine that a team wants to change the login flow. Suppose that one of the security engineers asks during the code review what protects the system from multiple recoveries for the same account. The question can prompt developers to examine the new design and understand that this scenario was not considered at all. Or maybe some protection will be added to this flow before implementation begins.
Turn Incident Lessons Into Specific Changes
Incident reviews can be another option for improving guardrails. If something went wrong earlier because of a service permission that was too broad, the team could request a specific review of similar permission changes. Developer and security collaboration is most effective when a conversation about the issue leads directly to a decision, an owner or a specific change to the process.
Measure Whether the Partnership Works
A final report on DevSecOps tool use says little about the evolution of security, as it focuses on quantity rather than impact. Fewer blocked releases can be explained by catching issues early, but they could also be explained by becoming less thorough in checking for issues.
However, some metrics are more telling than others, as they can highlight areas that need attention, such as:
- Time to Feedback: Are devs getting the problem information while the change is still relevant?
- Time to Remediation of Critical Issues: Are critical problems being left open too long?
- Repeat Issues: Do the same things tend to get broken in several services?
- Open Exceptions: Are they properly managed with an owner, a justification and a deadline?
Delayed feedback can be a sign of either a slow test or a lack of clarity about the review process. Practices that support DevSecOps initiatives are only helpful to the degree that they actually address such concerns rather than just adding another layer of complexity and reporting.
Conclusion
The vulnerable library that the team discovered on release day still needs a major decision. Defined requirements, ongoing reviews, relevant insights and designated decision-makers empower DevSecOps teams to enable developers at every stage of delivery.
For organizations that want to achieve secure software delivery, the first priority should be identifying one common source of delays: When did the risk first appear, and what could have helped the team address it earlier?