devopsy.kinsta.cloud

Tag: security

How Application Developer Turnover is Shaping Security

Observing the developer job market for the past two years reveals a surprising truth that has a profound impact on technology organizations. In the “new normal” of hybrid work, life moves fast and there’s no sign of slowing down. Since 2021, we’ve witnessed all-time record demand for software engineering developers. The emergence of privately held large and medium technology startups, fueled by a fear of missing out (FOMO) driven venture capital market, created vast growth opportunities for software engineer developers in particular. From developing and deploying new applications to scaling your business through multi-cloud environments, cloud and cloud-native companies are everywhere. Not only are cloud and cloud-native companies receiving enormous funding rounds, but worldwide spending on public cloud services is forecasted to grow 20.7% to a total of $591.8 billion in 2023.

This dramatic cloud growth is primarily driven by the warp speed developers are operating. Thanks to developers, every company is now a digital company and every industry is reaping the benefits of a cloud presence. A retail organization can now build and deploy an application for the holiday shopping season within hours, and workforces can access applications from their kitchen, local coffee shop, or halfway across the world.

However, as with any new, widely adopted technology, cybercriminals look to target and exploit organizations in their early cloud days as they begin to grasp and understand the technology.

It was reported earlier this year that the recent exposure of roughly a terabyte of Pentagon emails was likely due to a cloud configuration error. Cloud attacks are on the rise, and there is a major disconnect about who is responsible for keeping the organization safe. Developers who are told to build with speed don’t want their applications slowed down or user experience to be altered by security protocols. Still, accelerating application velocity has security teams caught between remaining responsive to application teams and securing an increasingly complex cloud environment. A major problem is bubbling up, and if security and developers teams cannot come together, the result can leave organizations vulnerable.

Developers and Turnover

In a recent survey, 75% of respondents reported a higher-than-usual turnover rate in DevOps. Similar to the well-documented cybersecurity talent shortage, multiple workplace stressors appear to be taking a toll on developers. The same survey reported that 38% of enterprises deploy code to production or release to end users every day, with 17% deploying multiple times a day. To put this into perspective, just two decades ago, developers were shipping limited updates one to two times a year.

Given the pressure to deliver new code every day, combined with security teams’ pressure to ensure that code is secure, it’s no surprise that we’re witnessing higher-than-normal developer attrition. Instead of pointing fingers at developers for unsecure code, security teams need to create tools and training that do not impede their number one priority: Shipping new code. We’re in a developer-friendly job market and organizations cannot afford to lose top talent because they are overburdening them with security tasks. Developers will find greener pastures.

Shifting Left

By now, everyone in the industry has heard of shifting security left—the practice of moving security to the earliest possible point in the development process to ensure code is secure and limit timely and costly remediation. In practice, this means bringing security to where the developers operate so vulnerabilities and misconfigurations are caught early when they are easy to fix.

This approach has brought about a deeper level of engagement between application developers and security teams, especially in the design phase. However, while developers might now be embracing tools and processes that help them code secure applications, there’s still a limit to how much shift left responsibility they can and want to handle.

Path Forward

Shifting security left is certainly a recommended starting point, but simply equipping developers with DevSecOps tools isn’t a winning approach. Developers must be trained and have the desire to effectively use the tools. Like the old saying, give a man a fish, and you feed him for a day. Teach a man to fish, and you feed him for a lifetime. The same holds when shifting security left.

To ensure developers and security teams understand their security responsibilities within the organization, consider implementing the following:

1. Define a clear shift left strategy with developers and security teams
2. Provision the necessary security tools that effectively protect the organization but don’t create roadblocks for developers. Additionally, more tools don’t necessarily mean more security, so be strategic.
3. Train developers to be more mindful of cybersecurity issues. Most developers haven’t had formal cybersecurity training, so it’s up to organizations to make sure that particular skills gap is closed

DevSecOps is as much about changing culture and approaches as it is about acquiring new tools. Developers will, unfortunately, come and go, but it should never be because security teams are overburdening them.

April 14, 2023
Awareness of Software Supply Chain Security Issues Improves

A global survey of 167 software professionals suggested that, while there is a lot more awareness of application security issues, the adoption of DevSecOps best practices is still not pervasive.

The survey, conducted by Chainguard, the Eclipse Foundation, the Rust Foundation and the Open Source Security Foundation (OpenSSF), found that just over half (51%) of respondents are either concerned or are extremely concerned about software supply chain security.

In addition, only a third of respondents (33%) reported they are familiar with the Supply-chain Levels for Software Artifacts (SLSA) framework for building more secure applications. Well over a third (36%), however, said they had never heard of it.

Only a quarter of respondents (25%) reported that their team always signed built artifacts, the survey found.

The survey identified the most challenging issues developers are encountering and found they are reproducing builds (64%) followed by creating hermetic builds that are isolated from other software libraries (62%).

Mikaël Barbero, head of cybersecurity for the Eclipse Foundation, said while awareness of software supply chain issues is higher, it’s apparent that other priorities are still getting in the way of adoption of DevSecOps best practices. For example, software bills of materials (SBOMs) may generate a lot of headlines, but their adoption remains mixed, he added.

John Speed Meyers, a security data scientist for Chainguard, noted that incentives still need to change; too many managers that hire developers still emphasize performance and delivery times over application security. Many developers, as a result, still tend to view cybersecurity as an obstacle to be overcome or worked around rather than a core capability that they will be graded on, he added.

In theory, as responsibility for application security shifts left toward developers, overall cybersecurity will improve. Most developers, however, have little to no cybersecurity training, so the odds they will be able to resolve these issues are low. Instead, many organizations are either trying to embed cybersecurity professionals within DevOps workflows or, at the very least, deputize at least one developer to advocate for cybersecurity within those teams.

Regardless of commitment level, it’s apparent that organizations need to address everything from locking down developer environments to thwart phishing attacks to ensuring application builds are not inadvertently infected with malware. The challenge organizations face is both technical and cultural, so the amount of effort required should not be underestimated.

In the meantime, organizations should expect to see cybercriminals increase their attempts to compromise software supply chains. Infecting malware into software components that will be used by multiple downstream applications is, from their perspective, one of the most efficient means of wreaking havoc. Right now, organizations make it too easy for cybercriminals to compromise application environments.

Organizations should also recognize that tolerance for insecure applications is also declining, as evidenced by the National Cybersecurity Strategy put forward by the Biden administration. It’s not so much a question of whether organizations that create software will be held liable for application security issues as much as it is to what degree.

March 21, 2023
Modern DevOps is a Chance to Make Security Part of the Process
I’ve mentioned before, and many of you have lived through, the slowly changing beliefs around DevOps versus security. We are past the days of “Security slows us down” and into “How can we bake security into the development process?” which is essential. Indeed, I would say it’s more essential every day as the number of threat actors and their use of automation to perform more attacks increases.

So where can we slip in a little security to protect those sweet, innocent apps from the Big Bad Wolf? Let’s make a list, then chat about it.
- Source code
- Inclusions–libraries, shared objects, etc.
- Container/cloud images
- Network
- Running applications and APIs
This is honestly not an exhaustive or detailed list, but we’re writing a blog here, so let’s stop there so we have time to discuss.

Source code security is greatly enhanced by putting in any of a dozen development security tools, which include static and dynamic security scanners along with a variety of other features.

Inclusions are often nested and confusing. A mod that developers include relies on other mods and libraries—sometimes so many that the developer can’t even list them all. So, this type of security starts with software composition analysis (SCA), then uses the list that generates to pursue vulnerabilities—normally by looking for the included libraries in a vulnerability database. Several prominent source code security solutions bundle at least the SCA part, and some also help nail down included vulnerabilities. Check with your vendor if you’re just getting started here.

Container images are akin to the “golden images” of old. Build one that has what most applications need, add on the security that an organization wants most applications to support and then have every project start there and only add what this specific app needs. That allows for scanning of the base image so all applications start out at least somewhat secure. Cloud images can be done much the same, but beginning at a higher level. Look into registry security or image scanning tools to meet this need. It is worth asking your source scanning vendor if they offer this functionality; some do.

The network on which an application will be exposed needs to be locked down. While the access points into the org probably are protected, portability of applications means that the application itself should be locked down wherever it uses that network. Tools are specific to the platform and networking in use, so look into it—I can’t offer much more generic advice.

Security has long tried to lock down running applications—generally with pushback from developers and even ops, a bit. Between web application firewalls (WAFs) and API security, we have the ability to stop a whole host of attacks on their way in … But you have to use WAFs and API security best practices. If you’re starting from scratch, consider tools in the application and API security space that protect against both the web UI and API attacks with a breadth of technologies. For those with a WAF, look at API security tools that protect APIs. For those with API security covered, consider a next-generation WAF.

My usual encouragement applies. You are rocking it, and all of these tools are increasingly integrated into the DevOps toolchain, so it’s worth the time investment to get them set up and let them run. If you’re just starting out, start with source code security. Honestly. It will make the overall app more secure by tightening its source. But don’t ignore the ops side, as that is part of the attack surface. Protect those apps you built and/or maintain that are keeping the org running.
March 15, 2023
DevSecOps Provides a Modern Security Model for Modernization

Developers and security experts are now tasked with bolstering, extending and adjusting cloud and Kubernetes security to protect against cyberattacks that are ever more complex, volatile, and frequent. To foil attacks and create a secure foundation for applications and infrastructure from the beginning, DevSecOps (development, security and operations) has become the trending development and operations practice. In the DevSecOps model, security becomes a shared responsibility. Thus, DevSecOps requires a mindset shift to enable collaboration between development, security and operations teams. This shift, commonly referred to as shift left, involves culture, processes and tools. With DevSecOps, whether it’s testing for security vulnerabilities or building security services that can be used directly for business purposes, the goal for all participants is to build security into applications from the start and throughout the continuous integration and continuous delivery (CI/CD) workflow of DevOps.

DevSecOps to the Rescue

In the past, security considerations and practices were often introduced at the end of the development cycle by a separate security team and tested by a separate (QA) team. This was manageable when software updates were released every few months or even years. But in today’s landscape, consumer habits and expectations are shaped by smartphones and digital commerce, which has fueled the demand for software services that are real-time and available 24/7. This requires modern enterprises to find ways to improve the efficiency of application developments, releases, and updates.

Because of these factors, enterprises realize that security can no longer be a small part of an afterthought in the production environment. The purpose of DevSecOps is to ensure that through collaboration, security can be carried through to every point in the development cycle. This enables teams and enterprises to deliver secure, high-quality applications in a more efficient manner without extensive security checks and fixes occurring during post-production.

To meet these more extreme demands, cloud computing, containers, and microservices have made it possible to accelerate the development and delivery of software releases. Developers adopting agile and DevOps practices can reduce software development cycles to days and weeks, thus meeting the diverse needs of enterprises and users.

This fast-paced development and upgrade frequency has created new security concerns and the need for companies to be more agile in responding to security issues. DevSecOps has been introduced into the DevOps framework to meet these needs by make security a shared responsibility. Today, continuous testing and integration, including security scanning of pipelines, is becoming the norm.

DevSecOps Business Benefits

In principle, DevSecOps shares the DevOps ideal of multiple teams working together to improve team efficiency and achieve secure, continuous software delivery.

In addition to the security benefits DevSecOps provides, there are significant business advantages to be gained, including:
● Efficiency–Under the DevSecOps practice, security is integrated into all periods of development to help all teams be more agile in responding to security risks, eliminating the need for teams to spend a lot of time tweaking and fixing during the production cycle.
● Cost reduction–By discovering security vulnerabilities before they enter production, organizations and teams can significantly reduce the time and labor costs of fixing them.
● Ensure compliance–DevSecOps can ensure compliance with industry-standard regulations, such as the General Data Protection Regulation (GDPR). DevSecOps gives teams a holistic overview of these measures that makes compliance easier.
● Establishes collaborative culture–Integrating security practices into DevOps enhances the value of DevOps and improves the overall security posture as a culture of shared responsibility. When everyone is involved in the process, it increases their awareness of security fundamentals and best practices and provides a sense of ownership in the results.

Meeting New DevOps Challenges

Kubernetes offers many advantages but also poses unique security challenges that can be difficult to address for organizations lacking in Kubernetes talent and experience. This is why organizations will increasingly see the need to reevaluate their security practices and prioritize a more advanced security-focused culture.

Because DevSecOps requires security to be addressed throughout all development stages, it requires developers to have security expertise while coding and operating. We are currently seeing a growing skills gap in the DevOps industry, and developers are feeling burned out. Security training is a way for teams adopting DevSecOps to acquire additional knowledge.

Deploying Kubernetes platforms with security built in by default also will be recognized as a means to reduce the burden of security on IT teams. This will reduce the pressure and burnout on all sides.

Platform Engineering Provides Additional Help

With the rapid development of Kubernetes and cloud-native applications, organizations are realizing the inadequacies of their IT teams to leverage DevOps practices. Just as platform engineering eases the burden of DevOps by providing an Internal Development Platform (IDP) that serves as a “golden path” for developers, an IDP can simplify the practice of DevSecOps.

We’ve seen that a DevOps workload is difficult to practice in small and medium-sized enterprises, as well as in large enterprises that lack sufficient talent. The gradually accumulated cognitive load ultimately leads to a less agile and efficient collaboration between teams. Given these issues, more organizations will adopt platform engineering as a better alternative.

March 9, 2023
FAA Ground Stop due to Technical Debt? | Don’t Do DIY Crypto!

Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

This week: The FAA’s NOTAM database gets corrupted, and Threema shows why DIY cryptography is bad. (more…)

January 12, 2023
Prioritizing Product Security With DevSecOps

Building software with strong security can no longer be an afterthought for organizations. The need for a reliable cybersecurity posture has proven vital amid the constant attacks we’re seeing across industries, all over the world. Product-specific security is top of mind for most companies, and a successful product security practice requires integrating with the ethos of the engineering team—DevSecOps. Consider a three-fold strategy, which has proven successful again and again in the industry.

It consists of first creating a culture of security, with all stakeholders understanding the priorities, strategies, and business value. Second, teams must integrate security with the software development life cycle (SDLC). It’s not enough to shift left–organizations must shift their thinking from shift left to shift everywhere. And lastly, monitoring product systems at all times is non-negotiable. In a 24/7/365 world, there must be teams to watch over security 24/7/365. Part of the engineering deliverable becomes providing data to these teams so that they can have all eyes on the products, always.

A Successful Practice Starts With the Culture

In building a strong culture, the goal is to make the entire engineering team a part of the mission. This includes putting a product security team in place to support engineering needs as well as providing continuous education with resources available for addressing any eventuality.

While every organization has its own unique needs, considerations and approaches to creating a product security team, these five roles are highly recommended:

1. Technical Product Manager (TPM): Responsible for driving security priorities in planning
2. Security Architect: Sits on the Architecture Review Board and ensures architecture and design have security processes incorporated
3. AppSec Engineer: Responsible for implementing the security development life cycle
4. Adversarial Engineer: Hammers on the attack surfaces to identify weaknesses
5. Security Champions: Engineers from each Scrum team with an interest in supporting their teams with security fundamentals; they are instrumental in scaling out the overall product security effort

Creating and executing an engaging security education program is also pivotal to winning over the hearts and minds of employees. Consider initiatives such as in-depth security boot camps, vulnerability-of-the-month presentations, capture-the-flag (CTF) contests and security consultation programs.

Integrating Security Into the SDLC

From a high level, all SDLCs consist of planning, development and release, and each phase should have its own set of security controls.

Planning: The product security TPM is responsible for working with the product management team to determine security priorities–derived from sources like risk reduction, executive priorities, customer requests, security findings and regulatory requirements. Determining what is most important is key.

Development: Much of the heavy product security lifting comes during the development phase, which includes implementing threat modeling, security baselines and automated scanning.

Threat modeling: The process of identifying potential threats in a system. Once identified, the threat modeling team can create mitigations, and results are integrated into penetration testing to verify that the threats did not make it into implementation. It’s most efficient to test models during the architecture phase before coding begins to avoid rework and Band-Aids. For example, a company may have a policy disallowing access to port 22 from the Internet. Threat modeling automation can detect violations in the design and notify the owning team asking them to close the port. Tools that support this use case include IriusRisk and ThreatModeler, as well as Microsoft Threat Modeling Tool, which is a free option but does not include automation.

Security Baselines: Teams should create secured images that are fully patched, meet security policies and have mandatory security agents preinstalled. Doing this for both OS images and containers allows developers to get a level of security for “free” and avoid ongoing security reviews. Baselines should automatically update and flow into CI/CD systems.

Automated scanning: To keep up with Agile processes, scanning should be automated and have a low number of false positives. In fact, it is better to report on fewer issues than kill credibility with a lot of noise. Continuous tuning will increase the fidelity rule sets and helpful tools include static application security testing (SAST), software composition analysis (SCA), and dynamic application security testing (DAST). Automation is critical to securing applications that use a CI/CD development process and providing security right at the code push is invaluable–developers get immediate feedback, not a ticket weeks later when they likely have forgotten important context. Integrating these tools into build pipelines is a solved problem.

For example, the below YAML demonstrates automating source scans for embedded secrets (e.g., passwords and keys).

sast:secrets-detection:
extends: .sast:secrets-detection
stage: deployment:development

Release: When teams reach the release phase, adversarial testing becomes an important component. Penetration testing can be in-house, outsourced or both. An internal red team will have a better understanding of the technology and, therefore, have a broader scope than external security partners. For example, they test across product boundaries, test processes and people and test live production systems. For a mature product, consider a bug bounty program. This will invite white hat hackers to find weaknesses in exchange for cash. But “mature products” is essential; otherwise, the number of payouts will become overwhelming.

24/7 Monitoring

As mentioned earlier, 24/7 monitoring is key, especially for any cloud-based technology. A dedicated security operations center (SOC) can help, scrutinizing for threats against the company, employees and customers. The SOC’s responsibilities are much broader than those of a product security team, and they allow product security folks to sleep well at night. Security teams should configure systems, load balancers, public cloud platforms and networks to feed data directly into the security information and event management (SIEM) system. The SOC can then respond to custom automated alerts that identify any data anomalies that are being streamed to them.

As products–and technology overall–become more advanced, the way organizations secure them must do so as well. The perfect security approach varies from business to business, but empowering IT, engineering and security teams is a beneficial first step to identifying, optimizing and integrating security for both employees and customers.

December 14, 2022
Implementing Shift Left Security in the Cloud

While ransomware has been the leading concern for enterprise security teams over the few past years, software vulnerabilities are nipping at its heels. The boom in cloud-based apps and services and increased digitization of work have been a boon for hackers, who are taking advantage of developers’ and DevOps teams’ attempts to work faster and smarter to keep up with demand. One estimate says that four out of 10 zero-day attacks carried out in the last decade happened in 2021 alone.

Many things account for this increase. Developers are stretched and are reusing code, which allows for misconfigurations and vulnerabilities to reappear unexpectedly in different programs, and the use of multiple cloud services fragments security measures and reduces visibility into the code running many enterprise functions. This is why developers and security professionals alike are paying more attention to security throughout the software development life cycle (SDLC), particularly in the early stages.

Shift Left Security Principles and Challenges

The zero-day surge has led to an increased interest in shift left practices as a way to make security a priority in the development process. Shift left culture brings security into the equation much earlier in the software life cycle, before the software is deployed, rather than patch bugs after users report them. This preemptive approach helps head off vulnerabilities that can affect an application’s security posture unbeknown to its defenders.

Shift left principles can also enhance security when developers build applications for cloud platforms—such as Amazon Web Services, Microsoft’s Azure or Google Cloud—where visibility into the proprietary code and security tools of the platform can be limited. In a shift left culture, DevOps embeds least privilege policies as part of the daily work on cloud workloads, to protect network infrastructure and avoid granting excess permissions on those workflows.

For example, setting up role-based access control (RBAC) on Kubernetes containers enforces a least privilege model on those clusters and avoids excessive permissions that can lead to a breach, while removing admin credentials from continuous integration/continuous delivery (CI/CD) workstations shuts off hackers from using those pipelines as an attack vector.

Shift left security is a great concept, but runs into hurdles in execution, mainly the internal conflict between development teams who want to move at the speed of business while security staff takes a more cautious approach. This leads to friction, with developers warning about access denials affecting their work and causing deployment delays, while the security analysts worry about data breaches caused by too many users with unchecked admin privileges.

Working out this friction requires a mindset shift, where DevOps understands that security is part of the development process, and management can trust that developers have security in hand. This also requires tools and training to integrate shift left practices, such as platforms that can bridge the security needs of the CISO (by granting greater visibility into the network infrastructure code, for example) and the agility needs of the DevOps teams (by automating security guardrails and entitlements, for example). The ultimate goal is creating what is known as a DevSecOps model, bringing together development, security and operations into one agile, safe and efficient workflow.

How to Adopt Shift Left Security on the Cloud

Four best practices can help:

● Gain deep visibility: With enterprises often working in multiple cloud environments—and mixing public and private clouds into hybrid infrastructures—visibility into assets typically suffers, which makes getting a clear view of risk difficult. Being able to discover all identities, permissions, configurations and resources in the environment and all the access paths to specific resources can help build a contextual inventory of cloud assets to better manage them and carry out policy analysis.

● Prioritize risk: Risk management is a key practice of any security program, but the flood of alerts from multiple cloud providers and systems can overwhelm security operations centers, which are often short on talent and time. Automation tools can now search and prioritize risks across an entire cloud environment, from development to production, spotting problem combinations and alleviating the workload of sorting manually through alerts across multiple silos, freeing the staff to work on higher-priority mitigation and remediation tasks.

● Just-in-time access: Zero-trust architecture has become the standard to aspire for most security defenses, but it is often challenged by the difficulties of managing access privileges. Just-in-Time (JIT) access, granting limited-time revocable rights, is a useful tool for building a zero-trust security architecture and enforcing least privilege policies to maintain it. Technology tools that actively manage and monitor developer access to cloud environments, including a strong audit trail of privileged activity, give security a means to enforce JIT access and zero-trust in turn.

● Find and prevent vulnerabilities and misconfigurations: Infrastructure-as-code (IaC) has made software deployment in the cloud (much more agile and efficient with the use of virtual machines, containers, microservices, etc. However, security is often an afterthought both pre- and post-production when standing up cloud infrastructure. Organizations should scan code for security policy violations and other risks in order to mitigate flaws before they reach production and to detect and fix problems in production environments. With feedback and automated guardrails built into development workflows, DevOps and security teams can reduce security risks in their code.

Like zero-trust, shift left is on its way to becoming a standard practice for cybersecurity assurance. But building a DevSecOps model takes more than good intentions. It requires practices, tools and training to execute that model, and a mindset of partnership across the enterprise and with outside cloud and security partners. Only a group effort will provide the cloud security assurance an agile business needs.

November 30, 2022
Faster and Better Testing?

The whole concept of Agile and DevOps was to iterate development faster and deliver results in a more timely manner. As we learned more about both methodologies, processes and policies were put into place that improved the quality of what was created. Early ideas of quality like, “We can just do another iteration,” still exist, but are largely not used. Rollbacks are a thing in most orgs, and though the environment is more complex, rollbacks are easier to manage in the smaller development changes that iterations generally represent.

And once we got past the whole, “Anything that slows the process down is bad!” mentality that permeated early implementations of both DevOps and Agile development, we started seeing amazing things. But including things like security and functional testing were not viable in the fast-iteration world that Agile and DevOps created. And they really weren’t viable at most orgs in the slow-iteration world of waterfall, either. Staffing and time constraints made them fall far short of optimal in the vast majority of organizations. But putting them into the DevOps process challenged vendors to find a way to do things faster and challenged organizations to look at quality and security a little differently.

And now, we have tools that can test—be it functional, integration or security—at the speed of DevOps. Tools that enable Agile development with an eye toward security. A far larger number of developers now acknowledge their part in application security and are creating more secure code.

Keep an eye on how AI/ML is being used in testing—all forms of testing—and how it evolves in the next year or two. The same with full-on automation. I’ve made no secret that I view AI/ML in the testing space as advanced automation because of what it enables, so maybe “Watch what AI/ML-enabled automation is available,” is more accurate. The ability to create, run, process results and even modify code to resolve issues are all out there today, and getting better. Faster, more accurate, more automated.

Much like back when intrusion detection became intrusion prevention, I would not just turn it on and walk away, but think about the ability to upskill developers; “Here is a list of things the automated system found, and a list of fixes it proposes.” They are literally being shown how to do their craft better.

I’ve been a developer, so I’m not being offensive, here. If you get something that tells you, “This is an issue; here is why it happened and here is what to do to fix it,”take the opportunity to get better at coding. Look closely at the what, the why and how to fix it and incorporate that into your work.

And keep rocking it. We’ll be making more stable and secure code, but you’ve kept systems running without that feedback and fixed the errors along the way. This just makes it easier to find the errors early—before they crash production and make a fire drill.

November 30, 2022
5 Tips for Securing DevOps: What You Wish You Knew Sooner

Foundations and frameworks, concrete and steel—not exciting. But that’s the foundation and framing of pretty much every modern building. Everything else that is part of a building–flooring, wiring, lighting, room placement and so on—is made possible by the foundation and the framework. If you want DevOps to succeed in your organization, you have to start by building a solid foundation and a framework, including security. What’s below is a suggested approach to the concrete and steel of DevOps.

Three Foundational Needs

1. Leadership and Governance
Someone has to make the decisions, be responsible for leading the team(s), give direction and provide operational governance. Securing DevOps is not just about compliance with a set of security concepts; it’s also about setting the right tone for the workplace. Leadership and governance are key enablers of development success.

The pressure to deploy software faster and more often is real—but so are the security, regulatory and many other concerns around automation that may uncover bugs or defects in software before it gets pushed to production. In an environment where there’s pressure to release code faster, how does one balance these competing priorities? What’s more important: Speed or quality? This tension between speed and quality remains one of the main challenges faced by organizations adopting DevOps practices today.

Organizational leadership and governance regarding people and data must be in place to keep people working on the right things, keep up morale, provide direction about project prioritization and budget the right tools for getting the job done.

An important subcategory of governance is data classification. Where does the data reside? Over which channels does it flow? What kind of data is it? Classifying the data has long-term implications for resource procurement such as what storage to use and if it needs to be encrypted, how stringent access control for the environments needs to be and interoperability of the various technologies.

2. Regulatory Compliance
It wasn’t too long ago that if a business remained only national, then there weren’t necessarily many regulatory requirements. Presently, if a business has a website (and who doesn’t?), then international and cross-border data transfer is a given unless one takes the course of geo-restriction, which in itself can be a major undertaking with repercussions.

Determine regulatory needs. From where do people or other services access the data? What cross-border transfer is involved? To paraphrase the late 1960s-1980s public service announcement, “It’s 10:00 p.m. – do you know where your data is?” Design and implement accordingly.

3. Risk Management
One cornerstone of technology management is that it’s all about risk. If there’s a serious incident such as data intrusion or theft, what will the cost be to the organization and was the company part of the problem (e.g., negligence)?

What are the technical vulnerabilities, threats and risks presented by the product or service? What kinds of testing (e.g., regression, penetration, vulnerability, CTI) would it take to assess and address the risks? What does the patch cadence look like?

On the personnel side, what are the risks of key players leaving? What’s the morale of the team regarding aspects such as culture and pay? What’s being done to address those risks? Included are third-party, contractual, and privacy risks (remember data classification above?).

NPM packages are a growing trend as an avenue of exploitation; uptime is a common requirement for meeting obligations, and privacy is considered by many to be more than a privilege but a human right, and not something to take lightly.

Two Framing Activities

The foundation activities undergird the entire structure of DevOps, but also consider what shapes the structure.

1. Software Development Life Cycle (SDLC)
The SDLC is vital. Without it, where do all personnel (team, contractors, consultants) know how to proceed with designing and implementing the product? What model is used, and who’s responsible for what? It’s all here in the SDLC. Of course, things like references for API architecture specifics and corollary application documentation would make one feel like wandering in the Library of Congress (and make developers want to burn it down like the ancient Egyptian Library of Alexandria!).

Speaking of APIs: Due to the enormous rise of API use in all industries, including health care and retail, any use of APIs must be part of the SDLC.

Not only are APIs on the rise for public use and interaction, but microservices use within an org has increased drastically. APIs are necessary internally and externally for businesses to reach, maintain and accelerate the speed of innovation, so including them as critical resources is crucial. The average API usage per company increased by 221%, so attending to that ecosystem by ensuring proper activities, especially security, are included is needed for developers.

The SDLC is there to make workflow and production consistent (and concomitantly easier) by codifying what needs to be done for both company and customer, not to make work more difficult by presenting nauseating tedium. Like any corporate policy, it needs to remain useful, while not watered down.

Here are a few specific aspects to include:

● Threat modeling in some form needs to be included. It doesn’t have to be a magnanimous effort, but the realities of threats will play a major part in development.
● Testing. Regression, stress, pentest, security – these and others will help guide what kind of testing environment one will need and lead to any need to increase items such as capacity and scalability.
● Secure coding. If it’s being done, take steps to improve. If it’s not being done, just get started. It’s always hard to go back and fix insecure code, but it’s a necessity due to the many risks and regulations involved in providing software to the public.

2. Training
Training is required because businesses have to adapt as technological changes create new opportunities that lead to continuously changing customer demands.

Training is everywhere! There’s no one right place or right approach – just get training and change as needed (specifics should be handed down by leadership).

On a strict and lowball budget? One option is OWASP membership (which costs $50 per person per year) and includes several types of AppSec and DevSecOps training. That small investment can help raise the level of clean and secure coding, threat and security consciousness and professional reputation.

Setting the Course and Always in Motion

An issue with setting any course in business is that it’s semi-permanent. A DevOps model put in place becomes part of the foundation, which can make one reticent to commit. The only way to prove the chosen methods is to set them in stone and do them consistently for a long time. Aspects of the model may change over time, but once that foundation is put in place, it takes an enormous amount of work, time and money to change it. The change wouldn’t be a remodel, but a reconstruction, so the model takes proper consideration at the outset to chart the right course.

While DevOps needs a solid foundation, it’s an always-in-motion model. There are new technologies, new people, business changes, customer demands and societal and cultural shifts. These concepts can be included in calculations, but what those changes will be is incalculable. Hire and train the right people and the necessary steps and adjustments will be possible.

November 16, 2022
Shift That [bleep] Left

Seriously. The more you can shift things like security and test to the left, the more responsive the DevOps process will be. Some things are better shifted right … But only while solving the problem with shifted-left work. A good example is blocking zero-day attacks proactively while the development process fixes the code to stop the attacks permanently.

One thing we in IT are terrible about is following up on that “fix it” part—because we are very busy. Once a problem is blocked, we tend to fight other fires because we have too many things going on. We have absolutely got to get better about that. But I don’t want to get sidetracked. Today’s topic is about fixing bugs/vulnerabilities as they are created.

One of the issues with the way we used to handle test and security was that by the time we had “This is an issue we need to fix,” feedback, the developer was off doing other things. But with the increasing inclusion of security and testing in the DevOps process, we now have the ability to provide feedback far more quickly.

My favorite innovation in the DevOps space is not some wild new bug/vulnerability detection feature, nor is it rampant use of AI (though some uses of AI do come close to being my favorite). It is the humble IDE plugin. As a developer, writing buggy or insecure code is never the goal, but it happens every day. IDE plug-ins, hooked into test and security software, allow developers to get a heads-up that there might be an issue while writing the code. This not only addresses the problem while it is fresh in the developer’s mind, it also allows developers the opportunity to improve their craft simply by understanding which issues they create often and learning how to avoid them.

We have all written bad code. I once actually over-wrote databases for a bank; it was a great humbling and learning experience for a young hotshot who needed to be reminded he was not perfect when coding. Others no doubt have similar, if less catastrophic, stories to tell. The ability to get warned about issues—but quality and security—while writing the code cannot be overestimated. Some developers will hate it because it is more intrusive than the old school “Okay, all the code is written, you can test now,” process, but it is powerful precisely because it is more intrusive and proactive.

The challenge for those in the testing and development security spaces will be to keep false positives at a minimum. I’d say zero, but I’d rather get the occasional “Oh, that. Yeah, not an issue in this case,” than go through having it tell me things throughout development and still end up with buggy/insecure code.

You all are still rocking it. The amount of software in use every day across the world is staggering, and you’re cranking it out and keeping it maintained—whether you’re working on internal applications or commercial packages. You’re making it work. Take a look at QA and security IDE plug-ins, make an honest assessment of whether they could make your work more secure and/or higher quality. I think they can for nearly all of us. And they can save time by doing it earlier in the development process. And maybe you’ll get recognized for rocking it. Okay, probably not, but it is possible.

November 16, 2022