devopsy.kinsta.cloud

Author: Nathan Eddy

Tech Leaders Dependent on SaaS, Face Continued Dev Challenges

There is a growing reliance of technology leaders on software-as-a-service (SaaS) tools to support software development, with 84% of IT leaders indicating that they depend on SaaS solutions.

Despite its promise of efficiency and scalability, SaaS adoption has not alleviated the challenges across the software development lifecycle (SDLC).

This is leading to heightened workloads and business risks, according to an Onymos survey of 300 leaders in application and software development, IT, and security.

According to the report, 65% of tech leaders consider software initiatives business-critical, yet nearly a quarter (25%) identified application performance as a major challenge.

Development (22%), quality assurance (20%) and maintenance (20%) remain the most arduous phases of the SDLC, with 78% reporting increased workloads for application updates over the past year.

Additionally, daily maintenance tasks consume significant resources, with 52% citing them as the predominant time sink.

Shiva Nathan, CEO of Onymos, explained the dilemma organizations face when relying heavily on SaaS tools.

“SaaS offers so much speed and convenience upfront that it’s hard to resist, but it’s clear teams are paying for that speed and convenience later on,” he said. “Technology leaders need to use the SaaS solutions that help them reach their long-term goals, not just the short-term ones.”

The study also underscores the risks of unaddressed software development challenges, including higher operational costs (53%), increased security risks (45%), compliance concerns (38%), reduced profitability (32%) and missed revenue opportunities (28%).

Nathan emphasized that SaaS adoption often exacerbates these risks due to over-reliance on third-party solutions.

“The SaaS model is broken,” he said. “If you’re giving away ownership and control of your products and services to third-party vendors, you’re making a strategic mistake.

He said he recommends technology leaders ensure they retain control of their data and avoid lock-in.

“Talk to your vendors about data portability and code export options,” Nathan said. “This is the bare minimum.

Another key issue raised in the report is the burden of maintaining applications, particularly as SaaS solutions proliferate across organizations.

Nathan highlighted SaaS sprawl as a significant contributor to engineers’ workloads.

“The more third-party integrations your product or service relies on, the more APIs your team has to manage, the more updates they have to test, and the more security vulnerabilities they have to mitigate,” he said.

He advised enterprises to start by auditing their existing SaaS portfolio to identify redundancies and streamline maintenance processes.

Low-code and no-code tools, often touted as a solution for simplifying development, were another focus of Nathan’s critique.

While these tools can be useful for non-critical projects, he cautioned against relying on them for complex, innovative initiatives.

“Non-mission critical projects, MVPs, tools for sales and marketing teams — that’s when you want to use low-code/no-code,” Nathan said. “But if you’re building complex, unique products and services, low-code/no-code still doesn’t work, and it will never work. You can’t innovate inside of a template, a box someone else built for you.”

The report also found that while 84% of technology leaders prioritize optimizing application development, fewer than half feel successful in achieving this goal.

Nathan pointed to the need for a paradigm shift in how organizations approach software development.

“Technology leaders have to prioritize ownership and control,” he said. “The SaaS model has normalized trading those things away. Security, agility, scalability — all those things flow from ownership and control over your own products and services.”

December 19, 2024
Security Concerns Loom as GenAI Adoption Grows in DevOps

Most security and software development professionals are using GenAI-based solutions for building or delivering applications, but despite its widespread use, significant security challenges persist.

The Regina Corso survey of more than 400 security professionals and software developers found nearly eight in 10 development teams regularly integrate GenAI into their workflows.

However, most developers (85%) and security professionals (75%) worry that depending too heavily on these tools could compromise security.

One of the most pressing issues involves the use of GenAI-powered code assistants, with 84% of security professionals surveyed expressing concerns about potential exposure to unknown or malicious code introduced through these tools.

Nearly all respondents (98%) said they agree security teams need a clearer understanding of how GenAI is employed in development, with 94% stressing the need for better strategies to govern its use in research and development.

Liav Caspi, CTO of Legit Security, the company that sponsored the survey, said when building an application with GenAI-generated code, there are new threats and security risks that are different than with traditional software services.

“Threat modeling must take into account AI security dangers like data exposure, prompt injection, biased responses and data privacy concerns,” he explained.

He acknowledged that AI-generated code tends to be riskier, error-prone, or even malicious.

Caspi said one best practice is to ensure it is getting the same or tighter security testing than human-developed code.

“You have to think about it as if this is coding the organization received from an anonymous contractor,” he said.

GenAI Brings Innovation, Risks

Chris Hatter, COO/CISO at Qwiet AI, said GenAI significantly enhances productivity in app development by accelerating coding processes and automating routine tasks, leading to faster innovation.

“While we embrace these benefits, it’s crucial to address the unique security challenges that AI-generated code introduces,” he said.

From his perspective, the solution lies in implementing strong governance frameworks to assess AI development tools, understanding their training data sources, and establishing robust AppSec programs to evaluate AI-generated code for security vulnerabilities.

“As leveraging AI to write code quickly becomes a necessity to deliver software in a timely fashion and keep pace with competitors, security teams must find a way to implement safeguards that mitigate the risks while keeping pace with accelerated software development,” Hatter said.

He added there is good reason for security professionals to worry about insecure code generated by AI assistants.

“We know that they’re generating insecure code,” he explained. “The underlying models are typically trained on vulnerable open-source and synthetic data; multiple studies prove this.”

Hatter said when using AI assistants for coding, developers must ensure that they clearly understand the model they’re using and where the inference takes place.

“You should ensure that AI-generated code is examined closely for vulnerabilities and use tools capable of detecting hallucinated package recommendations,” he said.

GenAI in DevOps: Oversight Needed

The report also suggested that security teams need better oversight of GenAI usage in development.

Hatter explained data and AI have their lifecycle intertwined with traditional software development lifecycles.

“As enterprise AI use is now here, two things need to happen,” he said.

First, organizations need to treat the AI lifecycle with the same level of urgency as they do the software development lifecycle.

“Data preparation, model selection, and runtime application of AI all need to be secured,” he explained. “There is an entire category of emerging tools in this space.”

The second is that the organization’s SDLC security capabilities need to be adapted to deal with the influx of new code generated by AI.

“Your toolchain needs to be able to scale vulnerability detection and provide developers a head start by delivering high-quality autofix solutions–free of hallucinations,” Hatter said.

Caspi said with reliance on GenAI in software development expected to grow, security teams must get educated on how to secure AI systems and AI-generated code and get visibility into the usage of GenAI in development.

“That’s the first step,” he said. “It is critical to ensure consistent security steps across all code changes, both AI and human.”

November 19, 2024
AppSec Teams, DevOps Teams Facing Security Strain

Application security (AppSec) teams are facing increased strain as organizations ramp up their adoption of DevSecOps practices, according to a report from ESG.

The shift toward integrating development, security and operations teams is growing rapidly, with DevSecOps adoption expected to rise from 38% today to 48% over the next two years.

However, this accelerated pace creates significant pressure on already overburdened and under-resourced security teams.

The report also indicated that the rise of generative AI (GenAI) is adding another layer of complexity, with integration posing challenges for security.

The majority (97%) of organizations surveyed said they are either using or planning to implement generative AI in their software development processes yet security teams are expressing deep concerns about safeguarding AI usage.

Lack of Visibility

A key issue exacerbating these challenges is the lack of visibility between security and development teams.

The survey found that 42% of respondents can test and fix their code independently, without engaging security teams, leading to security gaps and underscoring the urgent need for better integration.

Melinda Marks, practice director of cybersecurity for ESG, encouraged AppSec teams to talk to developers and DevOps teams to find out about their workflows, processes, security awareness and any security testing they have in place, and align on goals to secure their applications.

“These should overlap in areas such as application uptime, ensuring customer service and protecting company and customer data,” she said.

Second, consider ways to incorporate security tools and processes into how developers are working, including setting policies and automating testing early in development processes.

“Third, ensure security has control and visibility to roll out security tools and processes that support development so they can efficiently manage risk and remediate security issues,” Marks said.

Karthik Swarnam, chief security and trust officer of ArmorCode, said to manage the increasing pace and scale of DevSecOps effectively, even with limited resources, AppSec teams should leverage AI capabilities for enhanced security testing.

“Automating the DevSecOps pipeline is crucial for maintaining efficiency without sacrificing security,” he said. “Teams should also prioritize the use of tools that provide visibility into security risks and help with the prioritization of remediation efforts.”

He said it is essential for teams to remain laser-focused on critical tasks, such as identifying which vulnerabilities require immediate attention, determining specific deficiencies to be addressed, and pinpointing where developers need targeted training—avoiding the inefficiency of training on all possible topics.

Marks said security wants to help enable the usage of AI because it will help with productivity, but they need to ensure it is secure because if there is an incident, such as data being improperly shared, or the usage of AI introduces vulnerabilities, it can be a setback for AI adoption.

“We also don’t want to just see security and IT blocking usage because then it inhibits the ability for companies to use AI for its advantages,” she said.

Swarnam said to close security gaps effectively, businesses must improve visibility and communication between security and development teams.

“A key strategy is to integrate these teams into a comprehensive vulnerability management program,” he said.

Providing business and executive-level dashboards, along with visibility reports that measure the state of security, allows for clear prioritization of remediation efforts.

These reports should highlight “what matters” most, ensuring that critical security gaps are addressed promptly.

“Integrating these tools into workflow management systems, coupled with prioritization capabilities that emphasize efficacy, ensures that security and development teams are aligned in their efforts to close gaps and strengthen the organization’s overall security posture,” Swarnam said.

August 19, 2024
CISA, NSA Issue Supply Chain Security Guidance Report

The National Security Agency (NSA), Office of the Director of National Intelligence (ODNI) and Cybersecurity and Infrastructure Security Agency (CISA) have issued guidance to assist software developers and suppliers to shore up software integrity and security across various stages of the software lifecycle.

The report, Securing the Software Supply Chain: Recommended Practices for Software Bill of Materials Consumption, was developed by the Enduring Security Framework (ESF) Software Supply Chain Working Group.

The report offered recommended practices as a framework for assessing and improving security practices concerning software development, release, updates, vulnerability notifications and mitigation.

The guidance also addressed the escalating cybersecurity threats targeting software supply chains, which have exposed vulnerabilities susceptible to exploitation by state adversaries.

It highlighted potential risks such as design flaws, integration of vulnerable third-party components, infiltration of supplier networks with malicious code and malware injections in deployed software.

To combat these threats, the report underscored industry best practices, emphasizing open source software management and software bills of materials (SBOM).

Specific areas covered include SBOM consumption, life cycle, risk assessment and operational implementation, ultimately promoting enhanced transparency and risk management throughout software life cycles for organizations.

Alex Kreilein, vice president of product security at Qualys, said while the concept of an SBOM is not new, organizations now have concrete and foundationally important guidance on how to make operational use of SBOMs while developing, managing or consuming them.

“The guidance supplied by the Enduring Security Framework reduces the complexity of SBOM management, making it accessible to everyone for the first time,” he noted. “SBOMs are part of a healthy software ecosystem, but are by no means the only factor.”

He explained that while the SBOM is an effective output, it cannot be trustworthy if the system that created it cannot validate the provenance of the product.

“It’s critical for organizations to understand the whole supply chain from code through pipeline to production,” Kreilein said.

He added that this is why The Linux Foundation and Google started the Supply-chain Levels for Software Artifacts, or SLSA framework, as a practical set of methods to action NIST recommendations in the Secure Software Development Framework (SSDF).

John Gallagher, vice president of Viakoo Labs at Viakoo, called the report a “major step forward” in providing an overall framework and transparency for how SBOMs are constructed, maintained and used.

“One of the key ways this helps relieve concerns over software supply chain security is by providing clear guidance on the risk scoring mechanisms,” he said.

This augments other software supply chain efforts (though it doesn’t replace them) in areas like the use of SBOMs in SaaS environments or version control mechanisms for SBOMs.

He noted two significant points on implementation that come out of this framework; first is that it gives guidance to departments like procurement as to what they should be requiring from their vendors, and second it gives a framework for solution providers to base their consumption and generation of SBOM and Vulnerability-Exploitability eXchange (VEX) data on.

“While this document does not address the methods of remediation, it does help to more quickly identify software supply chain vulnerabilities so that they can be addressed early on and ideally before the widespread use of that vulnerable code,” Gallagher said.

He noted that the more vulnerabilities that are caught early or before they are broadly used the later stage of remediation is either much easier or not needed.

Anthony Tam, manager of security engineering at Tigera, said when vulnerabilities have been identified, it is important to prioritize them based on severity and potential impact.

“This allows organizations to allocate their resources more effectively and focus on the most critical vulnerabilities first,” he explained.

Prioritization can be done by using a risk-based approach, considering factors such as the likelihood and potential impact of a vulnerability being exploited.

“By prioritizing vulnerabilities, organizations can ensure that they are addressing the most critical security risks first and reducing the overall risk to their software system,” Tam noted.

November 21, 2023
Culture a Stumbling Block to DevOps, DevSecOps

Organizations are struggling to improve DevSecOps practices as they confront a lack of cultural alignment and investment, according to a global Progress survey of more than 600 IT, security, application development and DevOps decision-makers.

Many companies are behind in achieving their DevOps and DevSecOps goals, with 73% of organizations saying they could be doing more.

More than three-quarters (76%) of respondents acknowledged they need to be more strategic about how they manage DevSecOps, and 17% said they still consider themselves at an exploratory and proof-of-concept stage.

Prashanth Nanjundappa, vice president of product management at Progress, said the most surprising finding was the evident contradiction between what organizations think is very important and their actual alignment, commitment or investment.

“While security is the number-one driver behind most DevOps and DevSecOps implementations, just 30% said they felt confident in the level of collaboration between security and development,” he added.

Meanwhile, 86% of respondents are experiencing challenges in their current approaches to security and 51% admitted that they don’t fully understand how security fits into DevSecOps.

Nanjundappa pointed out that company culture is, in fact, biggest barrier to DevOps and DevSecOps success.

“We found 71% of respondents agreed that culture is the biggest barrier to DevSecOps progress, yet only 16% were prioritizing culture as an area to optimize in the next 12 to 18 months,” he explained.

The study revealed the top business factors driving the adoption and evolution of DevOps inside their organizations included a focus on agility as well as reducing the business risk of quality, security and downtime or performance issues.

Organizations also recognized the need to implement DevOps to support a cloud mandate or their move to the cloud.

In addition, concerns about public cloud workloads, increasing regulatory requirements and concerns about financial impacts are driving how security around DevOps is approached.

“Organizations are juggling lots of priorities in how they manage security with key focus areas for security being digital marketing efforts, employee apps and customer-facing apps,” Nanjundappa said.

Challenges with security include prioritizing the development of external-facing capabilities over internal apps, the ability to secure different workloads using different development and delivery methods, meeting delivery deadlines and difficulties meeting audit requirements.

“Through the survey, we have seen some of the differences that stand out between organizations that have achieved higher maturity versus others,” he noted.

Some are related to cultural aspects and bring to the fore a focus on business agility through fast and frequent delivery of application capabilities, as well as using DevOps to better manage distributed and/or remote work environments.

Nanjundappa said among the best practices to ensure fruitful collaboration are stakeholder sponsorship on leadership for tools and outcomes aligned with the strategic direction of organizational goals.

“Let teams innovate at the grassroots level with regard to tools and processes to accomplish expected outcomes,” he explained. “Identify teams which have seen success, celebrate those successes and implement those processes across other teams in an iterative model.”

As organizations head into 2023 with a tight IT labor market persisting, Nanjundappa pointed out a key concern will be to find the talent they need to make these DevSecOps initiatives a reality.

“While it is ideal to hire pre-trained talent which fits their organizational needs, it will either be expensive or sometimes close to impossible for multiple reasons,” he said.

He recommended using a multi-pronged approach to bridging talent gap, which includes an on-ramp for hiring experienced key talent and reskilling through cross-pollination of experiences from different teams, including security, dev and ops.

Nanjundappa also advised upskilling existing teams with modern technologies, replacing legacy platforms and tech with cloud and cloud-native technologies and augmenting that with the right level of tools and processes, including policy-as-code and CI/CD.

November 21, 2022
GitLab Research Shows Security Concerns in DevOps Teams

The threat of breaches is top-of-mind for many DevOps teams, and while security continues to shift left it appears to be a driving force for many decision-makers when it comes to choosing a DevOps platform or other technologies.

These were among the results of a GitLabs survey that also found getting security right is the number one challenge for DevOps teams.

Getting it Right

Among the top reasons respondents said they chose DevOps were better code quality, developer productivity and operational efficiency—cited by 37% of survey takers, followed very closely by better security/more secure applications.

The survey also revealed that 53% of developers said they are fully responsible for security in their organizations, a 14% jump from 2021.

Michelle McLean, vice president at Salt Security, said it is risky for organizations to depend on development teams alone for security.

“Security and developer teams need to collaborate and work together–at every point in the application life cycle–not just development,” she explained.

For example, security teams must bring runtime insights back to development so they can be applied for education and training.

“It’s fundamentally important to choose a DevOps platform that either has security capabilities built-in or that can easily integrate with security platforms to facilitate collaboration by security and DevOps teams,” she said. “Otherwise, organizations run the risk of pushing out unsecured software or introducing other risks into the software supply chain.”

Tim Mackey, principal security strategist at the Synopsys Cybersecurity Research Center, added that an obsolete mantra in the cloud-native community held that security wasn’t a high priority as proper DevOps workflows meant that incremental changes were small and easily deployed.

He said the Log4j and Log4Shell experiences called that philosophy into question with a recognition that security issues can, and do, occur out-of-band from a release or feature and that continuous monitoring for external threats is part of a properly functioning security process.

“By implementing monitoring for new threats, regardless of source or nature, the ability for cloud-native development to quickly issue tested patches becomes itself a security feature,” he explained.

Understanding the Pipeline

McLean pointed out that while there is a growing understanding of security in the pipeline with DevSecOps, it’s not enough.

“This is particularly true, for example, in the case of APIs. Because APIs aren’t just straight code, you must see them being exercised to spot logic flaws,” she said. “You need the ability to monitor them in runtime in order to spot anomalies and find areas where APIs might be exposing critical data. Although developers write APIs, their testing often falls short for security.”

To accurately see what’s going on with APIs, developers need to exercise the API–even in pre-production.

McLean said this is an area where security and development collaboration and integration is essential–APIs underpin a significant portion of the modern economy and the disconnect between security and development teams can have a direct economic impact.

John Bambenek, principal threat hunter at Netenrich, a security and operations analytics SaaS company, added there has always been a delay between the release of new platforms and the awareness that security capabilities for those platforms need to be implemented.

“With cloud-native and serverless technologies, that delay is as short as I remember in my career,” he said. “I greatly prefer the world where everyone at least understands the need for security instead of having to spend time getting buy-in that it matters in the first place.”

He said if you can manage and implement security in a seamless and efficient way early in the development process, it is far easier and cheaper to address issues than addressing them after code has already shipped—and that’s without adding in breach or liability costs.

“You can either fix it in dev or in prod, but you’re going to have to fix it sooner or later,” he said.

McLean noted many organizations are still maturing in their DevOps practices, not to mention their DevSecOps initiatives.

From her perspective, educating developers as to what is expected of them from a security perspective before a line of code is written is key.

“To help increase compliance, we’ve begun to see organizations taking steps to introduce security tooling into their pipeline process and add testing into the build process,” she said.

Then they’ll add a step that triggers an alert for non-compliant builds but won’t prevent the build. Then they’ll add a step that mandates sign-off by the developer that they’ve seen the security alert for a non-compliant build but still won’t interrupt the build.

“In rare organizations with highly mature DevSecOps processes, they’ll fail the build for non-compliance,” she added.

McLean said organizations need to understand what’s realistic given their maturity and risk tolerance and implement controls that match.

They also need to reevaluate those controls to look for opportunities to increase adherence to security policies and other compliance regulations.

Overall, continuing to educate through compliance automation in the software development life cycle (in development, testing and deployment phases) further helps reinforce best practices while ensuring issues are uncovered before they become a risk or technical debt.

Mackey said because DevOps platforms touch the software powering your business, when choosing any DevOps platform, the security of the platform itself and the competencies it enables should always be must-haves.

“In effect, any decision about new software should be based on how it improves the current security capabilities of the business,” he said.

August 30, 2022
Opportunities and Challenges of Observability

Observability platforms that unify metrics, logs and traces across applications and IT infrastructure are transforming how IT is managed. Most DevOps teams today are able to continuously monitor IT environments using tools that track a set of pre-defined metrics; observability platforms make it possible to aggregate data so that DevOps teams can launch queries that help them uncover the root cause of IT issues. DevOps teams leverage observability to allow them to understand what is happening within their applications more effectively and efficiently.

Historically, applications were black boxes—meaning it was (and still is, in many cases) extremely difficult to know what is happening under the hood of the application.

With the advent and subsequent growth of observability tools and techniques in the DevOps space, teams went from struggling to collect and understand the available data (which could vary from robust to almost non-existent) to easily aggregating data from numerous sources and being able to understand what that data means to being completely deluged with data—so much data it’s hard to keep up.

Gather Data is No Longer Enough

Josh Chessman, vice president of strategy and innovation at IT infrastructure monitoring specialist Netreo, explained that as applications become more and more complex, (e.g., running across multiple disparate environments with different technologies and in different locations) being able to collect data from these different environments is no longer enough.

“Instead, DevOps engineers need to be able to collect the data and efficiently analyze it, too, to understand what is happening within the application,” he said. “Observability is the concept that drives this.”

Chessman pointed out there are numerous observability issues DevOps teams face, but that they vary from organization to organization (and sometimes from team to team within an organization).

“Part of the problem is that many organizations simply lack tools to provide the data collection and analysis needed to move to an observability paradigm,” he said.

This could be for numerous reasons, including budget (if you already have an existing tool, can you afford to acquire a new tool?), functionality (not all tools that claim to provide observability really do), implementation (this stuff is not easy and a poorly implemented tool could actually make things worse) and more.

“One of the larger challenges I’ve seen is that organizations will go out and acquire an ‘observability’ tool and assume that is all they have to do,” Chessman said. “Unfortunately, that is not the case. Acquiring the tool gets you the technology, but the team must also be structured and ready to take advantage of the concepts and functionality provided by the tool.”

GitLab’s senior director of product management Kenny Johnston explained that, on average, DevOps teams use five different observability tools which tend to inadequately connect developers to observability.

In turn, developers struggle to respond to incidents swiftly while jumping across multiple tools and sources of truth and struggle to assign service level objectives (SLOs) and error budget definitions to the responsible product development teams.

“To remedy this, we recommend adopting a unified software development solution that replaces the DIY ‘duct tape’ of products constraining developers and helps them deliver software more securely, at speed and scale,” he said.

Johnston said the most flexible and future-proof solutions today are open source tools like Prometheus and its ecosystem, as well as the OpenTelemetry toolset for more advanced use cases such as tracing.

Prometheus has become very common in the developer space and, unlike other open source solutions or libraries, its format automatically emits metrics.

“This allows for simple and future-proof interoperability, as one of the hardest things to change later down the line is code running,” he said. “Using open standards reduces the need for developers to rewrite and reintegrate the same type of code over and over again.”

Observability Requires Understanding Your Goals

From Chessman’s perspective, understanding what your goals are is critical for observability.

“Observability tools can do many things, but their primary function is to collect and analyze data,” he said. “While that is all well and good, if you do not define your KPIs from the beginning, all the collected and analyzed data is meaningless since you have no concept of what good and bad are.”

Trying to define good and bad from the data collected is likely to result in failure because you are looking at where you are to figure out where you want to go.

KPIs vary from application to application and, if you do not have them identified and well-defined before beginning down the observability path, it is that much harder, he said.

Most tools (observability or otherwise) are capable of collecting hundreds if not thousands of metrics, but not all—not even most, in many circumstances—will be relevant to an organization.

“Instead of having to sift through the cruft to find the valuable data, it is better to start with identifying the valuable data and then figure out how to get there,” Chessman said. “If you don’t identify your KPIs in advance you could end up using the wrong KPIs for the wrong scenarios.”

Johnston added that KPIs, like DevOps, are an iterative process and starting with KPIs helps developers create systems that can meet performance and availability expectations.

“It’s important that DevOps teams continuously review their KPIs to allow for regular feedback on successes and failures,” he said.

From his perspective, the best place to start for maintaining and improving observability is by storing observability definitions, instrumentation, SLOs, thresholds, incident response templates and dashboards as code.

“This allows for rapid improvement via code changes by any team member, whether that’s a site reliability engineer (SRE), developer or platform team,” he said.

May 27, 2022
Boosting DevOps Productivity Requires Visibility, Automation

Engineering teams continue to struggle with roadblocks and bottlenecks during the development process, according to Software’s recent 2021 Code Time Report.

The study, which is based on feedback from more than 250,000 developers using the Code Time plugin, found the median code time per developer is 52 minutes per day, or four hours and 21 minutes of code time per week.

The data also revealed that only about 10% of developers spend more than two hours per day coding, including on weekends, while about 40% of developers spend more than one hour per day coding.

The study also indicated developers are battling distractions that range from too many meetings and too many interruptions to too much wait time on processes and systems.

Distractions, Disruptions and Meetings Dog DevOps

The conclusion was most companies ineffectively deploy their development and DevOps teams, bogging down engineers with distractions, disruptions and meetings as well as system inefficiencies, such as slow reviews, slow builds and bad tools.

Software’s CTO Mason McLead said based on the company’s estimates, developers spend an additional 41 minutes per day on other types of work in their editors, such as reading code, reviewing pull requests and browsing documentation.

“The key to ensuring the highest levels of productivity from DevOps teams is visibility,” he explained. “We’re all biased toward focusing on what’s readily apparent, but the biggest bottlenecks are often hard to find and difficult to measure.”

McLead said by improving visibility across the entire development life cycle in a way that measures system-level metrics and not individual performance, DevOps teams can focus on the biggest constraint and have a more meaningful impact on their team’s productivity.

He said one example of a productivity metric with hidden barriers is lead time.

“I’ve spoken with many managers who disproportionately focus on how long it takes to review pull requests as a key lever to minimize lead time,” he said. “However, there are many stages in releasing a new feature that might actually be the key constraint—for instance, how long it takes to test and validate changes in a staging environment, or how long it takes to release merged pull requests to production.”

While these stages of lead time are all measurable, McLead explained most DevOps teams don’t have a data pipeline or BI tool set up to measure them.

He added that once an organization can diagnose the key constraint, it’s a lot easier to come up with an effective solution to minimize that constraint.

For example: If pull requests are taking a long time to create, are tickets being properly scoped? Can you see a correlation between the size of the pull request and the time it takes to be approved and merged?

McLean pointed out that if pull requests are taking a long time to be reviewed, it is critical to ensure proper expectations are being set.

He added it’s important to determine if there is a system in place to remind reviewers and enough time set aside in the day to perform the review.

“Taking steps to further automate deployment processes are another potential fix if it currently takes a long time to deploy a change to production,” McLead said. “It really all comes down to being able to measure the top constraint so that you know where to spend your time.”

Remote Work and Visibility

Remote work is also expanding the role of traditional DevOps teams: The trend toward more flexible working styles will demand more investments in DevOps such as secure provisioning of remote infrastructure, better access management and improved developer onboarding to better enable productivity for development teams.

“Visibility into how your DevOps investments are making an impact will play a big role in the decision on whether to stay remote or return to the office,” McLead said.

He added one way to attract and retain a highly productive DevOps team is to show that your DevOps investments are paying off.

“I recently talked to an engineering manager whose team wasn’t getting features out fast enough,” he said. “It took a month on average to complete a ticket, which had a significant impact on their ability to move major roadmap items forward.”

He explained that by implementing measures to improve throughput (e.g. investing in CI/CD, reducing build times and improving test coverage), they were able to decrease lead time from one month to four days.

By setting up a data pipeline and BI tool and increasing visibility, the engineering manager was not only able to report on metrics to his executive team and board with confidence, but also boost morale on his team.

From McLead’s perspective, retaining talented and trained DevOps engineers is critical because many DevOps projects require a longer timeframe in which to implement and measure their success.

“Highly effective DevOps teams measure not only how their investments are improving efficiency in the short-term, but also how changes in throughput and productivity are trending over months and years,” he said. “If your team is experiencing high turnover, it’s even more difficult to consistently deliver results over these longer time frames.”

March 1, 2022
Infrastructure Access Management a Growing DevOps Priority

Organizations plan to address access management over the next 12 months, as the need to secure and streamline infrastructure-wide access controls serves as a prerequisite to other initiatives, like zero-trust.

This was one of the key findings of a survey of 600 DevOps professionals conducted by Pollfish and sponsored by strongDM. The report also found that legacy access processes created severe team inefficiencies.

These inefficiencies require intensive time and resources to fix and block agile development practices: Nearly nine in 10 organizations surveyed said they required two or more employees to review and approve access requests and that those could take days or weeks to fulfill.

The survey also indicated that organizations continue to use access management practices that are not secure and that make it difficult to track and audit users and permissions of critical business systems.

Tim Prendergast, CEO of strongDM, said as more jobs become technical, there’s a bigger need to deliver access to more people–and that can have a severe impact on a company’s ability to remain secure. He explained that when 65% of organizations are reporting their teams used shared logins—and over 40% used shared SSH keys—there’s virtually no way for you to know who is in your infrastructure or the havoc they may be wreaking.

“This makes it difficult to pinpoint any leakage or loss because you have 20 copies of your house key floating around,” he said. “It’s an example of the trade-off most organizations make when it comes to speed and ease of access versus ensuring that access is secure.”

Survey respondents said their biggest challenges were the time required to request and grant access (52%) and the task of assigning, rotating and tracking credentials (51%).

Hurry Up and Wait

“Using current approaches to access means you’re hiring these high-paid, technical resources and telling them to hurry up and wait,” Prendergast said.

Nearly half (47%) of respondents said they struggled with onboarding employees and contractors and Prendergast pointed out that one in four organizations said simply getting approval for access required a process that involved four people.

“Think about that—in 25% of organizations, you have technical resources basically twiddling their thumbs while they wait to get access to this database or to that Kubernetes cluster,” he said. “Now multiply that by however many databases, servers, employees and third-party vendors that you have. And that’s not even counting when new technologies like Kubernetes are added to your infrastructure. Eventually, even just the frustration of your team as they wait for access becomes a liability.”

2022: A Year of Convergence

Prendergast predicted 2022 will see DevOps and security converge beyond what we’ve already seen with DevSecOps, where it has been heavily focused on shifting left and bringing security into the development cycle earlier.

“This convergence will be marked by new workflows, technologies and solutions that not just improve security, but that also improve the development cycle,” he said. “One great example is optimizing infrastructure access—when done right, you can improve your security posture with zero-trust methodologies while also making it easier for DevOps teams to access systems quickly and easily.”

He added that two of the biggest workforce dynamics facing zero-trust are remote work and the Great Resignation.

“You used to have this environment where you’d have to be physically present or on the VPN to have access—remote work broke that,” he explained. “And now you also have this large number of employees leaving their jobs. Do you know what systems they had access to? How do you know if all of that access has been turned off? What happens if they were using shared credentials?”

Prendergast said that’s why addressing access is critical to meeting this challenge and getting to modern security—if you don’t know who has access to what or what they can do in each system, you can never get to zero-trust.

“Organizations need to find a way to understand the relationship between each technologist and each technology and then be able to track and audit those relationships,” he said. “Until you do that, you’ll have a really hard time getting to zero-trust. These are the table stakes for modern security.”

January 21, 2022
Will Automation Fill Gaps Left by the ‘Great Resignation’?

Between burnout, job-hopping and the ongoing pandemic, labor shortages will surely get worse before they get better, and automation is increasingly sought after as a way to mitigate the ripple effects of understaffed IT departments.

For DevOps, automation has several benefits, from freeing up employees to work on more constructive tasks to minimizing software bugs and technical debt.

However, starting off slow and learning how automation will help address staff shortages without complicating processes is a critical factor.

“There’s still a lot of teams that are struggling with far too many manual or repetitive tasks,” said Rich Lane, chief strategy officer at Netenrich, a digital IT and security operations company. “Obviously, trying to do your day job of delivering features and functionality into the wild can be challenging with the speed that business needs demand.”

Lane, a former senior research analyst at Forrester Research, pointed out that slowing down in the present to do foundational work that can lead to higher levels of automation in the future can be hard to sell to digital services stakeholders.

“And yet, without doing so, it becomes increasingly difficult to meet commitments in the event a team becomes shorthanded or delivery requirements increase,” he said.

Generalist or Specialist?

Cody Michaels, application security consultant at nVisium, said when it comes to hiring people for a DevOps environment, you have one of two options: You can hire someone with a wide range of knowledge that spans the entire process or you can hire someone with expertise in one core area like programming or deployment.

“At an individual level, a generalist will be better than an expert, however, a team of experts will outperform a team of generalists,” he said. “Not only can automation fill in gaps between an expert’s area of influence, but it can also serve as a permanent fixture on the team. As a result, the turnover rate won’t affect any automation so long as it’s implemented properly.”

From Lane’s perspective, the main areas DevOps teams should be looking to automate are continuous integration and continuous delivery (CI/CD), IaC and AIOps-enabled incident management platforms.

“By taking the manual nature of day-to-day work off of DevOps engineers’ plates, they are freed to focus on digital transformation,” he said. “The number-one stumbling block is not starting with process.”

Lane noted unless you understand all the steps in a procedure that you’re trying to automate, it is very difficult to maximize the power of automation tools.

“Much of the process that is still adhered to today is outdated for the digital age,” he said. “Spend the time up front to map out what you hope to achieve with an automation project, what all the touchpoints are and how one can measure the quality of automation when it’s implemented.”

Michaels added that while the internet is flooded by companies shouting they have the “best” tools, that proclamation of “best” is going to be determined by budget and known languages.

“The top tools that I would recommend are Jenkins, Docker, Git and Jira, since these four play very well together with regard to automation,” he said.

Git, for example, has hooks that can trigger upon performing certain actions, like freshly submitted code, while Jenkins has automation as its middle name and it’s used in building, testing and deploying, which in turn facilitates CI/CD.

Meanwhile, Docker provides virtualization of operating systems and running applications in a secure manner in conjunction with containers.

“Lastly, Jira is one most don’t think about, but the platform has a lot of macros and filtering functionality that can update the work process automatically from events like code passing a test before it’s deployed, for instance,” he said.

Michaels said a couple of core considerations organizations need to make when approaching the idea of automation and AI are time as well as safety nets in place.

“Automation will take time to set up before you can benefit from it, but in the case of smaller teams, it won’t be too time-consuming to configure it all in one fell swoop,” he said. “But if you’re working with a lot of technical debt, unfortunately, it’s going to be payback time. If this is the case, my best advice would be to start small and not overhaul everything at once.”

Minor Automation Improvements Have Major Benefits

He said he thinks incremental improvement will be the best route for most organizations, and that even minor improvements will compound far faster than waiting a year to see if a redesign will work for you.

“As I mentioned previously, it’s imperative to have safety nets in place,” Michaels said. “Make sure you have a contingency plan if something goes wrong in the process to recover back to a working state. However, strictly speaking, everyone should make it a practice to have a recovery plan in their software development life cycle (SDLC).”

As the DevOps world moves into 2022, organizations choosing to embrace automation could find themselves in a win-win situation—for both developers and operations.

“The more automation takes over the ‘housekeeping’ items of the process, the more time will be freed up to improve the process as a whole,” Michaels said. “As a result, taking time to go back and clean up technical debt can now be set aside between sprints and maintenance.”

He pointed out that senior roles can tackle optimizing the architecture, which will ensure fewer bugs and a healthier work-life balance for all: Just imagine a world where developers can focus on perfecting their craft—instead of waiting hours or days for server reboots or disrupting their flow to update a ticket status.

“It really comes down to the fact that organizations can no longer make automation an afterthought,” Lane added.

He pointed out that given the difficulties in recruiting and retaining staff today, DevOps leaders must make their teams as efficient as possible—and automation can help achieve that goal.

“The number-one way to achieve this is applying as much intelligent automation throughout the IT environment as possible so that the highly-skilled engineers can focus on transforming the business,” Lane said.

December 1, 2021