devopsy.kinsta.cloud

Category: DevOps Culture

  • The Everything-As-Code Revolution and the OWASP Top 10

    The Everything-As-Code Revolution and the OWASP Top 10

    After years of stagnation, the Open Web Application Security Project (OWASP) Top 10 list finally saw some shakeup.

    Most notably, insecure design debuted on the list as the number four security risk to web applications.

    “If we genuinely want to ‘move left’ as an industry, it calls for more use of threat modeling, secure design patterns and principles, and reference architectures,” OWASP says.

    That hits the nail on the head. It’s a great sign for the industry that organizations are adapting and continuing to shift left responsibly, and points toward the everything-as-code future we’ve been imagining.

    This addition by OWASP finally recognizes a key reason why organizations struggle to move the needle on addressing security issues. A lot of security today is reactive and organizations find issues long after they deploy applications, leaving security teams to act as firefighters at a time when there are too few security professionals in just about every industry.

    To effectively move forward, we need a shift in thinking.

    Studies have shown that if organizations addressed these same issues in design, they would eliminate the costs of firefighting and significantly improve developer productivity. It would also free security teams to focus on the strategic needs of security. 

    This change from OWASP recognizes that security needs to have a seat at the table when key software design decisions are made and it requires empowering security teams in the same way the DevOps teams have been empowered to own and manage the application’s security design and implementation. 

    A Unique Inflection Point: DevSecOps

    Security breaches continue to pile up, yet many organizations haven’t reexamined their reactive approach, which ignores the architectural problems causing the issues. We can’t continue to operate this way.

    We need a more dynamic and comprehensive approach to security in cloud-native applications. We need to acknowledge the failure of detecting misconfigurations too late in the development life cycle. It’s impossible to scale that way, and you end up creating too many rules in the process.

    We need more proactive engagement, which can be difficult in an application development world where velocity is king.

    Developers and DevOps teams should view security as trusted partners and engage them early in the development life cycle. They should understand that they have a shared responsibility for security. Security teams should embrace automation that fits into the developers’ workflow and makes their lives easier by democratizing security.

    The Everything-as-Code Future

    If we can continue to shift left responsibly, we’ll keep moving the industry forward in a way that will have ripple effects, including in the OWASP top 10.

    In a perfect world, we would see frameworks and application infrastructure address the input validation issues that make up a large part of the top 10 today. Those issues should be off the list and the top 10 should focus on the systemic issues—like insecure design—that lead to the current list.

    As we move toward an everything-as-code world where the boundaries between the application and the infrastructure blur, OWASP may focus more broadly on everything-as-code, which would include a new class of issues such as cloud code, infrastructure-as-code, pipeline-as-code and security-as-code. While there’s certainly more work to do, that would be a strong signal that we’re moving in the right direction. 

  • Majority of Open Source Devs Open to Change Jobs in 2022

    Majority of Open Source Devs Open to Change Jobs in 2022

    Open source is the foundation of most modern software applications. Now more than ever, retaining developers with experience working on and with open source projects is critical to maintaining software development agility. Yet amid the Great Resignation, it can be challenging to keep talent happy—organizations must demonstrate a commitment to employee benefits, mentorship and cutting-edge technologies to remain competitive.

    The EDB Open Source Talent Survey 2022 surveyed over 1,400 developers and DevOps professionals working on open source projects. Below, we’ll highlight the key takeaways about their volume of work, the quality of on-the-job training and top motivators that could influence workers to take a job elsewhere.

    The Great Resignation Persists

    It’s not news that IT is under mounting pressure to deliver new features and applications to meet surging digital demands. As a result, the amount of work has increased across the board. The report found that 67% of respondents said their volume of assigned work has increased in the past 12 months.

    Employer favorability has also declined slightly—28% have a less favorable view of their employer over the last 12 months. Although 46% are satisfied with their current job, 44% are satisfied but would be open to new opportunities. And 10% said they were not satisfied with their current position. Taken together, that means 54% are open to or actively looking for new roles.

    Mentoring: An Enticing Benefit

    So, what are the main drivers that would influence IT workers to change jobs? It’s no surprise that increased pay and benefits are a top reason behind most resignations. However, salary is not the only factor—43% say improved career opportunities influence a job shift, a steep rise from 21% in 2021.

    On-the-job mentorship and training are also becoming more desirable. A little over a third (38%) said that better access to mentorship would make them consider another job opportunity. This was followed by working with more cutting-edge technologies (32%), greater access to training and certifications (30%) and allotting more time for open source projects (19%).

    Interestingly, one-fifth of responders who said they were satisfied with their current employer said their organization introduced a mentoring program in the past year. This indicates that investment in employee career advancement helps increase morale and retain skilled workers.

    Career Advancement And Training

    Out of the above reasons developers are open to switching jobs, one factor in particular is increasingly important: Working with cutting-edge technologies. This figure doubled in the last year, rising from 16% to 32%. Most engineers prefer not to work with cumbersome legacy technologies or manage rising technical debt. And if an organization isn’t evolving its software infrastructure, employee skillsets become stale.

    Working with new open source or cloud-native technologies can help retain workers, but will likely require training. In terms of specific categories, 41% said that training in emerging technologies, like cloud, is a favorable factor in considering another job opportunity.

    From a security perspective, ongoing training is also vital to staying on top of emerging exploits. As a result, 43% said security training is an interesting perk when considering a job shift. These areas are followed by training on core technologies like database and infrastructure (38%) and training on compliance (37%).

    Final Thoughts

    Open source software is pervasive and at the heart of today’s digital innovation. Nearly all applications depend on some form of open source software, but it’s no less prevalent in the DevOps sphere, where open source projects are used in continuous delivery, container orchestration, observability and a multitude of other stages of the software life cycle.

    Supporting the needs of skilled labor will be necessary to leverage the benefits of open source projects. As such, organizations will require not only competitive wages but also relevant training and investment into cutting-edge technologies to retain such developers.

    Another factor that could drive workers away is unnecessary toil, which can negatively affect morale. A company’s environmental, social and governance (ESG) initiatives can also affect its talent pool. Furthermore, a commitment to high-quality developer experience can ensure the needs of developers are heard and respected.

    The EDB Open Source Talent Survey 2022 surveyed over 1,400 application developers and IT professionals from around the world. It targeted open source technology users, specifically those holding application development, information technology and operations titles. For additional insights, you can read the full report here.

  • VW CEO Fired for Dev Fails | Fiber Shortage Hits | Google Fires Blake Lemoine

    VW CEO Fired for Dev Fails | Fiber Shortage Hits | Google Fires Blake Lemoine

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: Herbert Diess is out at VW because software is hard (yo), fiber optic cable is hard to find, and the guy who said LaMDA was sentient is dumped by Google. (more…)

  • Empathy for the API Developer

    Empathy for the API Developer

    Security teams have always been perceived as an impediment to delivery by software teams who feel that security imposes arbitrary and unreasonable policies and use poorly-integrated tools that are beset with high false-positive rates. With the advent of DevOps, security has been seen as an increasing obstacle to rapid deployment cycles. 

    Security teams believe developers do not care about security and will try their best to avoid security controls or policies. Security teams are often frustrated by developer pushback when remediating vulnerabilities, particularly when they fail to appreciate the cascading effects of making code and dependency changes late in the life cycle.

    From my perspective, with a foot in both camps and a decade of experience helping developers produce more secure code, I believe the key to addressing the disconnect between teams is to develop greater empathy for the developer.

    Why Insecure Software Exists

    Software development is a complex endeavor with many (often conflicting) requirements. Blaming software developers for insecure software is an overly simplistic and counterproductive approach, best summarized by a former colleague:

    “I can tell you that virtually all developers I talk to care about application security. They care about the quality of the products they create, the integrity of the data their software handles, and the reputation of the organizations they work for.” —  Jim Jastrzebski, Veracode

    From my experience, developers don’t deliberately set out to create insecure software; instead, there are a number of reasons it happens.

    Devs are overly optimistic: Developers are creative folks and natural problem-solvers. They can often visualize the perfect end result of their labor without necessarily considering the technical and security debt they may encounter along the way. Remember—things do go wrong.

    Overconfidence: I often refer to this as the Dunning-Kruger effect for developers. Faced with complex platforms, languages and frameworks, developers tend to overestimate their ability to comprehend and master them. Complex systems take time to fully understand and being overly confident can lead to unintended consequences and compromise security. Beware of insecure defaults and configurations.

    Bad things only happen to other developers: Think of this as a kind of schadenfreude for developers. During code reviews with developers, I have been surprised to hear comments like, “ … But that won’t happen to my code!” Despite vast evidence to the contrary, some developers think that bad things only happen to others and to others’ code. Assume a defensive mindset; expect the worst and hope for the best.

    Taking shortcuts: Developers are goal-oriented and want to produce working code on schedule. And schedules are tight. The inevitable consequence is that sometimes shortcuts are taken (missing error handler or input validation, for instance) leading to security vulnerabilities. Track and tackle your technical debt.

    Technical debt or legacy code: Speaking of technical debt, it’s probably one of the biggest issues facing developers: The burndown of technical debt and maintenance of legacy code. Software systems are complex and minor changes may wreak havoc, especially in microservices-based applications and architectures. Remember that developers may have legitimate concerns about making changes that are not absolutely necessary.

    This stuff is difficult: The biggest thing to remember to develop the empathy muscle is to appreciate the difficulty of the software development process. Developers are facing multiple conflicting requirements against tight deadlines and are required to master a variety of technologies. Mistakes will happen; the important thing is to learn lessons and improve the process.

    The last point is perfectly captured by this quote:

    “90% of security problems are just complexity problems. It’s technical complexity, it’s network complexity, it’s organizational complexity.” — Adrian Ludwig, Atlassian

    Care and Feeding of Developers

    Once security engineers have a better understanding of the challenges facing developers, they can focus on their messaging when communicating about security issues. Remember that the developer is unlikely to be personally responsible for every issue and should not be blamed or scolded. This will produce defensive behavior and is unlikely to achieve a positive outcome. Focus on the shared security challenge at hand and how a joint approach can solve the problem.

    A frequently-cited problem for developers is the relatively high false-positive rates security tools produce, particularly with code scanning tools. Security engineers should make sure that the tools are suited to your technical stacks and fine-tune the tool as much as possible. Ideally, the security team should triage the initial report and remove obvious false positives. 

    Additionally, certain legacy security tools do not integrate well into CI/CD pipelines and cannot be easily automated within the build process. If a developer has to use a manual or asynchronous process to perform a security scan, they will become frustrated and possibly ignore this requirement. Make sure security tools work well with the developer’s environments and workflows.

    Do not assume every developer has a deep understanding of the security issues identified—take the time to explain them thoroughly and/or document them internally. A great approach is to conduct shared learning sessions where developers and security teams work through a vulnerable application together to learn the concepts.

    Another frustration developers cite is a lack of clear understanding of the security requirements or requirements that are constantly changing. Ensure security policies are clearly stated and available to developers, and explain the reasons why the policies exist in the first place—for example, to meet PCI DSS accreditation. 

    Finally, make sure the combined teams are given time not just to learn but to absorb lessons and that success is recognized and rewarded. 

    “The sky is always falling—we never celebrate success.” – Chris Romeo, Security Journey

    Empowering Developers

    So, what does developer empathy look like for API developers?

    First, API developers and security engineers can both benefit from the positive security model that uses the OpenAPI definition as the contract for API behavior. Anything not specified by that contract is considered invalid—in other words, the contract specifies the allow list. This is in stark contrast to the traditional approach for web security which is reliant on blocking according to a block list—obviously, not all bad input can be known leading to false negatives with this approach. Overly aggressive blocking may block valid input and typically result in false positives; familiar to users operating web application firewalls (WAFs).

    The OpenAPI definition offers another advantage: The applicable policies can be expressed as code in the definition as opposed to via some verbose, abstract policy documentation. The benefit of policy-as-code is to uniformly enforce policy across the life cycle starting when the developer first starts coding the API. 

    Finally, developers need security tools that work in their preferred environments (for example, first-class IDE and CI/CD plugins) and provide value in the form of remediation guidance or actionable findings that help developers reduce technical debt. 

    Security and development don’t have to be at odds; help developers to help themselves by building empathy and working with them rather than against them.

  • 3 New WFH and Hybrid Work Trends That YOU Need to Grok

    3 New WFH and Hybrid Work Trends That YOU Need to Grok

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week, a remote/hybrid work special: Bribes to relocate you, Fridays are forgotten, and big tech firms stop building offices. (more…)

  • Survey Warns of Looming Software Testing Crisis

    Survey Warns of Looming Software Testing Crisis

    A survey of CEOs and IT professionals involved in application testing finds a significant gap in terms of how acceptable it is to release software that has not been properly tested.

    The survey, conducted by the market research firm Censuswide on behalf of Leapwork, a test automation platform provider, polled 480 CEOs in the U.S. and the United Kingdom (UK) from organizations with more than 500 employees and 480 heads of quality assurance, testers, release and delivery managers and site reliability engineers (SREs).

    A full 85% of the CEOs in the U.S. and 69% in the UK said it’s acceptable to release software that hasn’t been properly tested, as long as patches are tested and applied later. Less than half of the IT professionals surveyed (49%) agreed. A full 94% of CEOs also said they’re confident the software released is tested regularly.

    However, 79% of the IT professionals admitted that as much as 40% of software goes to market without sufficient testing. As a result, 68% of UK and 52% of U.S. IT professionals said their teams spend five to 10 days per year patching software.

    The survey also made it clear that everyone involved is well aware of the risks that lack of testing creates. Nearly every CEO (95%) is concerned about losing their job in the wake of a software failure. Slightly more than three-quarters of the IT professionals (76%) surveyed have the same fear.

    More than three-quarters of CEOs (77%) also noted that software failures have harmed their company’s reputation in the last five years.

    Many organizations today assume that Agile development and DevOps best practices will enable them to apply patches quickly in the event an issue arises. The trouble is, in the absence of sufficient testing many of those organizations are essentially experimenting on end users.

    Leapwork CEO Christian Brink Frederiksen said the primary reason so much software is not being properly tested is that most organizations continue to rely on manual testing processes that can’t keep pace with the rate at which applications are now being developed; one major factor is the use of low-code and no-code tools.

    Of the IT professionals at organizations that release insufficiently-tested software, a full 40% cited underinvestment in test automation as a factor, with only 43% of respondents using some type of automation. Other reasons cited for the lack of appropriate testing included lack of time (34%) and increased rates of deployment (29%).

    Just over a third of CEOs (34%) and about four in 10 testers (42%) whose companies use or develop in-house software that goes to market without being tested also cited a lack of available skilled developers. More than a third of CEOs (34%) also cited underinvestment in testing personnel.

    At a time when more organizations than ever are dependent on software to drive digital business transformation initiatives, it’s apparent that an application testing crisis is imminent. The only thing left to be determined is how catastrophic the fallout will be because existing processes don’t scale to meet the challenge.

  • Why More Incidents Are Better

    Why More Incidents Are Better

    Ask most SREs how many incidents they’d have to respond to in a perfect world, and their answer would probably be ‘zero.’ After all, making software and infrastructure so reliable that incidents never occur is the dream that SREs are theoretically chasing.

    Reducing the number of actual incidents as much as possible is a noble goal. However, it’s important to recognize that incidents aren’t an SRE’s number-one enemy. What matters more than the number of incidents you experience is how effectively you respond to each one.

    Plus, there’s value in incidents. They are a learning opportunity. If your business never experienced them, it would arguably be facing more risk, not less.

    We know: These ideas may sound a little counterintuitive. You might even accuse us of being “pro-incident”—which we sort of are. Allow us to explain.

    The Silver Lining

    In many respects, incidents are inherently bad. When one occurs, it means something broke. That’s bad. It may also mean that users were disrupted, operations halted or money was lost. Those things are even worse.

    On the other hand, incidents aren’t all bad. They actually benefit SRE teams for several reasons:

    • Learning opportunities: Incidents are opportunities to figure out what went wrong and prevent it from recurring. They can also help teams learn how to react more quickly or efficiently the next time something fails.
    • Get ahead of bigger issues: Sometimes, working through one incident means you can avoid another that’s even worse. Perhaps one server fails, for example, and your response revealed that the failure was due to a larger issue that would have eventually caused a worse outage if left unaddressed. But thanks to the incident, you detected the larger issue before it triggered a more massive failure.
    • Reinforce team culture: Nothing breeds camaraderie or a spirit of collaboration like working alongside other engineers to respond to a crisis in the middle of the night. Although being in this setting may not be anyone’s first choice, it does often have a positive impact on your team’s culture and esprit de corps.
    • Demonstrating value: Assuming you handle them well, incidents are an opportunity for SREs to prove how valuable they are to the organization. If incidents never happened, it’s a safe bet that some bosses would start to wonder why they need SREs in the first place. (It would be a flawed train of thought, of course, because SREs would deserve credit for preventing incidents, but it’s a thought that may float around some C-level brains nonetheless.)

    We could go on, but the point is clear: Although incidents cause problems in some respects, they actually create value in others.

    Focus on Response, Not Avoidance

    This is not to say that you should welcome incidents with open arms. Obviously, any decent SRE should focus first and foremost on being proactive and preventing incidents from happening whenever possible. They should use chaos engineering to identify problems that could be lurking unseen in production environments. They should leverage IaC to minimize risks. And so on.

    That said, what ultimately matters more than incident frequency is the effectiveness of incident response. It’s better to experience ten incidents that you resolve in under an hour each than one incident that takes mission-critical systems offline for a week.

    So, in addition to investing in tools and processes that mitigate the risk of incidents, SRE teams should place equal emphasis on ensuring that they can react quickly and effectively when an incident happens. This means having the ability to share information efficiently, define clear roles, know what to prioritize when working through complex incidents and have clear plans in place that spell out how you’ll handle a problem as soon as you detect it. Without these abilities, you’re at risk of letting incidents that should be small turn into major outages.

    ‘Zero Incidents’ is not Realistic

    It’s important to recognize, too, that while it can be fun to imagine a world where zero incidents occur, the reality is that such a world will never exist. If it could, we wouldn’t see new records set each year for the number of security incidents that businesses collectively suffer, for example.

    Nor would we see headlines about major outages at huge enterprises like Facebook or AWS on a recurring basis. If those companies, which have world-class reliability teams and virtually endless resources at their disposal, can’t reduce incidents to zero, neither can anyone else.

    Conclusion

    The bottom line: There is no such thing as total incident prevention, no matter how hard you try. And even if there were, that wouldn’t actually be a good thing, for the reasons explained above.

    So, by all means, undertake reasonable proactive efforts to prevent as many incidents as you can from happening. But don’t let investment in prevention cause under-investment in response. Being prepared to handle incidents when they happen—which they inevitably will—is what matters most.

  • JFrog Integrates DevOps Portfolio With Microsoft Teams

    JFrog Integrates DevOps Portfolio With Microsoft Teams

    JFrog this week announced it has integrated its DevOps portfolio with Microsoft Teams to enhance collaboration.

    Deep Datta, a senior product marketing manager for JFrog, said JFrog Artifactory repository and the JFrog Xray software composition analysis (SCA) tool have been integrated via a JFrog App for Microsoft Teams tool that is now available.

    The goal is to make it simpler for DevOps teams to collaborate with other members of an organization that has standardized on Microsoft Teams for collaboration, said Datta.

    In general, as more organizations are realizing how dependent their business is on software, individuals ranging from application owners to cybersecurity professionals are now looking to collaborate with DevOps teams, noted Datta. In fact, in many cases, business process workflows now need to align with DevOps workflows as businesses become more dependent on software, he added.

    While many DevOps teams already use a range of collaboration tools, many of those same teams also need to access Microsoft Teams to engage individuals outside of those teams, said Datta.

    The emerging challenge is the number of communication channels that DevOps teams now find themselves using. Each new communication channel added doesn’t necessarily supplant existing platforms; it’s not uncommon for DevOps teams to use Microsoft Teams alongside Slack as well as traditional email depending on the type of messaging and collaboration required. In some cases, for better or worse, DevOps teams find themselves required to juggle multiple collaboration platforms over the course of any application development project.

    In general, the relationship between DevOps teams and the rest of the business is evolving in the age of digital business transformation. The pace at which applications are delivered and updated now has a material impact on business goals. Organizations are finding they need to align everything from marketing and customer support to financial planning with software development. A delay in one software development project can have a cascading effect that impacts every aspect of the business. The sooner information is shared via a collaboration platform the easier it becomes for the rest of the business to adapt and respond.

    Naturally, not every DevOps team is excited about the level of transparency required. Many issues a software development team would previously have handled internally are now exposed to the agenda of the board of directors. Application delivery and update deadlines are influenced more heavily by revenue goals than the available developer expertise. The key to managing all those expectations is to communicate more frequently. However, all that communication can also take time away from building applications. It’s important to isolate development teams from distractions—like a constant barrage of messages shared with multiple lines of business and across multiple communication channels.

    Of course, the only thing worse than all those messages is a failure to communicate critical information in a timely fashion. The challenge is to find a way to filter out all the noise and get critical information to the right person at the right time.

  • Red Hat CEO: Out | Blind Users: Revolt | ARM: Google Joins Party

    Red Hat CEO: Out | Blind Users: Revolt | ARM: Google Joins Party

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: Matt Hicks is the new Red Hat CEO, visual accessibility is in focus, and Google Cloud rolls out ARM instances. (more…)

  • Security Compass Makes Visualizing AppSec Threats Simpler

    Security Compass Makes Visualizing AppSec Threats Simpler

    Security Compass this week updated its threat modeling platform for developers to make it easier to surface application security issues.

    The latest version of SD Elements 2022 adds support for developer-centric threat modeling diagrams, reusable components, more advanced reporting capabilities and 114 more just-in-time training (JITT) modules. In addition, Security Compass has now integrated its platform with the Black Duck software composition analysis (SCA) tool.

    Drew Koelemay, director of product management for Security Compass, said the goal is to make it easier for application development and cybersecurity teams to collaborate using a platform that visualizes cybersecurity threats within a DevOps workflow.

    That approach to application security is critical because it becomes easier for developers to understand how changes to applications improve the overall cybersecurity posture of complex app environments where there are lots of dependencies, he said.

    Once that visualization is achieved it then becomes possible to create reports highlighting the most prevalent threats and weaknesses across that portfolio, Koelemay noted.

    In the wake of a series of high-profile application breaches, organizations of all sizes are looking to better secure their software supply chains by implementing DevSecOps best practices. The issue that immediately arises is the cultural divide between application development teams and cybersecurity professionals. The Security Compass platform surfaces developer-centric visualizations of security issues to help bridge the historic divide that exists between those teams, said Koelemay.

    For the most part, cybersecurity professionals view development teams’ lack of cybersecurity expertise to be the root cause of many of the issues they encounter. It’s not that a developer deliberately sets out to build and deploy vulnerable applications. By the time the application is reviewed—often just a few days before it’s supposed to be deployed—it’s often too late to do much about most of the vulnerabilities that are discovered. Cybersecurity professionals and application development teams need to meaningfully engage each other much earlier in the application development life cycle.

    The simple truth is developers are not likely to reach out first simply because they don’t know what cybersecurity issues to look for in an application environment. Most of the time they are rushing to meet an application delivery deadline, which makes them perceive cybersecurity as a hurdle to overcome rather than an automated process integrated within their workflow. Cybersecurity professionals, therefore, need to find a way to more constructively engage developers. The onus is on cybersecurity teams to insert themselves at the front end of the application development process rather than waiting at the backend for an inevitable issue to arise.

    One way or another, application security is going to improve. The only thing yet to be determined is how painful the process will be. Tools that enable DevOps teams to see for themselves what issues are being encountered should go a long way toward reducing both the level of pain being experienced and the amount of time required to resolve the cybersecurity issue at hand.