devopsy.kinsta.cloud

Tag: shift left

  • Competing Priorities Prevent Devs From Creating Secure Code

    Competing Priorities Prevent Devs From Creating Secure Code

    The recently released Secure Code Warrior State of Developer-Driven Security Survey revealed that developers continue to wrestle with secure coding practices in a working environment that has long prioritized features and functionality and speed at the expense of security. 

    Of the more than 1,200 developers who took part in the survey, only 14% named security as their top priority. Security was seen as a bottleneck—an astonishing 67% of these developers said they routinely left known vulnerabilities and exploits in their code and 48% shipped products with known vulnerabilities.

    These statistics raise eyebrows, sure, but they are also unsurprising. In the survey, application security fell behind creating quality code, application performance and the ability to solve real-world problems on the list of developers’ priorities.

    In many cases, these results showed that security is either ignored in favor of meeting demanding timelines, developers would rather improve an application’s performance or they simply lack the required training to fix security problems. Regardless of the reason, it is a continuing problem.

    The Need to Balance Security in Development

    As the survey showed, developers often worked with secure code in silos. Security controls were limited to a single category instead of a holistic view of fundamentals. In many cases, developers relied on existing or pre-approved code rather than writing new code free from vulnerabilities. This assumed that the code inserted meets proper security needs, but that assumption often fell short.

    That said, developers who write code often find themselves inadvertently introducing vulnerabilities. This is not entirely the fault of those developers, but some of the blame lies with the cultural risk acceptance of the organization.

    The software organizations themselves must put more onus on secure development. As we saw from the SolarWinds attack, deficiencies in code can lead to massive disruptions. While not all application vulnerabilities have the reach of SolarWinds, businesses often allow for too much risk to pass in their code; that gets passed on to customers and developers are made the scapegoats.

    Provide Developers With the Right Skills

    Security leaders must first ensure developers have the requisite skills to code securely and to make improvements. That includes not just writing the code but also how code is committed through the entire development process. Developers often need hands-on training in proper coding patterns created in the languages and frameworks they commonly use.

    Organizations can no longer overlook security and ship potentially exploitable software. Security must be added earlier in the development process and be seen as imperative to success. The survey showed that many developers believe secure coding is out of their realm of expertise. A staggering 86% said they found it challenging to practice secure coding, while 92% of developer managers said their teams needed more training in security frameworks.

    This highlighted that developers do not get the right kind of training—nor are they get enough exposure to security best practices. It also underscores that designing for security is not seen as an essential skill for developers to have.

    Reasons for Optimism

    While our survey highlighted a disturbing trend, it also offered reasons for hope. The survey showed that attitudes have changed, and 66% of developers expected security to become a more significant priority over the next 12 to 18 months. Hiring managers echoed this sentiment—82% expressed an interest in hiring developers who are knowledgeable in security over those who are not.

    Businesses today have put themselves and their customers in a risky situation; organizational leaders must be agents of change to mitigate this risk. They need to apply a security-first mindset and empower their developers to better integrate security into the development process. Timelines must be revised to allow this to happen. If it continues, the current state of application security will only lead to more (and more severe) breaches. Now is the time to ensure DevOps teams and developers have better access to security training and more time to put those practices into action to ensure long term success. 

  • SRE Vs. DevOps: The Wrong Question?

    SRE Vs. DevOps: The Wrong Question?

    The age-old question about the competition between DevOps and SRE sets up a false dichotomy. DevOps is a methodology while SRE is a team within operations. Although the two are often pitted against one another, developers also embody the skills and the capabilities to implement fixes. 

    Traditionally, application changes were reactive. The process of SRE formalizes problem resolution playbooks for better efficiency within the development life cycle. This has changed substantially over the past few years with DevOps seeing more traction and mainstream adoption.

    So what is the modern relationship between DevOps and SRE and how can they best be used together? Let’s understand DevOps first.

    Defining DevOps 

    According to Chef’s Embrace DevOps guide, DevOps is about transforming the way companies run; part of that transformation is understanding that companies should focus on managing people rather than products. DevOps evolved over the last decade and was originally born into the world of software. The adoption of cloud technologies, changes in the tech stack architecture, SaaS platforms, and security regulations have all impacted DevOps culture

    Some of the fundamentals of modern DevOps are:

    Everything-as-code: The current state of IT architecture has adopted an “everything-as-code” approach. Infrastructure, configuration, compliance and security can all be transformed into code.

    Automate everything: Codifying everything enables standardization of processes and policies across the organization. The processes are simplified into repeatable functional code, and this makes it easier to automate the development and deployment processes while significantly reducing human errors.

    Validation: There are many tools used for continuous integration, delivery and deployment in addition to other tools used for maintaining infrastructure and compliance. Without validation, there would be no way to know if everything is functioning as needed and it would be difficult to ensure seamless delivery in a highly agile environment.

    Above all, teams adopting DevOps methodologies take end-to-end responsibility of managing the application development life cycle (develop, deploy and maintain). They bridge the responsibilities gap between Dev and Ops into one team. 

    SRE Focus Areas 

    Successful site reliability engineering (SRE) teams focus on the volume of performance metrics and the process of shifting left the responsibility for managing and mitigating risk. Fundamentally, they share the same values as developers in creating efficient applications. However, SRE teams focus on introducing efficiencies based on incidents that occur and sometimes need development teams’ intervention to shift the risk left in the product development cycle. 

    Troubleshooting is one of the main responsibilities of SRE teams since they have to manage incidents. As a result, SRE teams have a strong knowledge of product design which is essential for product reliability, rollouts and reducing recovery time. Scalable automation is also a big part of the challenge SRE teams face. Oftentimes, tasks are done manually and this can increase the likelihood of human error. A solution is implementing a policy-as-code approach in concert with developer teams.

    It should be noted that when it comes to the responsibility and scope of work between SRE and DevOps, the lines are blurred—especially with the proliferation of APIs and when evaluating microservices. The types of tools being used are a differentiator since SRE teams measure efficiency through monitoring, detection and incident reporting systems. 

    Breaking the Silos 

    Modern organizations will fail when divided into silos. Developers understand that to optimize their internal processes, they need to use a DevOps methodology and involve SRE teams. For example, a startup growing and looking to scale needs development teams that are performing SRE functions with strong operational knowledge. It requires buy-in and understanding of every team member to know their role and the needs of the organization as it grows and responsibilities shift. 

    These fundamental values are crucial for an enterprise looking to achieve an efficient CI/CD pipeline. In a reactive cycle, SRE teams step in and are responsible for some parts of operations, but the DevOps paradigm is different because it integrates the responsibility of deployment, monitoring and remediation throughout the entire team. 

    Organizations embracing DevOps practices should embrace SREs’ expertise and welcome their practices into developer teams. SREs can leverage automation to fix issues as they occur and also make changes in software architecture to avoid such issues in the future. Compared to the reactive approach, developer teams are incentivized to keep things constantly running. 

    Scaling DevOps practices requires shared responsibility. Of note, developers have to answer when the phone rings and this can cause chaos in engineering teams responsible for large enterprise-grade applications with fast time-to-market and frequent update cycles.

    A Modern Perspective

    Efficiency is a core value shared by both SRE teams and the DevOps methodology. When organizations can fully integrate operations within the DevOps cycle, those silos can be broken down and integrate a shift left mentality. The traditional reactive remediation cycles won’t cut it anymore. Proactivity is essential for today’s lightning-fast application needs and continuous updates. 

    Application problems can be addressed immediately when approached with modern tooling and a policy-as-code approach enabling developers and SREs teams to work together. The modern relationship between DevOps and SRE teams should be one of coexistence. Both are needed for successful scalability within the enterprise. Many once thought SRE teams and developers couldn’t coexist in harmony, but debunking that myth helps organizations optimize their collaboration efforts. 

  • Shift Left is Only Part of Secure Software Delivery

    Shift Left is Only Part of Secure Software Delivery

    We’re living in the age of accelerated consumption and delivery. You can get a seemingly infinite selection of products delivered to your door within two days, for free, from thousands of miles away. You can access an endless variety of services online within mere seconds: Movies, music, games, education and even health care. These modern marvels are afforded to us by relatively recent advances in software development, delivery and operations. 

    Among those advances are the adoption of DevOps practices, continuous integration and continuous deployment (CI/CD) pipelines, and automating everything from infrastructure provisioning to testing and quality assurance. These advances enable developers to deliver software faster and think about security earlier in design and development to ship safer software.

    What is Shift Left” and Why Does it Matter?

    “Shift left” is a philosophy for addressing the pitfalls of traditional waterfall-style development. In waterfall, a number of IT teams work over the course of many weeks or months, culminating in a big and beautiful software release. Of course, people make mistakes, forget things, and maybe don’t communicate well with each other. When it turns out that application elements don’t interoperate well, the teams frantically attempt to fix what’s possible and jettison what isn’t, with a very serious deadline staring them in the face. Products inevitably ship late, still somewhat broken, and with security bolted on as an afterthought.

    To shift left means to introduce security process and tooling earlier into the design and development phases. Security should be just as critical as functionality and quality. It’s a great idea because discovering a critical, build-failing vulnerability in a critical dependency after you’ve built dozens of functionalities on top of it hurts a lot more than if it’s found before you build anything at all. Maybe fixing the problem is as simple as using a newer version of a dependency, but now you have to make sure everything you’ve customized still works as intended.

    There are a number of forms of application security testing (AST) that aim to detect code-level issues early in the systems development life cycle (SDLC). Two common types of AST when shifting left are software composition analysis (SCA) and static application security testing (SAST). Put simply: SAST tests the custom code your developers write; SCA tests dependencies you include in code. 

    Assessing the security of modern applications requires both testing types. If you believe shift left means not only finding problems but also fixing them, then developer-friendly tooling is necessary. Exactly what that means can be subjective depending on developer workflows in the organization, but at the very least, developers need self-service, seamless integration into their existing CI/CD toolchains and actionable results. Because most developers aren’t security experts, security findings must prioritize the riskiest problems, point to the source of the issue and provide contextualized fixes, rather than making the developer hunt for them on their own. 

    Augmenting AST With Runtime Intelligence

    One challenge with doing a lot of early testing—especially security testing—is that the more you test, the more findings you discover. Unfortunately, some of the results will be false positives and others may be true positives that pose a relatively low risk. Development teams with a high degree of security expertise must sift through and vet findings to prioritize flaws that are severe and actionable, but such practitioners are in short supply and this type of work is tedious.

    Addressing the pitfalls of security testing isn’t simple, and it involves gathering as much contextual information as possible to begin to reason about the risk associated with each finding, usually expressed by what exploitable code is actually exercised. 

    Reasoning about vulnerability risk is not easy. We try to rank the criticality of issues to decide on an action. We can split the problem into two parts: 

    • The context of the vulnerability itself includes metrics of severity, like the CVSS score, and threat intelligence, like whether exploits are publicly available or attacks are prevalent in the wild. This information is readily available from scanning tools and third-party sources like NVD, MITRE and CISA.
    • The afflicted asset’s environmental context, however, is specific to your application and your infrastructure. Only your organization can really fill this part in. For example, is the impacted application protected by mitigating security controls? Is it exposed to the internet? Does the vulnerable component even get used at all? This is difficult or impossible to know until the application is running in its intended production environment. 

    Shift left is a way forward for modern cybersecurity, but the approach can be vastly enriched by pulling in runtime intelligence. This helps to prioritize what to fix first, removes friction, saves cycles of numerous IT teams and reduces your organization’s vulnerability risk.

    What Happens After Release?

    The whole point of shift left is to release secure software faster, but what happens next? Your flawless, beautiful code is delighting your customers with its stunning feature set and impressive performance. Then, on some idle Friday afternoon, right before what should have been a long holiday weekend, you get hit with a horrendous zero-day that you could never have predicted even if you tried. 

    When that happens, you need to respond by quickly identifying all of the affected workloads running in your environment and prioritizing them for remediation. You may need to take those workloads offline to patch immediately, limit network access to them, or otherwise mitigate the problem if a patch is either unavailable or taking too long to deploy. The point is that in real life, shift left is only one part of a successful security program, and the protective, detective, and remediation activities for your production environments on the right are also critically important to your organization’s safety and success.

    Conclusion

    Shift left alone was never going to solve all security problems, but it does create foundations for reliable, fast and secure software delivery pipelines. Security and development teams alike often suffer from a deluge of noise and few useful signals. What they really need are consistent and prescriptive ways to make sound decisions about risk. Gathering context from multiple sources through multiple layers of the stack, including runtime signals, substantially improves the quality of the information that DevSecOps teams receive from their tools and must act on.

    Effi Goldstein, director of products at Snyk, co-authored this article. Check out Sysdig and Snyk’s 2022 Container Security Trends: Expert Panel Livestream for more information. 

  • Embedded Connectivity Solves the Right (and Left) Problem

    Embedded Connectivity Solves the Right (and Left) Problem

    In Steven Bellovin’s book, “Thinking Security: Stopping Next Year’s Hackers,” (2015) he wrote:

    “Companies are spending a great deal on security, but we read of massive computer-related attacks. Clearly, something is wrong.

    The root of the problem is two-fold: We’re protecting (and spending money on protecting) the wrong things, and we’re hurting productivity in the process.

    InfoSec is hurting productivity in the organization it is supposed to be protecting.”

    Bellovin clarified what he meant by “the wrong things.” He observed that security measures tend to stick around even when the threat they were intended to address has faded into the past. Those security costs have only increased since 2015 when the book was published, and today represent a trillion-dollar drag on the global economy despite the emergence of security-integrated models of software delivery such as DevSecOps, shift left and shift right.

    The 2021 IBM Security Threat Intelligence Index reports the No. 1 attack vector was a surveillance technique known as scan-and-exploit. Defenders face an asymmetric engagement with such network threats. The successful attacker needs only to exploit a single vulnerability whereas defenders must secure 100% of the network’s ports of entry 100% of the time. The fundamental problem is the anonymous and permissionless nature of the internet. This makes it both useful and easily abused. Anyone anywhere may conduct reconnaissance and launch attacks at a massive scale with little cost or risk. How can we make it prohibitively difficult or costly to successfully surveil and attack our own applications? What if we could reliably defend against the entire category of active scanning tactics?

    There are no secure networks, only isolated networks. Taking this back to first principles, a network’s purpose is to convey packets of data from sender address to recipient address. The network has no built-in ability to identify either party or to discern the legitimacy of any activity. Active scanning techniques identify potential targets that are then probed by scripts that may be intrusive or stealthy or both. We could simply reject all incoming network traffic with a firewall, but application servers must be continually reachable in order to listen for incoming requests. Any interruption of availability causes errors like “connection refused” and “service unavailable” for the user. 

    This necessary network exposure leaves the upper layers of the network and the application server itself vulnerable to intrusion and denial of service attacks. “Private” enterprise applications are similarly exposed to the famously-vulnerable group that are VPN users’ devices, and so are by no means immune to this problem. This presents the problem of automatically discerning which incoming connections are legitimate, continually, forever. Are we trying to solve the wrong problem?

    Ideally, we would identify the sender before they connect to the application server. That way, we can know whether they have permission to connect at all. This is something we simply can not do when the sender is known only by a network address. We can’t ask for a credential because that introduces yet another application layer with all of the same problems. This is precisely why VPNs do not go far enough: A superficial barrier is erected for users that prevents the application server from checking for permission until after surveillance and subsequent attacks have occurred. 

    Embedded connectivity-as-code solves the right problem. This represents a shift in thinking about the relationship between the application and the network. Now, the application brings its own point-to-point overlay with a verifiable identity on both ends. This has significant implications for the security of the application that can be summarized in terms of shifting security left in the development timeline and shifted right in the life span after deployment.  

    Shift left: OpenZiti makes embedding connectivity-as-code convenient for developers and transparent for auditors and users. As open source software, it fits into DevSecOps processes that scan the source code. The management side of the OpenZiti network is cloud-native and API-driven and fits into the DevOps paradigm of continuous integration testing. For example, a network-as-code may be continually created and destroyed by a pipeline to exercise any imaginable configuration of endpoints, services and policies, or to facilitate dynamic scanning of the data edge and fabric APIs. 

    Shift right: OpenZiti allows us to shift right security to measure and monitor applications after development, in production, with a novel ease and without compromising security. Operators maintain perfect visibility of deployed applications and need not cope with the complexity of hopping through security gateways or bastions or worrying about external malicious actors and inbound connections. This best-of-both-worlds reality is a natural effect of  an access control scheme that is based upon verifiable cryptographic identity, not a weak identifier like an IP address. Importantly, this means it is always knowable which identity is accessing what services, when they were accessed and to what extent.

    OpenZiti gives applications superpowers! To name a few: Identity-driven addressability, private domain name system, shelter from unauthorized connections, portable and programmable micro-segmentation, end-to-end encryption, solves the IPv4 shortage without IPv6, smart routing to choose optimal routes across the internet and is transparent to end users.

    Learn more about OpenZiti by checking out our Quickstarts or engaging in conversation on Discourse. 

  • Improving Software Security in 2022

    Improving Software Security in 2022

    The recent Log4j vulnerability showed just how quickly a security bug could disrupt not just an industry, but the entire world.

    Organizations, especially federal agencies, will always find themselves at some level of risk, but they can also do more to mitigate those challenges. In November 2021, the Biden administration issued a directive through the Department of Homeland Security’s Cybersecurity and Infrastructure Security Agency (CISA) to fix hundreds of software and hardware vulnerabilities. 

    While this effort created an immediate call-to-action to patch known security risks, organizations from all industries must constantly manage known and unknown threats. A study released in July 2021 found it took organizations an average of 205 days to fix critical vulnerabilities, a timeframe that provided bad actors a wealth of opportunity to conduct serious damage.

    Agencies must adopt a more proactive approach to cybersecurity by improving software code quality. Let’s look at how to do that.

    An Ounce of Prevention

    Shaking up the status quo does tend to raise a few eyebrows, but the truth is that security programs should be in a constant state of continuous improvement. Today, our security programs work in an overly reactive state that relies too heavily on mitigation once a threat has emerged.

    Emphasizing a preventative approach may not be widely understood outside the security team, especially when an agency has a relatively clean security record. It might be seen as something that isn’t broken and, therefore, doesn’t need fixing. In this instance, getting leadership buy-in at all levels of an agency is important.

    Some pertinent points security executives should emphasize to change security culture inside their organization include:

    • The time and cost savings achieved through preventative measures, such as role-based training and related tools, as opposed to the potential cost of a critical incident.
    • Finding and fixing software vulnerabilities as code is written keeps releases on time with fewer showstoppers from the security team.
    • Preparing for and preempting potential security risks from the development team to release saves time and money overall. 

    Shifting Left of Left

    Shifting left has been popular in Agile and DevOps environments for over a decade. It involves testing small software components as early as possible rather than waiting until the end of the sprint.

    To create more secure code, organizations need to start left of left, eliminating common vulnerabilities as early as possible to create a safer user experience. 

    Starting left of left is a developer-first concept and requires organizations to get serious about uplifting their engineering cohort, placing emphasis on the creation of high-quality code. Security-aware developers are worth their weight in gold, and they need support in the form of job-relevant, hands-on training in secure code as well as the ability to provision the right tools. The opportunity to be mentored by more experienced developers will also foster an environment where code is crafted with a security-first mindset and the precision required to take software to the next level.

    One key area that often gets overlooked is the user experience; particularly with regard to how users access information.

    Security misconfigurations accounted for 21% of cloud-based data breaches in the past year, and amateur-hour errors (like storing passwords in plaintext, for example) resulted in serious productivity and customer trust losses. To avoid these errors, aim for a secure user experience that weaves tight security into a flow that makes sense. Adding more barriers—complex password requirements, a CAPTCHA, a horde of flesh-eating zombies—can turn users away. On the other hand, getting too permissive with security measures renders the entire point moot.

    A successful, secure user experience needs to weave tight security into a flow that makes sense, presented in a way that doesn’t detract from anything compelling about the software. 

    Improved Developer Upskilling

    Developers, of course, want to write secure code, but they often lack those skills or need a refresher course. Meaningful training too often gets overlooked as the day-to-day needs of organizations allow for little time to improve skills.

    In working with developers, we’ve found that about 75% prefer structured on-the-job learning instead of opening a manual. They would rather learn by doing and want training focused on practical applications, something that most current training programs are missing.

    Look for any and all opportunities to upskill your developers. They are on the front lines when it comes to stopping vulnerabilities. Provide them both the time and right-fit resources, understanding that these efforts will pay considerable dividends in the future.

    Government agencies, in particular, face inherent challenges in keeping their systems secure. They often must manage outdated systems with few financial resources and fight for top talent in an incredibly competitive market. There is no panacea for removing vulnerabilities, but taking a proactive and preventative approach that emphasizes the customer experience with highly skilled developers can help agencies take significant steps forward.

  • 2022 Will Be the Year of the Cyber ‘Shift Show’

    2022 Will Be the Year of the Cyber ‘Shift Show’

    As ransomware continues to plague organizations across industries, remote work continues and innovation brings with it new threat vectors for bad actors to exploit, 2022 is poised to be another messy year in cybersecurity.

    In July 2021 alone, organizations lost $45 million to ransomware. What’s more, ransomware costs are projected to reach $265 billion by 2031. The firm Cybersecurity Ventures predicted that “there will be a new attack every two seconds as ransomware perpetrators progressively refine their malware payloads and related extortion activities.” But ransomware isn’t the only cybersecurity concern business leaders should be thinking about heading into the new year.

    As businesses increasingly turn to digital channels like applications and websites to drive everyday business and customer interactions, those endpoints present new threat vectors for bad actors to potentially exploit. According to Okta’s 2021 Businesses @ Work report, large organizations (think AWS, Zoom, Box) deploy an average of 175 consumer and developer-facing applications, with smaller companies averaging slightly less at 73.

    What’s more, upticks in malicious activities like application cloning, credential harvesting and app spoofing will continue to cause headaches. In 2021, organizations and consumers across all sectors were forced to confront new and emerging bad actors in cyberspace. One example: The Android banking Trojan SharkBot (first discovered in October 2021), could implement overlay attacks to steal login credentials and credit card information and could intercept banking communications sent via SMS. To date, the botnet is thought to have targeted banking applications and cryptocurrency exchanges across the UK, U.S. and in Italy.

    But there is a way that organizations can bolster their cybersecurity wherewithal today to future-proof their business against the cybersecurity threats and concerns that will proliferate in the new year: It starts with “shifting left” and embracing DevSecOps within your organization.

    Bracing for Impact

    One of the ways organizations can better prepare and defend their organizations against the rise of cyberattacks to come is by adopting a developer security philosophy known as shifting left. In other words, building security into your DevOps strategy from the start. In industry circles, this practice is also known as embracing DevSecOps.

    DevSecOps involves cultivating a flexible partnership between engineers, operations and security teams to build security into all DevOps processes. Essentially, aiming to mitigate the bottlenecks of older security models on the CI/CD pipeline. DevSecOps also requires increased communication and collaboration between development, IT and security teams to ensure that security practices like testing are done in iterations throughout the code development life cycle. According to Gartner, by 2022, 90% of software development projects are projected to leverage DevSecOps practices—a 50% increase from 2019.

    But to be sustainable, cybersecurity needs to be embedded into all facets of the DevOps process—from production to testing to deployment. But even then, it’s important to remember that DevSecOps alone isn’t a security silver bullet. It’s a starting point. And as cybersecurity becomes all the more essential to bolstering business success and resiliency, it’s the least your organization should be doing in 2022.

    DevSecOps Alone Won’t Cut It

    While DevSecOps serves as a mission-critical foundation for scaling business resiliency, it won’t equip your organization to withstand the ongoing onslaught of attacks in cyberspace alone. Since most of the time, businesses are fighting nations in cyberspace—and far too often, that’s a losing battle.

    In 2022, as threat vectors widen and worsen, it will be essential for organizations to innovate with cybersecurity top-of-mind, across all facets of the development, deployment and production pipeline. Adopting DevSecOps practices across your organization, or shifting left, is one way organizations can bolster business resiliency as we head into yet another unpredictable year in cyberspace.

  • Shift Left: Right or Wrong?

    Shift Left: Right or Wrong?

    The term DevOps emerged into mainstream software development communities roughly 10 years ago. It represented a combination of tools and techniques which helped solve many pertinent issues of the time, namely responsibility for code in production and lack of a positive feedback loop. As of today, DevOps is a massive movement. It is considered the default way in which software development should be practiced. It has allowed even small teams to function with the same efficiency as much larger ones. Several engineering teams have adopted derivative approaches of the DevOps methods in order to suit their niche workflows.

    The Dark Side Of DevOps

    DevOps in practice presents a lot of tangible benefits to software engineering teams. Automation, transparency, and velocity are the chief advantages. From a business perspective, engineers are now more invested and accountable for the experience of users. The era of “Your Success Is Our Success” has finally arrived for engineers—and business teams couldn’t be happier. 

    For the individual developer, the number of things they were responsible for was piling up. More objects to monitor in production. As more features were getting built and deployed, alarms from internal systems kept going off more frequently. Individually, an engineer became responsible for the behavior, performance, security, compliance and so many other aspects of the application and underlying systems. Often, this became a distraction and began to defeat the very purpose of the DevOps methodology which meant to remove hurdles that were in the way of developers. 

    This is DevOps done wrong. Lack of maturity within software engineering teams has caused a lot of people to go down a DevOps path that is toxic and consequently forced engineers to burn out quickly. The core tenet of DevOps is to enable engineers to do more, but not to the point where they’re having to undertake enormous toil. DevOps still recognizes two distinct forms of specialized effort – software development and platform operations. The nature of services, scaling, and security may vary the degree to which dev and ops responsibilities converge. 

    Dev-Oops: It Goes Wrong Over Time

    Interestingly, engineering teams view DevOps as development, and “everything else”. From the time a developer checks code into version control, there is a large assortment of workflows that kick in. Ideally, there is testing, integration, more testing, deployment and feedback. This is a large body of work, often underestimated and underrepresented. To pile this responsibility onto members responsible for development alone is where things begin to go wrong. Often, as a product ages, incident management, maintenance, and other engineering efforts pile up–resulting in fewer cycles being available for product development over time. 

    There is also the complementary case, where engineering teams are focused on product development and assign most free cycles to it. Platform operations are forced to become a watered-down effort and consist of the bare minimum it takes to keep the lights on, so to speak. This is a common observation in small teams and startups that typically have a new product release. 

    When implemented correctly, DevOps must allow engineers to take responsibility for all aspects of developing a feature and managing it in production. This means that programming the business logic, creating the declarative and non-declarative software, binding with services, building the artifact, and finally deploying to production all come under the gamut of a developer. In addition, a feedback loop—created by baking observability into the mix will provide insight into the health of the service (or feature)—will require the engineer to stay on top of the operational aspects of performance and reliability. 

    The current definition and practice of DevOps originated from Agile methodologies. They take off from the limitations of typically Agile teams which were released in “sprints”, still kept dev and ops efforts separate, and were agnostic of underlying tech. DevOps, on the other hand, is more tightly-coupled with the technology employed, for example: Microservices and automation. It facilitates continuous deployment and a high frequency of releases. Most importantly, it converges the notions of development and platform operations. We find that the interpretation of this “convergence” is what differentiates good implementations of DevOps from bad ones. 

    The Right Kind of Shift Left

    At the heart of healthy DevOps adoption lies a technology shift complemented by a cultural shift. The fundamental principle that belies successful teams is a great developer experience. Irrespective of the infrastructure that powers the technical stack of a web application, software engineers who work on it need to be able to develop and deploy with ease. Let’s explore how. 

    Examples of tools that aid engineers during the development phase include IDEs, Source Control, TDD suites, documentation generators, and many more. Tremendous amounts of research and product development are currently underway in enabling developers to program better and facilitate more efficient development. 

    Different skill sets and focus areas are required to optimize different portions of the operations spectrum. In the case of large enterprise organizations, the dichotomy is clear. There are Platform Operations teams that are responsible for keeping all the systems running by providing automated means to set up and sustain reliable systems on the server side. Application Development teams are tasked with creating the applications, API endpoints, integrations and various other artifacts that will enable users to execute their tasks. 

    Because of the common underlying nature of both these functions–writing software–they tend to cross-pollinate. This coalescing is welcome, provided it is managed well. This is where abstractions come in handy. In particular, PaaS abstractions that provide in equal parts agility and intransigence. Pliant enough to accommodate custom workflows and bespoke needs of software products and teams, and yet preventing needless osmosis between Dev and Ops responsibilities, ultimately resulting in a superior developer experience as the nucleus of this atomic transformation.

    PaaS With Flying Colors

     

    The PaaS-low’s Hierarchy! 

    The perfect balance can only come from PaaS systems that comprise an architecture that is modular and offers the functionality of application build and deployment. The ability to work across multiple languages and frameworks combined with packaging and deployment best practices is a basic tenet. Abstracting the underlying infrastructure to favor support for container-based artifacts and orchestrating them at scale is the next most desirable component of a PaaS framework. A batteries-included approach to automating continuous delivery, security best practices, and other workflows goes a long way in ensuring that the PaaS substrate works for all kinds of developer needs. When using abstractions, the biggest opacity comes in the form of a lack of production parity. If the PaaS can be functional on all staging instances viz. dev, test, pre-production, prod, or any other remote instances, it would save tremendous toil arising from a lack of parity. 

    Bonus points for being open source and accommodating compliance, policy mandates, and customization to suit the dynamic needs of different organizations. An active community supporting the ecosystem and driving innovation and change management would make a more compelling case for PaaS tools. 

  • Secure Software Summit: Exploring Secure Coding Best Practices

    Secure Software Summit: Exploring Secure Coding Best Practices

    In an era where software is dominating the world, the security and quality of code must remain a high priority. Delivering secure and reliable software at a rapid pace is crucial for most organizations today, but it’s not an easy task. What security best practices do organizations put in place to successfully produce secure software? How should DevOps teams incorporate updated security practices and tools throughout the software development lifecycle? At Secure Software Summit, taking place Jan. 27 at 9 a.m. PT, the world’s leading cybersecurity experts will come together to share and teach the latest methods and breakthroughs on secure coding and deployment practices.

    The virtual event features thought-provoking discussions, networking opportunities and educational sessions focused on:

    • Open source supply chain security
    • DevSecOps
    • Shifting security left
    • Security Chaos Engineering (SCE)
    • Security and compliance
    • Security metrics
    • Zero trust

    Secure Software features an outstanding lineup of innovators, practitioners and academics of secure software development, including:

    • Aaron Rinehart, co-founder and CTO at Verica
    • Dan Lorenc, founder and CEO at Chainguard, Inc.
    • Shinesa Cambric, principal program manager, Identity Signals and Protection Team at Microsoft
    • Vickie Li, developer evangelist at ShiftLeft
    • Fabian Yamaguchi, chief scientist at ShiftLeft
    • Jonathan Schneider, co-founder and CEO at Moderne, Inc.
    • Abhishek Arya, principal engineer and manager, Google Open Source Security Team at Google
    • Steve Springett, senior manager of product security at ServiceNow and chair of the OWASP CycloneDX Core Working Group
    • Jasmine Jackson, senior application security engineer at Disney
    • Suchakra Sharma, staff scientist at ShiftLeft
    • Harini Rangarajan, engineering manager, product security at Twilio
    • Mark Wireman, senior manager at Accenture
    • Shannon Lietz, vice president of Vulnerability Labs at Adobe
    • Vandana Verma, security relations leader at Snyk
    • Yash, head of security at Sendbird
    • Chetan Conikee, CTO at ShiftLeft
    • James Chiappetta, SVP of cybersecurity at Blackstone

    Attendees will have access to valuable resources available for download and will also receive a link to access the virtual environment on-demand. To see the full agenda and to register, please visit the Secure Software Summit website.

  • Best of 2021 – The Next Evolutions of DevOps

    Best of 2021 – The Next Evolutions of DevOps

    As we close out 2021, we at devopsy.kinsta.cloud wanted to highlight the most popular articles of the year. Following is the seventh in our series of the Best of 2021.

    As society continues to recover from the global COVID-19 pandemic and some semblance of normal starts to return, technology continues to move forward. The pandemic certainly changed what normal means and people, processes and technologies all have adapted and evolved to meet those challenges. The narrative that the pandemic caused of your organization’s digital transformation is true for many practitioners. With 2022 just around the corner, DevOps is on the cusp of maturation of previously bleeding-edge paradigms and a focus on engineering efficiency. Here are five DevOps evolutions you should keep an eye on going forward.

    Engineering Efficiency is Front and Center 

    Engineering efficiency is a particularly broad umbrella term. At the foundational level, engineering efficiency focuses on making someone (for example, an engineer) more productive. There are a multitude of disciplines that intersect, from organizational design to engineering/developer experience. Engineering efficiency is increasingly critical for organizations today. The pandemic had two major impacts on the technology sector in terms of resources and staffing. The first, at the start of the pandemic, was the unprecedented and unpredictable availability of resources; physical locations were being shut down because of the medical severity of the pandemic and the “getting hit by a bus” factor was playing out in real life. Second, during the recovery period, the Great Resignation began, and with it, a fight for resources that has become increasingly intense. Both have meant that the availability of resources, especially ramped resources, is scarce. 

    The Spotify model of tribes/guilds of engineering resources is a promising evolution; folks can frequently move around, reducing the toil that engineering resources face and allowing for engineering resources to not only ramp up more quickly but also increasing engagement and retention. The industry continues to march toward standardization in areas that were typically bespoke, such as the release process, and I think we’ll see that continue to evolve, as well.

    Git is Ubiquitous

    Leveraging a source code management (SCM) solution used to be reserved for software engineers. But as the proliferation of “something-as-code” has touched multiple levels of the technology stack from networking, storage and, ultimately, a development pipeline, the march of operations-focused engineers adopting many of the traits of software engineers continues. Having iteration in your infrastructure stack with more ephemeral/disposable infrastructure is becoming the norm. Saving and packaging the multitude of configurations for the “something-as-code” source control is a natural area of evolution. 

    Building off of source control are the package managers—your Docker registries, Helm chart repositories, etc.—for more deployable artifacts; that said, getting those artifacts to production is a process. Because Git is becoming ubiquitous across the technology landscape, this leads to the continued adoption of GitOps. Since the seminal piece was written by Weaveworks in 2017 defining the GitOps paradigm, leveraging GitOps is seen as a viable paradigm for many organizations five years later, especially when starting with greenfield initiatives. 

    Kubernetes is No Longer Bleeding Edge

    In 2022, Kubernetes will celebrate the eighth birthday of its first commits on GitHub. Taking a jog down technology memory lane, eight years before Kubernetes (in 2006), VMware was still a private company and it would be a few years before vSphere was even released (remember how it felt to run a workload on a virtual machine in 2014?). The Kubernetes ecosystem still moves quickly, but placing workloads on Kubernetes is not a novel concept anymore. As application infrastructure and architecture have adopted the Kubernetes way (in other words, being idempotent and ephemeral), there is maturity around running a suitable workload on Kubernetes.  

    Kubernetes by design is highly pluggable. If you do not like the opinion or implementation of something inside Kubernetes, you can replace that opinion. Don’t like how Kubernetes handles ingress traffic? Then choose from one of the many ingress controllers available today. That is just one example of dozens of pluggable areas. Because of this, Kubernetes is viewed as a dynamic, non-static resource. Using Kubernetes takes trial and error and getting your cluster to be robust, performant and reliable is an ongoing journey that requires ongoing iteration. 

    Authors Can Be the Enforcers

    There is no question that in a modern software stack, we are inundated with data. The metrics soup and the continued science of and exploration of observability (metrics, traces, logs) provide us with a lot to work with. Making decisions, even automated ones, based on that data continues to be critical. With the rise of site reliability engineering (SRE) practices, reacting or anticipating spikes in usage, failures and even security-related items are par for the course for modern teams. 

    These decisions can be authored into policies that specific platforms understand, such as autoscaling rules for a cloud vendor or something like Open Policy Agent in the Kubernetes ecosystem. With so many items shifting left toward the development team, finding the right resource, skillset or institutional knowledge to author these rules can involve the input of several teams. Because of the rise of “something-as-code,” regardless of whether the author of these rules is a software engineer, DevOps engineer, platform engineer or anyone in between, that author has the ability to enforce. 

    Shifting Left … But Complexity Shifts Left, Too

    The old adage is that complexity is like an abacus: You can shift complexity around, but it never really goes away. With the movement to shift responsibility left to development teams, this also means that associated complexity is shifting to the development teams. Modern platform engineering teams provide the infrastructure (compliant Kubernetes clusters) to teams and any workload that is run on those clusters is up to the development team that owns it.  Typically, development teams then focus on features and functionality. Managing lots of non-functional requirements—and even core infrastructure requirements such as networking—can be a burden; think about how your organization would handle a service mesh.

    If you are a DevOps or platform engineer, making your internal customers—your development teams—successful is a great goal to work toward. Crucial to this is disseminating expertise. This can be in the form of automation and education. A common practice with the DevSecOps movement is to have some sort of scanning step as part of the build or deployment process, disseminating the internals as far as how the scan is performed, what happens if something is found, etc. Gaining internal adoption is a journey and having a good developer experience built around clarity and stability are important. 

    Evolution in 2022

    Even with all of the challenges, bumps and learnings that the last year has given us, technology continues to evolve and advance and even lower the barriers of entry to continually become more inclusive. Focusing on improving engineering efficiency and reducing toil will allow for more participation and lower the barriers to entry.

     

  • Does Shift Left Matter to SREs?

    Does Shift Left Matter to SREs?

    If you’re a software engineer, you’ve likely heard all about shift left, a practice that can streamline certain aspects of software development.

    But shift left isn’t just for developers. It can be equally valuable for site reliability engineers (SREs). Although the main mission of SREs is ensuring the reliability of software after it has been deployed rather than actually developing software, encouraging shift left practices as part of a software development strategy can nonetheless make it easier to optimize reliability.

    To prove the point, here’s an overview of what shift left means and why it matters for SREs.

    What Does Shift Left Mean?

    Shift left is the detection of application problems early in the software development process. The core idea behind shift left is that instead of waiting until just before deployment to test software for bugs, security issues or other problems, teams should begin the testing process as early as possible. In other words, they should shift some testing to the “left” of the software development life cycle instead of testing only in the middle—hence the term “shift left.”

    In so doing, they can detect problems early in the development process, at which point they are typically easier to resolve. If you don’t catch a performance, reliability or security bug until just before you plan to push your application release into production, you may have to overhaul a large part of your codebase to fix the issue. In many cases, you’ll have to change not just the specific code that triggered the bug, but also other code that depends on or integrates with the buggy code. But by finding buggy code early, developers can often fix the problem with minimal adjustments to the application as a whole.

    Shift left is a high-level concept, and it can be implemented in multiple ways. In some cases, developers may start running performance tests on new code right after they write it, even if it has not yet been integrated into the main codebase. In other cases, shift left testing could mean compiling some parts of the codebase in order to run tests against it before the application as a whole is built and tested in a dev/testing environment. Either way, the result is a testing process that begins earlier than it would under a conventional approach.

    Importantly, shift left doesn’t mean that teams skip the tests that they would normally perform just prior to deployment. Those tests, which are important for catching bugs that materialize only when testing the application as a whole rather than individual parts of it, typically still happen as part of a shift left strategy. Thus, shift left means that testing begins earlier and coincides with a broader portion of the overall development pipeline, not that testing is simply shifted to the beginning of the pipeline at the expense of mid-pipeline tests.

    Shift Left and SREs

    Again, the main job of SREs is to manage the reliability of systems. They may spend part of their days working alongside software engineers to help devise architectures and coding strategies that maximize the resiliency of applications against reliability and performance issues. But SREs also devote much of their time to managing what happens post-deployment. They use monitoring and observability tools to detect problems that arise in production environments. They then take the lead in responding to them.

    In that sense, SREs may not seem to have much to gain from shift left. Shift left is a strategy that aligns with software development first and foremost, and software development is not the core focus of SREs.

    Nonetheless, a shift-left strategy can help SREs do their jobs better, for several reasons:

    • Fewer problems in production: Most obviously, shift left reduces the rate at which reliability issues reach production environments. In turn, it minimizes the number of incidents SREs need to respond to.
    • Optimizing software pre-deployment: By exposing reliability issues early in the development process, shift left is one way for SREs to acquire the insights they need to collaborate with developers on building reliability into applications. Shift left could highlight buggy dependencies or suboptimal architectural patterns, for example, which SREs can encourage developers to address.
    • Granular reliability insights: Because shift left is a great way to trace software problems to the specific code that triggers them, it provides highly granular visibility into reliability. It can help SREs find the weakest links within an application’s codebase or architecture. These weak spots can be harder to detect when monitoring or observing an application as a whole.

    How SREs can Follow the Shift Left Methodology

    Because the implementation of shift left practices is a task that ultimately falls to developers, SREs need to work closely with development teams (and DevOps engineers, if they exist in the organization) to make shift left happen.

    The first step in that process is to get buy-in for shift left among developers. SREs can do this by highlighting the ways in which shift left can streamline the work of developers and reduce the complexity of fixing bugs. That’s important to communicate because it ensures that developers understand that shift left benefits everyone, not just SREs.

    SREs should also work with developers to identify the best approaches and practices to implementing shift left. A key factor to consider here is where reliability problems most often arise, and how shift left can help to detect them earlier. If most performance problems stem from code written by individual developers, for instance, testing code as soon as it is generated may be the best way to detect problems early on. In contrast, in situations where performance issues are triggered most often by environment configurations, the ability to test applications early under different environment variables could be the best way to surface issues early in the development process.

    Finally, SREs should partner with developers to respond to the issues that shift left reveals. After all, while developers typically don’t pay close attention to what happens post-deployment, SREs have a special ability to understand how problems in development translate to problems in production. SREs may therefore be able to help developers identify the most effective way to fix bugs based on production environment requirements, even if it’s not the quickest or simplest way.

    Why SREs Need to Shift Left, Too

    Although the shift left methodology has not traditionally been closely associated with SRE, shift left is a crucial tool in the SRE toolbox. By helping teams to detect reliability issues when they are easier to resolve, shift left puts SREs in a stronger position to maximize the overall reliability of applications. At the same time, it reduces the number of production environment issues that SREs have to manage.