devopsy.kinsta.cloud

Category: DevOps Practice

  • How IaC Helps Relieve Development Pain Points

    How IaC Helps Relieve Development Pain Points

    Developers spend much of their day digging into their systems’ guts, allocating storage to an application, connecting the security system to the user interface and tracking which version of the application is where in the development cycle, among other tasks.

    That’s why a new category of software has emerged, infrastructure-as-code (IaC), that promises to automate system infrastructure provisioning and integration. It frees up time so developers can focus more on frontend solutions rather than backend configuration management. First, though, let’s find out what IaC is and how it helps your organization.

    What is Infrastructure-as-Code (IaC)?

    IaC reduces system infrastructure creation and configuration time through the use of sets of tools, protocols, languages and processes. In conventional data center system management, each configuration entry and change has historically required manual action. Now, technology teams can manage everything in an intelligent, automated manner. 

    With IaC, infrastructure configuration management data is stored within standardized files. The tools monitor and maintain the state of the infrastructure and ensure that the right resources are available whenever an application runs. This saves DevOps teams from excessive manual effort. In essence, large-scale configuration and management become dramatically simplified.

    Development Pain Points 

    Currently, development is a cumbersome process because friction arises in many places. A great deal of progress has been made in speeding up the front end of application development—building the feature set. Still, the process slows down on the back end when the software components are tied to the computer infrastructure. 

    Historically, software was deployed as one large, monolithic block of code. Updates were tedious to make because all of the application’s code had to run, even if it wasn’t involved in the update or bug fix. The advent of containers and microservices enabled developers to design applications that are modular in nature and therefore easier to update, add to or debug.

    However, the application still has to be tied into the existing compute infrastructure. Cloud solutions—like AWS, Google Cloud Platform or Microsoft Azure, for instance—provide companies with quick access to large volumes of computing processing power. But that application software must be told how to access the network, present security information, allocate storage and run code on central servers.

    Because of this, developers spend a lot of time building and maintaining infrastructure-as-code. To do so, they use languages like Python and solutions such as Chef and GitHub. They want to use automation to simplify and accelerate the process, but here, again, they encounter some roadblocks.

    Workflow Bottlenecks

    As applications become larger and components become smaller, developers find themselves mixing and matching software from a wide variety of places. Tracking what’s where in the process is a frequent challenge. Current workflows are often non-intuitive and rely heavily on pull requests. Some solutions try to address the problem with multiple workspaces, but the end result is fragile and nondeterministic code.

    Without mapping of projects to branches or tags, someone who comments on an approved pull request can push unapproved code to production. In today’s fast-paced DevOps work environment, an approval meant for a short-lived test run becomes the current release, creating a ripple effect of unintended consequences that create confusion and defective code, slowing development—often substantially.

    Access Control

    Most tools do not offer access control models. Developers can track a single repository and a single Terraform project, for example, but as the number of repositories and projects grows, ensuring that proper access control features are in place becomes more daunting. 

    Workspace Challenges

    Legacy waterfall and static development projects allowed IT teams to set their system infrastructure up once and forget about it. But in a dynamic environment, microservices come and go in the blink of an eye. New environments proliferate and product teams must continually configure their own workspaces, which eats up time and resources.

    Drift Detection

    Generic IaC platforms don’t provide mechanisms to detect if the infrastructure is drifting. Because so much configuration is done at runtime, differences arise between the desired and the actual state of the infrastructure.

    Drift has many causes, such as developer or machine script contingencies or the lack of system resources. Too much drift results in unstable applications. Catching such problems early in the development cycle is more efficient than trying to fix a production system. But needed visibility is often lacking, and shortcomings become clear only when the system is fully in use.

    Resource Visualization

    Given today’s dynamic environment, developers need clear visibility. They must be able to closely examine what happens as their job runs. Though they offer many advantages, CI/CD platforms deliver little insight into resource usage from a real-time or historical perspective.

    So, whenever performance problems arise, these platforms offer little insight into what the source of the problem is or how to fix it. Consequently, developers’ attention shifts from application enhancements to troubleshooting.

    Lack of Reusability

    Legacy infrastructure failed to let teams reuse tried-and-true system images across multiple environments. To address this problem, teams often create a new image for each version of each application, slowing development time. 

    High Prices

    Setting up each IT environment manually requires time and manpower, which raises costs. Companies must have dedicated teams of engineers responsible for creating the infrastructure. But they also need supervisors and data center resources, which ultimately boost technology expenses. IaC tools centrally manage the process and create the right environment. That way, organizations pay only for system resources that they truly need.

    How Does IaC Improve the Business? 

    Legacy IT practices are expensive, slow and inconsistent because they rely heavily on manual input. Infrastructure-as-code automates much of the infrastructure management process, which delivers a range of benefits.

    Speed 

    With IaC, provisioning and management are lightyears faster and more efficient than with a manual system. The environment’s infrastructure is defined in code. That code-based infrastructure significantly accelerates each phase of the software development and delivery cycle. Consequently, DevOps teams deliver applications faster and at scale while improving software quality.

    Consistency

    IaC deploys consistent configurations constantly and with no discrepancies. Because of this, it enables fast delivery of consistent infrastructure and environments. This reduces errors and overhead while boosting productivity.

    Eliminating Management Overhead

    Infrastructure automation helps to reduce the number of people with hands on the routing parts of the software development process. That’s because human resources aren’t needed in the management of networking and storage. Companies free up personnel to work on more strategic initiatives.

    Reducing Risks and Errors

    IaC helps to reduce risks and avoid human errors. It also protects against runtime issues and security weaknesses caused by missing dependencies or configuration drift.

    Visibility

    Developers version IaC configuration files the same way they do any other source code file. This gives them complete traceability of any changes. In this way, IaC ensures efficient, exceptionally transparent workflows and speeds up the development process.

    Finding an IaC Solution 

    Maintaining legacy infrastructure was a time-consuming, complex process. IaC is emerging as a solution that alleviates much of the stress and pain that developer teams feel daily. Now, they can speed up the deployment and connection of system resources to applications. Developers can become more efficient and enable the company to respond to business demands in a faster, more reliable and more cost-effective manner. The development process no longer slows down the business. 

  • Embrace Change in Production to Improve MTTR

    Embrace Change in Production to Improve MTTR

    DevOps creates value through constant deployment changes. When done right, more frequent deployments allow teams to continuously improve their products or services while making it easier to manage the effects of the change. One way to do this is with continuous integration and continuous deployment (CI/CD). CI/CD helps to reduce the risks to each build and allows teams to release valuable features quickly and safely. But CI/CD alone is not sufficient for understanding change and service impact in production. The recent DEJ State of IT Performance Report found that change is the largest source of production issues. A full 76% of all performance problems can eventually be traced back to changes in the environment but 67% of organizations don’t have the ability to identify changes in their environment that caused performance issues.

    In a modern IT environment, different teams are constantly integrating new features and new technologies and renewing their stack. With many moving parts, it’s impossible to mimic the production environment when testing new features in pre-production. By the time you’ve implemented and tested new code, your branch will be different from what is in production.

    Here’s the reality we’re working in:

    ● Integrating testing in CI/CD pipelines will not keep bugs out of production. Some organizations don’t do true CI/CD, meaning some processes are not even tested and automated in the pipeline.
    ● User configuration errors are common. Here’s a consolidated list of configuration errors that brought down production.
    ● Change anxiety is real: Ops folks get nervous when hitting the deploy button, even for small changes.

    Changes are inevitable in DevOps, but organizations can protect themselves from a performance incident or an outage by integrating change in production safely with a few simple steps.

    Build Data Connections

    Today, data is kept in silos, which is why it’s time-consuming and challenging to make associations and leverage them to find the root causes of performance issues. DevOps teams need to connect their data in meaningful ways to troubleshoot complex issues in distributed systems. Building the “connections” foundation is critical to understanding the state of a system and how the pieces interconnect and interact.

    How well an application performs in production is related to a combination of code, infrastructure and operations. For instance, when an incident occurs, teams should quickly know if that issue is related to code, infrastructure or operational activity. Understanding these connections and correlating the data cannot be performed manually. We have hit the wall in linking together data silos; we just don’t scale as software does. Teams need to consider a monitoring and analytics approach that connects operational data together from the start and models the causal links between the data for real-time root cause diagnosis.

    Understand Change Impact to Avoid Outages

    How well can your DevOps team identify the key changes made to your production environment? If your answer is “not very well,” then you’re missing critical data that helps to speed up troubleshooting. The lack of visibility into changes and how they impact production makes it impossible for operations staff to identify the cause—the very thing that SREs care most about.

    Changes are made to the code or the cloud infrastructure, either directly or through infrastructure-as-code (IaC) templates, for a variety of reasons. The risk of misconfiguration and performance issues increases when these operational activities are not continuously monitored and correlated across the stack.

    Metrics don’t tell the whole story. Logs don’t pinpoint where to act. Simply collecting these data types leaves many companies struggling with knowing what is important and what’s not when it comes to real-time troubleshooting.

    Having a shared “system of record” for changes for all the different streams of activities increases visibility and control of all changes in real-time. Teams can see pre-incidents and be aware of increased risks when changes are made, especially during high-velocity change periods. By seeing how a code change can impact things down the line, developers can proactively troubleshoot and debug code.

    Implement Real-Time Root Cause Diagnosis

    In the same DEJ report, organizations shared that 66% of MTTR is spent on the ‘diagnose’ stage. This correlates to the 67% of teams that are unable to identify the changes in their environment that caused performance issues.

    With a shared “system of record” for changes, DevOps teams can troubleshoot incidents in real-time, dramatically cutting the time spent on the “diagnose” stage of MTTR. This is especially effective when teams can see all the changes and other events that are happening to the system in a single, searchable/filterable real-time feed; this timeline would provide the potential causes of issues.

    The best tool would allow you to start from an effect and work backward, chaining multiple events overlaid with observability data to determine the causes quickly. Building the causal chain from the end helps teams hone in on the connected change and answer each “why” in a straightforward way. This capability makes it easier and faster to prioritize relevant changes and resolve performance degradations or outages with actionable context.

    Communicate Everything

    During diagnosis and resolution, it’s important to see what’s being done without having to ask on Slack, wait for each other or unknowingly interfere with remediation actions. The best option would be leveraging the same timeline to capture all the context of an incident at the moment it occurs, and at any moment in time during diagnosis and resolution, to avoid reverting each other’s changes.

    While an incident postmortem brings teams together to gather data and discuss the details and learnings from it, this process is often inefficient and time-consuming. Each team member involved in the incident response effort would have to recollect what they observed, what actions they took and what they communicated to other team members. What if a system could capture all the actions and communications during an incident response effort? Wouldn’t it be nice to just summarize your thoughts in one sentence and be done with the reporting?

    When preparing for planned and unplanned changes that can impact the bottom line, DevOps teams need to consider these steps to gain true observability of their operations. Focus on finding the connections in the data to explain cause and effect. Understand change impact —as those changes are being made —to avoid outages. Capture and communicate activities as seamlessly as possible and identify and capitalize on opportunities to automate the workflow.

  • New Relic Survey Unearths Observability Challenges

    New Relic Survey Unearths Observability Challenges

    A New Relic survey found only 27% of respondents have achieved full-stack observability with only 5% claiming they have a mature observability practice in place. A third (33%) of respondents also said they still primarily detect outages manually or based on complaints, the survey found.

    The global survey polled 1,614 respondents, including 1,044 practitioners and 570 IT decision-makers and was published today by New Relic.

    On the plus side, the survey also found nearly three-quarters of respondents said C-suite executives in their organization are advocates of observability, and more than three-quarters of respondents (78%) saw observability as a key enabler for achieving core business goals.

    However, more than half (52%) of respondents said they experienced high-business-impact outages once per week or more and 29% said they take more than an hour to resolve those outages.

    Peter Pezaris, senior vice president for strategy and user experience at New Relic, said a big reason it takes so long to resolve issues is the time required to determine the root cause of issues using multiple tools. A full 82% of respondents said they use four or more tools to monitor the health of their systems. Only 7% said their telemetry data is entirely unified and only 13% said the visualization or dashboarding of that data is entirely unified. Nearly half of respondents (47%) said they would prefer to have a single, consolidated observability platform, but only 2% said they currently used one tool for observability.

    New Relic, of course, is making a case for a single observability platform that spans both the application environment and the underlying infrastructure an application depends on. Today, most IT teams are using a wide range of tools to monitor infrastructure and applications.

    Nearly all respondents said by 2025 they expect to deploy capabilities like network monitoring, security monitoring and log management, as well as Kubernetes monitoring. More than a third (36%) believe observability increases their productivity and enables them to find and resolve issues faster. Just under a third (32%) said observability enables cross-team collaboration, while more than a quarter said observability improves their ability to innovate.

    Nearly three-quarters (72%) expected to maintain or increase their observability budgets next year. More than half (52%) of respondents, including 57% of C-suite executives, expected observability budgets to increase over the next year. A total of 14% of all respondents and 16% of IT decision-makers expected to increase observability budgets significantly or extensively. IT decision-makers anticipate needing observability most for artificial intelligence (51%), internet of things (48%), edge computing (38%) and blockchain (36%) in the next three years.

    It’s still early days as far as the adoption of observability platforms is concerned, but it’s apparent that as application environments become more complex the plethora of monitoring tools that DevOps teams rely on will need to be streamlined. Observability platforms promise to unify logs, metrics and traces in a way that makes it simpler to launch queries to identify the root cause of an issue.

    The rate at which DevOps teams will embrace observability will naturally vary, but the biggest obstacle might not be the platforms themselves. Instead, the issue is understanding what queries can help DevOps teams better understand the root cause of an IT issue before there is a major disruption. In the long term, it’s expected that machine learning algorithms will leverage the data collected by observability platforms to automatically identify issues that might lead to a disruption long before it actually occurs.

    In the meantime, the hope is that monitoring tools will give way to observability platforms that will make managing complex IT environments much simpler for all concerned.

  • Survey Surfaces Massive Number of Application Vulnerabilities

    Survey Surfaces Massive Number of Application Vulnerabilities

    A survey of 16,510 IT and IT security practitioners published today by Rezilion, a provider of a platform for automating the remediation of software vulnerabilities, found nearly half of respondents (47%) worked for organizations that have a backlog of vulnerable applications. More than half (66%) said their backlog consisted of more than 100,000 vulnerabilities and the average number of vulnerabilities that need to be remediated is 1.1 million.

    The survey, conducted by the Ponemon Institute, also found more than half of respondents ( 54%) said they can only patch less than 50% of the vulnerabilities.

    Factors that keep teams from remediating included an inability to prioritize what needs to be fixed (47%), a lack of effective tools (43%), a lack of resources (38%) and not enough information about risks that would exploit vulnerabilities (45%).

    More than a quarter (28%) also said remediation is too time-consuming. The survey found that 77% of respondents said it takes longer than 21 minutes to detect, prioritize and remediate just one vulnerability in production.

    In addition, 85% said they spend more than 16 minutes detecting one vulnerability in development. A full 82% also noted it takes longer than 21 minutes to remediate one vulnerability in development. A majority of respondents said it is either very difficult (36%) or difficult (25%) to remediate vulnerabilities in applications.

    On the plus side, more than half (56%) said they use automation for vulnerability remediation, with 43% reporting times to respond are significantly shorter as a result.

    Rezilion CEO Liran Tancman said the survey makes it clear there’s still plenty of room for improvement. The number of vulnerabilities that need to be addressed is staggering, he said. IT teams need to focus their immediate efforts on identifying the vulnerabilities that are the most critical, he noted.

    Remediating those vulnerabilities will soon become a higher priority as more regulations focused on securing software supply chains come into effect. In fact, the software bills of material (SBOMs) that these regulations require will expose much of the dirty laundry that currently exists in application environments, said Tancman.

    Ultimately, the goal needs to be enabling organizations to address vulnerability issues without slowing down the rate at which applications are being built and deployed, Tancman added. The degree to which that goal can be achieved will vary widely as more responsibility for security is shifted left toward application developers. However, the need to provide tools to developers that enable them to address vulnerability issues has never been more apparent, noted Tancman.

    It’s clear the level of application security vulnerability debt that organizations will need to address can no longer be ignored. The issue now is to determine how best to resolve these issues either by fixing existing applications or replacing them with new ones. Either way, it’s clear that there is a need to adopt DevSecOps best practices to better secure software.

    In the meantime, as application security becomes a boardroom-level issue, there be a lot more scrutiny applied all across the software supply chain.

  • The Definition of Done: Why it’s Important for Success

    The Definition of Done: Why it’s Important for Success

    Establishing clear communications and expectations across software organizations is essential to the success of any project. A company is less likely to experience unexpected delays or other unpleasant surprises if everyone is on the same page. To ensure all team members are synchronized and to eliminate unnecessary ambiguity, engineering teams should create a “Definition of Done” to guide projects.

    What is the “Definition of Done”?

    In Agile software development, the definition of done is a working agreement between teams that defines a set of standards to identify something as “done.” Because Agile is an iterative approach to software development, sometimes it is unclear when a task is complete. Product managers often report that something is ready for delivery when it’s “Dev done” and not “QA done.” In other words, the code has been written, but no documentation or tests have been created or executed. For other parts of the software organization, code without QA or documentation is not “done.” Intuitively, the definition of done exists to eliminate confusion about what “done” means across an organization.

    How to Create a Definition of Done for Your Software Organization

    For some software organizations, the definition of done is a static set of guidelines and benchmarks established at the start of a project. For other organizations, including here at Allstacks, engineers have the latitude to amend the definition of done as they go. That approach empowers an engineering team to make decisions that will produce the best possible results.

    When establishing a definition of done within an engineering organization, consider these tips:

    ● Communicate criteria in a way that the whole software organization can universally understand.
    ● Ensure that the definition of done addresses both tactical and technical requirements.
    ● Create a living document for alignment that can be amended because of unforeseen circumstances, like losing a team member or a significant market shift.
    ● Consider that the robustness of the document will vary by team maturity, and specific teams might need an addendum for special requirements. For example, bigger enterprise IT teams frequently need more detailed documentation compared to two-person shops.
    ● Establish an understanding among all teams that, until all criteria are met, the user story, task or project is not “done.”

    A definition of done acknowledges the shared responsibility between the frontend, backend and QA. Beyond those roles, other stakeholders might contribute to defining “done,” like Scrum masters, board members or business analysts, depending on the organization and its structure. Further, the product owner may also define “acceptance criteria” with goals targeted toward the end-user experience that the engineering team must also address as part of their work.

    How a Definition of Done Can Help Teams be Successful

    The definition of done helps a software organization maintain alignment on projects and keeps everyone on the same page. Misalignment can have heavy consequences. For example, it can initiate premature conversations with customers before a product is really ready, create friction between internal stakeholders and/or risk other go-to-market initiatives. When the definition of done encompasses defined steps for a project down to the task level, Agile teams can create more cohesion across an organization and eliminate unforeseen delays.

    A definition of done outlines each task in a software engineering project. Noting the completion of each task can be an indicator of whether a project is on track to meet established deadlines. By noting the time it takes to complete each task, an engineering team can measure the velocity of a project, and that can help protect due dates or raise a warning flag about the need to shift timelines.

    The Definition of Done Helps Support Software Team OKRs

    Objectives and key results (OKRs) are a goal-setting framework to help senior leadership focus on what matters to the organization. By defining software engineering OKRs, senior developer leaders can keep their teams focused on aligning engineering goals and activities to business outcomes while also giving them room to experiment.

    Creating engineering OKRs involves taking the organization’s objectives and crafting actionable, generalized engineering objectives while identifying the metrics and milestones that drive progress. Translating the organization’s objectives for the engineering team helps them to understand exactly how their work impacts the company.

    One of the key benefits of the OKR framework is giving the engineering team a shared language and vision for success. However, without an established definition of done, OKRs will have a lot of unintentional gaps (and teams will have a lot of misunderstandings). Start with a definition of done to ensure that each team knows what their part is and when it’s officially ready for handoff to another team. Teams will pass the baton smoothly across the organization and there will be fewer obstacles in the race toward release in the market.

    A Definition of Done Can Help Organizations Grow Smarter

    In a small startup with a handful of software employees, establishing a shared understanding of timelines and goals is easy. As software organizations grow, casual chats at the water cooler aren’t enough to keep an organization in alignment. All team members must understand and have input on goals and objectives and those goals must be in writing and accessible to everyone. The benefits of establishing a definition of done include creating a shared understanding and unified language for software delivery, ensuring that new employees have access to tribal knowledge and process expectations, and detecting and removing false agreements between teams to create real consensus. By creating a definition of done early in an organization’s history, company leaders can ensure alignment even as more people join.

  • The Right Tool for the Job: Container Edition

    The Right Tool for the Job: Container Edition

    I have written about “the right tool for the job” repeatedly. Because (again) we are absolutists in IT. My father was an antiques restorer that dabbled in flipping houses and was rather forceful about the “right tool for the job.” When I left the service and joined IT, it was something of a shock to me that the “right tool for the job” was considered situational.

    At the time, there were a large number of Microsoft zealots that loved Visual Basic, and a larger number of non-zealots that preferred C or Java. That was how we got to my first “right tool” article. VB was very strong with regard to UI and record crunching, while to-the-metal was C’s strength and Java fell in between the two—not being perfect at either. So my article stated that. Front-ending databases? Use VB. Writing drivers and low-level processes? Use C/C++. Need a bit of both and easy web page generation? Go with Java.

    It is many years later and time for an article that seems just as obvious to me, but in a different IT space. We have all heard of the three architectures: Data center, cloud and mixed/multi-cloud. We have also heard of how, in this rapidly changing world of DevOps, Agile and containers, different organizations are using different deployment methodologies. That’s all good so far. Just as the variety of programming languages available today is a bonus in many ways, so are multiple deployment and operations options.

    Where it gets sticky is when people include the “We’re an X shop” in conversations. Or try to do everything under the sun with one of these methodologies. For many of us, container infrastructure is perfect. Use an orchestrator like Kubernetes, build systems and security practices, then not have to care much if the Kubernetes instance it runs on is on-premises, hosted or cloud. Of course you have to care some. Networks have to connect, etc. But most of the work is done.

    Then there are the people that have use cases that don’t fit this mold and have not acknowledged that fact yet. This blog is for you. We’ve all heard of companies that have containers whose life expectancy is measured in seconds. Beyond that, I was told about a company that is using containers as message queues. Honestly. Creating a container with a startup command and putting it on a queue. When it is pulled off the queue, it executes that command and then goes away. We’ll call this crowd the “Over Users”.

    Finally, we have the people that cannot or will not be moving their systems anywhere. There are a variety of reasons for this, but these are companies that know their applications will continue to run in the data center for the foreseeable future. We’ll call this group the “No Needs”.

    For Over Users, that is literally what serverless is for. Create a queue with a (much smaller) call to an API in a serverless environment and be done. Fewer configuration issues, less maintenance, less everything but code. Add a trigger, remove entire OS/network/configuration generation and maintenance. The most likely resistance to this advice is “But we have an investment…” Yes. And for most orgs it is an investment in unnecessary technical debt. Divest. Keep the good parts—a container architecture meant to shuffle through many containers that exist for a short time is well-architected.

    For No Needs, before you port everything to containers—which I am generally in total support of—you should consider if you need to; think about if it offers any benefits and what the ROI of such a move is. Honestly. For DevOps-heavy shops or orgs that require highly portable solutions, the ROI of containers is quick and obvious. For orgs whose applications are not changing a ton and are highly unlikely to move anywhere, the ROI is a whole lot murkier. “But it gives us options!” is the most likely negative response to “We don’t need that right now.” But that’s the problem—you have those options now. What it gives you is a shifting of cost from some possible future where it might be needed to today’s already constrained budget. If there is a verifiable need—normally in portability, but any verifiable need—then make the move. Until then, spend that time and money making those apps rock.

    You’re doing it. The lights are on and the applications are useful. Use the architecture that makes sense for what you are trying to achieve. Right tool for the job, as my Dad often said (he said it a bit more forcefully when I thought a chisel was a screwdriver, but that’s a different story). Keep rocking it, and stay focused on need.

  • Kintaba Creates the First Conference Dedicated to Incident Response – Techstrong TV

    Kintaba Creates the First Conference Dedicated to Incident Response – Techstrong TV

    John Egan, Co-founder and CEO at Kintaba, talks about IRConf, the first conference fully dedicated to Incident Response. The company behind the conference is Kintaba — a modern incident management platform for the entire organization. The video is below followed by a transcript of the conversation.

    Alan Shimel: Hey, everyone. Welcome to Techstrong TV. We have got another segment for you here. I’m happy to be joined by John Egan. John is the CEO of Kintaba Incident Response. 

    Hey, John. Welcome to Techstrong TV. 

    John Egan: Thanks, Alan. Good to be here. 

    Alan Shimel: Nice to have you. So, John, you know, I might have messed up what your position with Kintaba is. 

    John Egan: That’s right. CEO and Co-Founder. 

    Alan Shimel: Okay. And you know what? I mean I’ve heard of Kintaba, but that doesn’t mean our audience knows Kintaba, so if you wouldn’t mind, give us maybe a quick little background on Kintaba and maybe a little bit of your own background? 

    John Egan: Yeah. So, Kintaba is an incident response platform. We’re designed to let any organization be able to easily implement best practices for responding to critical moments within the company, so this tends to be things like site outages, major customer issues, and other sort of technical and non-technical problems that require kind of a group real-time response, you know, separate from like project management where you might have planned things that are happening. You take, like, the outage that happened, I think yesterday, at GitLab was a good example, right? Or a slack outage like that, that has an incident response team that’s responding to that situation and making sure it gets fixed quickly, and they’re probably using some kind of tool, either internally or off the shelf, and we’re that tool that any organization can implement in minutes instead of bringing in a consultant and spending months on this kind of stuff. 

    Alan Shimel: Absolutely. You know what was interesting, John, is I mean look, I’ve seen, for whatever reason since I’ve started DevOps.com eight years ago, incident response has always been like this, you know, it’s part of the DevOps story, and we’ve seen of course, you know, several companies have very successful exits and be, you know, very successful. They all kind of follow this pattern where they, you know, were replacing the beeper, then they became incident response. Then it becomes a whole platform, and APIs and integrations, and so forth. What makes Kintaba different? 

    John Egan: So, we’re really purists on the incident response side of things. I think a lot of organizations that come into this space end up either kind of pivoting over towards alerting, all right, which is sort of Pagerduty’s world – 

    Alan Shimel: Right. 

    John Egan: Which is more about automated detection, ping someone, it gets closed out, you know, you might get thousands of these things across the organization, or they go broad and start to become, you know, more of like an SRE platform to try to handle all of these different things that the site reliability engineers are really managing inside of the organization. But we really believe that incident response stands on its own as a practice that cuts across all of the organizations inside of your company. So, something unique about Kintaba is normally when you’re rolling it out within the organization you might start small. You might start with five or six seeds, a couple of people in SRE, but really rapidly we tend to get deployed across the entire organization and what we do really well is we’re a collaborative layer that helps everyone from customer success, to your SREs, to even your PR and marketing guys all be connected during these critical moments and all have one place that they can go to see what’s happening and participate in that response. 

    It’s a bit more of, you know, we call it an evolved sense of incident response, right? Versus kind of the old-fashioned way of looking at it, which might be two or three people in your SRE team are responsible for this. It’s really more about the entire community of your organization that responds to these company critical moments and building software that makes it easy for all of those people to participate. So, we’re a very non-technical feeling piece of software. We’re very human driven, everything from our reporting, to our collaboration experience, to the way you go and write documents, which is a new feature we’ve just released, all of this information is designed to be as easy to use as something like Google, as opposed to being, you know, more like a, you know, almost like a service now, kind of like complex enterprise implementation. 

    Alan Shimel: Got it. Good stuff. And, John, we’re going to talk about this conference you guys are putting on, but before we go, before we dive into that, you know, so we understand what Kintaba is. You know, in your go-to market here what are you guys finding? What are you seeing? You know, what’s kind of the state of the market, if you will? 

    John Egan: So, we’re really learning a lot about this community as it’s just starting to form, right? I think when communities are young and they’re new you actually find the best practices are pretty fluid and I think it’s one of the unique things about this space is Google put out a book back in 2016 where they sort of defined how they handle incident response internally. It was the first time anyone really wrote it down in a way that could be absorbed, but it turns out every organization has a slightly different attitude that they take towards this space and when you’re implementing a product, something like Kintaba, it really turns out that that product has to be pretty flexible on its way in to make sure that it’s easy for people to adopt, and then it also has to bring with it sort of almost like a light hand to say these are the very high-level macro steps you’re going to take to be successful, right? 

    And so, what we’ve learned early on is when Kintaba first launched, you know, over a year ago, we were very rigid about the way that you sort of had to come in and step through, you know, each part of incident response. What we’ve really learned and what the products evolve towards is to have rigidity around the high level steps report openness around visibility to when incidents are happening, like easy to use collaborative spaces, post mortems that editable and changeable, but also, be flexible in terms of who’s getting access to this thing, how are you giving people titles that are non-technical in the, like, customer success roles, you know, and in the marketing and PR roles, and how do you bring all of that together at once. We thought when we came in that, okay, maybe this would be a couple of seat experience for people, you know, you come in and maybe you need two or three seats, greater, four or five, and what we’ve really learned is these are company practices, like I was saying before, and I think that’s been kind of our big learning about the space is as the community grows this is really a cross-functional community inside of organizations rather than just a technical practice inside of DevOps and I think that’s really exciting. 

    Alan Shimel: Yeah. Absolutely. Absolutely. So, John, you guys are coming in behind the first dedicated incident response virtual conference coming up in April. Why don’t you – well, I already said it I guess, but why don’t you take it from there? Tell us more. 

    John Egan: Yeah. So, IR Conf, or Incident Response Conference, was really built out of the fact that we just couldn’t find another one to be part of that was focusing purely on the incident responder. When we talk about communities internally, you know, you think of communities as what makes people put their hands up and say, “I’m part of that,” right? I think DevOps, and SRE, and a lot of these spaces have done a good job, right? We have crowds and people will say, “Okay. I’m part of that.” Incident response has always  been one of those ones where I think people sort of nervously said, “Well, I think I’m part of that.” What IR Conf really does is it focuses not on sort of that statement of I’m in the incident response department, but instead focuses on the idea that I’m a responder, right? I’ve been part of responding to a major incident and when you ask that question it turns out a lot of hands go up and it turns out that there’s a community there that’s really looking for other practitioners to come together and talk about these things I was just talking about, right, best practices, right? What are other companies experiencing? How are we defining this role and the people who are involved? And so, what’s really cool about this conference is we get to focus really directly on that incident response process, you know, not even necessarily as it relates to other parts of DevOps, but just how we deal with this very specific problem that faces all organizations. 

    So, I’m really excited we have one our keynote is Emily Freeman, who’s out of AWS – 

    Alan Shimel: Sure. 

    John Egan: And has done all kinds of great talks about kind of the wider, softer development lifecycle inside of DevOps, and here she can speak much more directly, you know, to this response, incident response space. I think you can see the excitement around it. You know, we have sponsors, basically everyone across the DevOps world, Honeycomb, Commodore, Rookout, BridgeCrew, Gremlin, Teleport. All of these folks, you know, really came running in and said, “Yes. This is great. We need a space to come in and talk to our sub-communities that are already using our products that care about this space,” and so really, we just felt like there was a hole in the market and so this is a free conference. Anyone can join. This is a virtual conference. It’s on April 1st. It’s at IRCONF.IO if you’re out there looking for it. We’re really hoping to just get everyone to come along who’s ever been involved in an incident before and has experienced that fire moment inside of a company of, you know, hopefully not panic, but maybe panic, and wants to come together and talk to other folks, who have had the same experience. 

    Alan Shimel: Sure. I’m thrown back to my cyber – we didn’t call it cyber then – to my infosec days in the war room, you know, when stuff hit the fan. You know, best practices that were to set up a war room on how to respond to a breach, you know, some sort of attack. 

    John Egan: And that’s really where a lot of this grew out of, right? 

    Alan Shimel: Sure. 

    John Egan: We have incident response really lived in the security world – 

    Alan Shimel: Yeah. 

    John Egan: And it’s one of these parts of the evolution of the space, right, that now we’re adopting, you know, that process that really got its feet inside of the security incident response space across the organization to say, “Wow. These guys are doing a great job,” like, “We should be doing similar work across the org.” 

    Alan Shimel: Not all companies had such a great job on their war room, but that’s a whole other story. 

    Hey, John, you mentioned Emily Freeman speaking. Who are some of the other speakers? 

    John Egan: We’ve got two more who are announced right now and it’s Emily Freeman, and it’s Dave Rensin. Dave Rensin was previously at Google and ran a good chunk of the SRE Team there. He’s at a new organization now and he’s spoken at a couple of other conferences in the DevOps space, specifically about the process of incident response. I’m really excited about him being able to come in as well and talk, both, from a large company standpoint, right, a company like Google, where he really pushed a lot of customer metrics, right, for dealing with incidents, not just internal technical metrics, as well as the smaller company, like where he’s moved to now is a startup.

    Those two are kind of our cornerstones. Then we’ve got a series of other speakers, who are announcing over the next couple of weeks – 

    Alan Shimel: Who?

    John Egan: Who are coming in from all over industry, so I’m really excited. I’m can’t announce them here yet. They’re not finalized yet. 

    Alan Shimel: Okay. Good. 

    John Egan: Really exciting. Great organizations. 

    Alan Shimel: Can’t blame me for asking. All right. Hey, John, for people who are interested, that was IRCONF.IO? 

    John Egan: That’s right. IRCONF.IO. You go in there and just click Attend and register. We’re still at early bird, so everything is free right now. 

    Alan Shimel: And it’s April 1st. Do you have times on that? 

    John Egan: I believe we’re planning to start it early, around 8:30 is the beginning of it. 

    Alan Shimel: East Coast time? 

    John Egan: It’s technically a half-day, but it runs until about 2:00, I believe. 

    Alan Shimel: So, that’s East Coast time though? 

    John Egan: East Coast time. Yes. 

    Alan Shimel: Yep. We’ve got a global audience here, so that’s East Coast U.S. time. 

    John, sounds great. You know, if there’s a need in the market, you know, the market has a vacuum, so it’s good to see it getting filled here. I want to wish you the best with it, with the conference and with Kintaba. 

    John Egan: Great. Thanks, Alan. 

    Alan Shimel: It was great having you. John Egan, CEO, Co-Founder, Kintaba, the folks behind IRCONF and you can find out more about that at IRCONF.IO. That’s coming April 1st. Check it out. 

    We’re going to take a break here on Techstrong TV and we’ll be right back. 

  • Rust Foundation Allies With OpenSSF and JFrog to Secure Code

    Rust Foundation Allies With OpenSSF and JFrog to Secure Code

    The Rust Foundation announced today it is working with the Open Source Security Foundation (OpenSSF) and JFrog to help maintainers secure open source software created using the Rust programming language.

    Rebecca (Bec) Rumbul, executive director at the Rust Foundation, said even though Rust is a memory-safe language, there are still security issues that maintainers need to be able to address. The goal is to provide the resources those maintainers need to better secure their software supply chains.

    As part of that effort, the Rust Foundation is adding a dedicated security team to help maintainers of open source projects built using Rust to better secure their software supply chains. That team is being underwritten by JFrog and the Alpha-Omega Initiative funded by Google and Microsoft which was recently launched by OpenSSF. JFrog last week announced it is joining the Rust Foundation at the Platinum level alongside Amazon Web Services (AWS), Google, Huawei, Meta, Microsoft and Mozilla.

    The first initiative for the security team will be to conduct a security audit and develop threat modeling tools that will provide the foundation for a set of best security practices for maintainers, said Rumbul. The team will also help advocate for those security practices across the Rust landscape, including the Cargo package manager and Crates.io directory projects.

    The OpenSSF has previously called for replacing programming languages like Java that are not memory-safe as part of a larger plan to secure open source software supply chains.

    In general, the Rust Foundation is committed to enabling Rust developers to achieve their scale, security and sustainability goals, said Rumbul. Given the early stages of the Rust programming language, there is no better time to start addressing security issues that application developers can’t ignore, she added.

    In the longer term, The Rust Foundation is also committed to providing developers with Rust security education and is also working to come up with a fair approach to certifying which projects have achieved an appropriate level of cybersecurity maturity, said Rumbul.

    It’s not clear whether developers are abandoning older programming languages in favor of a memory-safe language such as Rust. Rust employs a compiler to track the ownership of values that can be used once and a borrow checker that manages how data is used without having to rely on traditional garbage collection techniques.

    Of course, replacing trillions of lines of code that have already been constructed using a variety of legacy languages is a gargantuan task that might never be fully completed. However, as new applications are built using a memory-safe programming language, the overall security posture of an organization will steadily improve. The challenge, of course, is that the number of developers that know how to build applications using those languages is still relatively small.

    Nevertheless, there may come a day when the way applications are coded is a lot more secure by default than it is today. In the meantime, however, the number of vulnerabilities that exist with application environments is still too numerous to count.

  • The Curious Connection Between Cloud Repatriation and SRE Ops

    The Curious Connection Between Cloud Repatriation and SRE Ops

    I have a fondness for philosophy. I’m about three classes short of a degree, and every few years I tell myself one day I’ll finish it. Thus, I am very familiar with what is known in statistics—and logic—as a post hoc fallacy, from which we get the saying “correlation is not causation.” This is the logical error of assuming that if event Y followed event X, event Y must have been caused by event X. The most famous callout of this fallacy came from Bobby Henderson, who illustrated the absurdity of assuming causation from correlation with his chart demonstrating that global warming was caused by the declining number of pirates in the world. Yeah, that doesn’t make sense, but neither do a lot of charts from which people draw causality. Just because two data points are mapped against each other does not mean one caused the other. In many cases, it doesn’t even make logical sense to correlate the two. After all, pirates and global warming? No one actually takes that seriously. But it’s an important point to make as we dive into the question of the relationship between site reliability engineering (SRE) operations and cloud repatriation.

    To be clear, I’m not suggesting that adopting SRE practices causes cloud repatriation. But I am suggesting that there is a close and meaningful relationship between the two. The fact that Google—a cloud provider—created SRE as a practice is not a mistake. The model, mindset and skillsets associated with SRE are integral to successfully operating cloud infrastructure and services.

    Cloud Repatriation Is Real

    Public cloud repatriation itself is a somewhat taboo topic in certain circles. Consider the controversy raised by Andreessen Horowitz when it published “The Cost of Cloud, a Trillion Dollar Paradox” and suggested companies were repatriating from the cloud and realizing significant cost savings as a result. Some would have you believe it’s not happening, but there’s enough data and anecdotal evidence to indicate that yes, it is.

    F5’s 2021 state of application strategy report asked the market about public cloud repatriation. A mere 13% had repatriated apps and another 14% were planning to. One year later, that combined total rose 40 percentage points to 37% and 30%, respectively. This is not an anomaly, as there are multiple credible analyst firms reporting similar results. Interestingly, the rate of repatriation is not globally universal. APCJ and LATAM are both far less likely to repatriate than EMEA and NA.

    I maintain that companies are repatriating apps from the public cloud—and the question isn’t ‘Are they?’ but rather ‘How many workloads are they pulling out—and where are they going?’ That’s a question for the State of Application Strategy 2023 research.

    For now, let’s consider a possible enabler of repatriation—SRE operations. Because even if the increasing cost of cloud is a driver of the desire to repatriate if you don’t have the skills to operate as efficiently elsewhere—and thus benefit from lower cost—then why would you repatriate?

    And we posit that it is SRE operational practices and skills that enable companies to repatriate and maintain the efficiency and cost savings needed to justify the decision—whether they’re moving those workloads to another public cloud, on-premises or to the edge.

    Digging Into the Data

    On the surface, there is a strong correlation between the adoption and application of SRE practices with cloud repatriation that seems to indicate that organizations with the ability to operate in a cloud-like manner, i.e., they’ve adopted SRE practices, effectively pick up their toys (apps) and go home (on-premises or elsewhere) because they can.

    Cut another way, only 4% of organizations that have not adopted SRE practices have repatriated apps from the public cloud. A whopping 73% of those who have adopted SRE practices have also repatriated apps.

    Of course, adopting practices does not necessarily mean applying practices. So, we looked at how organizations are actually operating applications, systems, and infrastructure. Specifically, we looked at the percentage of their operations that use SRE practices. Perhaps unsurprisingly, that generated similar results.

    Of those who operate 0% of their apps, systems, and infrastructure using SRE practices, 81% are not repatriating. Conversely, of those who use SRE practices for 76%–99% of apps, systems and infrastructure operations, 54% have repatriated. The point at which repatriation appears to begin picking up steam is when organizations surpass using SRE practices to operate more than one-quarter (25%) of their apps, systems and infrastructure.

    Remember I noted that APCJ and LATAM were far less likely to repatriate? They’re also far less likely to be leveraging SRE practices to operate their apps, systems, and infrastructure. In fact, over one-quarter (26%) of respondents in LATAM and APCJ (29%) were operating ZERO percent of apps, systems and infrastructure using SRE practices. In EMEA? That’s only 5%. And in NA, even lower at 2%.

    Meaningful Relationship or Curious Coincidence?

    There appears to be an inarguable correlation between organizations embracing SRE as an operational practice and public cloud repatriation rates. But is it a meaningful relationship or merely a curious coincidence?

    I’m going to argue that it’s a meaningful relationship.

    The practices and skillsets associated with SRE are wholly suited to operating a cloudy environment—at scale. As I said before, it’s no mistake that it was Google who created SRE and has literally written the book on it. The value of cloud is in its operational model, which can dramatically lower the cost per transaction, whether measured by HTTP exchanges or customer sessions. That enables a cost-efficient scale of applications and digital services.

    The use of automation and practices that tend to focus on meaningful incidents rather than non-disruptive occurrences provides the cost-efficient scale of the people (and thus their expertise) who are tasked with maintaining a high level of availability and performance.

    The adoption and use of SRE practices enable organizations to efficiently scale operations whether in the public cloud or on-premises or at the edge. And what the data tells us is that organizations appear to be using the capability to do just that.

  • DevOps World 2022: DevOps Tail Wags the Dog of Digital Business

    DevOps World 2022: DevOps Tail Wags the Dog of Digital Business

    As a methodology for building and deploying applications faster, DevOps has been around for more than a decade. But it’s only now that the methodologies and concepts pioneered years ago are starting to wag the dog of digital business transformation.

    Since the onset of the COVID-19 pandemic, digital business transformation has been all the rage. Organizations of all sizes now have a much deeper appreciation for digital processes that enable them to respond more adroitly to changing business conditions regardless of whether the global economy is expanding or contracting.

    DevOps Wags the Dog

    Not too long ago, ironically, business executives routinely complained that IT simply wasn’t fast enough to keep pace with the needs of the organization. At the CloudBees DevOps World 2022 conference in Orlando next month, it will be made crystal clear that businesses are now adopting concepts originally defined by DevOps pioneers to not only keep pace with the current rate of application development innovation but in many ways actually manage the business itself.

    As Mitch Ashley, principal analyst for Techstrong Research, an arm of Techstrong Group, publisher of both DevOps.com and Digital CxO, recently noted during a podcast, “It’s all about the software.”

    Of course, the shift in the relationship between IT and the business it supports has been a long time coming. The debate over how to close the divide between IT and the rest of business has been going on for decades. Even with the rise of DevOps, it’s been more than 11 years since the Jenkins continuous integration/continuous delivery (CI/CD) platform was officially launched. Today, advances in IT are now happening at a rate that many businesses are struggling to absorb.

    In fact, what started out as small teams of application development and deployment specialists looking for ways to automate the building and deploying of applications has now evolved into a multi-billion dollar sector of the software industry. And it is driving a level of business transformation that hasn’t been seen since PCs running word processing and spreadsheet applications arrived in the 1980s.

    Software is (Still) Eating the World

    Just how big an impact DevOps is going to have going forward is still to be seen. It’s still early days as far as pervasive adoption of DevOps best practices is concerned but, as Marc Andreessen once noted, software continues to eat the world. Today, the bulk of that software is being constructed with DevOps best practices in mind and using everything from containers to serverless computing frameworks that make it simpler than ever to add new capabilities to applications both inside and outside of the cloud. The pace at which technological innovations are being made is only going to accelerate.

    As such, it’s arguably never been more important for DevOps professionals to gather in person to share knowledge and insights. During the COVID-19 pandemic, DevOps platforms evolved tremendously. The teams that use those platforms exercise more influence than ever within their organizations. The best way to appreciate the value of DevOps has always been to learn firsthand how practitioners are employing those principles to achieve a wide range of strategic business goals.

    No matter how large or small the organization, the need to embrace and then continuously improve DevOps processes and workflows is, as always, a pressing concern. Achieving that goal in isolation may be possible, but there’s no doubt in anyone’s mind there’s nothing quite like spending a few days with a few thousand like-minded souls to make attaining those goals a whole lot easier.

    Want to learn more about this and other related topics? Join CloudBees for DevOps World 2022, held in Orlando, Florida, at the World Marriott Center. Use discount code DW22 when registering and get access to all the fantastic offerings, including keynotes, sessions, training and other interactive activities. We can’t wait to see you there for the DevOps Remix!