devopsy.kinsta.cloud

Category: Features

  • Best of 2022: CI/CD is the New Lock-In

    Best of 2022: CI/CD is the New Lock-In

    As we close out 2022, we at DevOps.com wanted to highlight the most popular articles of the year. Following is the latest in our series of the Best of 2022.

    Welcome to 2022! Let’s start this year’s blogging off with a bang, and point out that we are in the process of creating the biggest lock-in that we have ever had. And I do mean ever. Maybe the hardware lock-in of the old days where vendor A was incompatible with vendor B was worse, but at least that one we acknowledged when we signed the contract. This one is growing on us as I type.

    Our DevOps environments in 2022 cannot run without what I like to call the “toolchain controller”: GitLab, Github CI/CD, Atlassian, CircleCI, Jenkins—whatever you are using. Like you need some form of faucet or flow control to make running water work, you need the CI/CD environment to make DevOps work.

    You. Can’t. Live. Without. It. And just like any other vendor solution, CI/CD vendors are not going to be in a rush to help you move to a competitor. It won’t be easy or cheap to switch, largely because we are moving everything that might have been on a computer that once had developed code running on it into the toolchain.

    Sooner or later, we’ll back off the kitchen sink definition of “What is DevOps?”, but I predict even then we will have to answer “…A lot of what we do,” because DevOps is, by definition, end-to-end. It is about the process and all the steps from design to production.

    No matter how long or complex the process through which an organization’s CI/CD tool runs, a plan must be in place to replace it. A workable plan, not a check box plan. When Atlassian decided they preferred cloud-based customers, their customer base was shocked. Atlassian was wise enough to leave an option for customers who really had to have an on-site solution, but it is not difficult to imagine them phasing that option out, or to imagine another vendor dropping on-site support altogether. Some of the more competitive offerings in the space—GitLab springs to mind—are online-only already, making that option more appealing to other vendors as it’s now proven viable.

    So, have a plan. This is more important, IMO, than cloud vendor lock-in. Cloud vendors are a small community, and each understands the others, so if push came to shove you could find help moving between the big three cloud vendors. There are dozens of DevOps master controller-type solutions in regular use, and some of them are not at all CI/CD solutions (application release orchestration tools and software infrastructure management tools are two areas doing CI/CD duty at some orgs, for example), so there is not going to be much help moving between them unless you manage to find a very specialized skill set that matches your desired move.

    So, have a plan. Yes, I am indeed repeating myself. Know what you’re going to have to do before you have to do it. Include what to do if the master DevOps controller goes down. Because it will, sooner or later.

    And keep rocking it. Protect your masterworks by protecting what builds them. And keep rolling out the tools your users need. It’s a new year—learn a new skill and use that to turn out even more cool stuff. I haven’t picked a new skill yet, but I’m taking that last bit of advice myself. And again, in the new year, thank you, for all those who don’t know to say it.

  • Best of 2022: Return to the Office? Nah.

    Best of 2022: Return to the Office? Nah.

    As we close out 2022, we at DevOps.com wanted to highlight the most popular articles of the year. Following is the latest in our series of the Best of 2022.

    I was watching Charlene O’Hanlon’s interview with the CRO of Rollbar, discussing the slow growth of “return to office” policies. High-tech, because we are more mobile, had more options than many people—the janitor, for example, must be physically at their job to do it. High-tech employees did not have that issue—or if they did before the pandemic, they certainly didn’t after a few months of lockdowns. And now our infrastructure allows us to work from anywhere.

    One of the things that some companies are considering is requiring that all employees come back to the office. I rarely say things super bluntly, but this time, I need to be blunt.

    Fight that tendency if it shows up at your organization. At this point in history, that way lies corporate doom. Okay, maybe not doom, exactly, but certainly bad things. Some people back in the office is fine, a choice about coming back to the office whenever your state/provincial/local government decides to allow it? Fine. A required return for everyone? No.

    Of course, one cannot make such a blunt statement without an explanation. Do I think that remote work is somehow better for the company? No. Some people are more productive working from home, while others are distracted by the many non-work things around them that they want/need to do. So it is a mixed bag that requires monitoring. In all my years working from home and mentoring others just starting to do so, I’d say this split is closer to 50/50 than you might think. To paraphrase a sergeant from my army days, “Those who most want to are sometimes those who absolutely should not.” He was talking about living off-post, which is outside jurisdiction in much the same way that working from home is. And I’ve found some truth in that statement when applied to WFH. In fact, one interesting thing for me to watch were the success rates for companies that went 100% WFH. It’s gone better than I expected, probably because it was a broader swath of staff than just those who wanted to.

    So why would it be better if I, who have worked from home for more than 20 years at a number of different companies, think it’s a mixed bag?

    Hiring and retention.

    We know that we’re short-staffed. We know that it is hard right now to find any flavor of IT peep, but it’s brutally hard in InfoSec and pretty tough in DevOps. A decent percentage of employees is going to want to keep working from home, and in the current environment, they can change jobs faster than you can say “retention bonus” if you give them a reason. I don’t have hard numbers, but let’s assume a hugely conservative 10% of IT staff preferred working from home. Can you afford to lose that number of your staff? Or even half that number, assuming some won’t leave? Probably not, considering the openings versus employee counts we’re currently seeing. More to the point, can you afford to never hear from that percentage of applicants because they don’t want to work somewhere that mandates their physical presence in the office? From talking to people, it seems money doesn’t resolve this issue. Working from home is a tangible benefit, and whatever a company offers, the next one might match with the WFH option.

    There was a slow change going on in IT, where a significant percentage of staff were working at least part-time from home; the pandemic just sped that process up—shifted it left and iterated, so to speak, and now HR and management are going to have to deal with this reality head-on.

    I’ve felt that some of the positions I’ve held at some of the organizations I’ve worked at/for were best performed on-site. But we have enough successful 100% virtual companies now to acknowledge this might be a corporate culture issue and not a “job fit” issue.

    I don’t know how companies will handle it, but absolutism is not the way forward here, IMO. Charlene and Michael Davis had some thoughts on WFH and returning to the office you might want to listen to.

    After all, you all are kicking it, and I know some of you don’t want to go back to the office. Can your org afford to lose you? Not if there is an easy way to avoid it.

  • Best of 2022: Measuring Technical Debt

    Best of 2022: Measuring Technical Debt

    As we close out 2022, we at DevOps.com wanted to highlight the most popular articles of the year. Following is the latest in our series of the Best of 2022.

    What is enterprise technical debt? Technical debt slows down organizations and hampers their ability to deliver. Studies show that technical debt can triple the cost of support for technical products and services. Beyond this, technical debt can cripple a company’s ability to make the rapid changes needed to compete in today’s marketplace. Those all describe the effects of technical debt, but doesn’t get to the heart of its definition. Therein lies the problem.

    Technical debt is a term that many in technology are familiar with but few seem to understand. Fewer still have a clear way to measure it. Are security vulnerabilities debt? What about teams or applications that have not been migrated to standard communications platforms or shared processes? And, if all of this is debt, how do you measure it?

    Technical debt is often too narrowly scoped, focusing primarily on software defects and other code-related ‘debt’. However, the technical debt that large enterprises acquire goes far beyond code and has much more serious consequences. To truly address issues related to technical debt, we need to understand it not just in terms of software defects and backlog, but also security debt, process debt and organizational debt. Once we understand the scope, we can leverage financial tools as the basis for our measurement. 

    History of Technical Debt

    The term technical debt was originally coined as a metaphor by Ward Cunningham, one of the original authors of the Agile Manifesto. He introduced the concept to justify the work his team was doing to refactor a financial application, saying, “If we failed to make our program align with what we then understood to be the proper way to think about our financial objects, then we were going to continue to stumble on that disagreement which is like paying interest on a loan.” In Introduction to the Technical Debt Concept, the Agile Alliance noted that, “ … [I]n 1992, at the OOPSLA conference, Ward provided additional details (slightly paraphrased here based on feedback from Ward): ‘Shipping first-time code is like going into debt. A little debt speeds development so long as it is paid back promptly with refactoring. The danger occurs when the debt is not repaid.’”

    So, the original definition provides a metaphor that compares the impact on companies’ ability to deliver code based on the need to refactor to the financial debt incurred on a loan. While this concept has gained wide acceptance, there remains no single agreed-upon definition nor a single way to quantify the concept.

    This concept of debt is, however, very useful as it recognizes there is a cost involved with code defects; defects impede the ability to deliver. However, in modern enterprise environments, this concept needs to be expanded to include other elements which impact the ability to deliver value to the customer, including security-related debt and even organizational debt. There are many items that can (and should) be understood as part of technical debt including:

    • Known bugs that go unfixed 
    • Problems associated with poor code quality or poor design
    • Lack of code test coverage including tests to ensure quality, security and performance
    • Code or artifacts that aren’t cleaned up when no longer in use
    • Obsolete technology that is expensive, difficult to manage and unable to meet the demands of digital transformation
    • Incomplete or outdated documentation or missing comments
    • Known security issues that need to be remediated
    • Diversified tool sets that are not aligned with organizational standards
    • Technology organizational constructs that are not aligned with organizational standards

    It is important to note that, just as with financial debt, not all technical debt should be repaid. There is a concept of “good debt” or, to further the financial metaphor, money borrowed because the opportunity for investment earnings is greater than the rate of cost incurred by the debt. If, for example, one can borrow at 3% and earn at 10%, then it makes good financial sense to accrue debt. Similarly, if the money a firm can make by introducing a new feature outweighs the cost of resolving some debt then the team should prioritize feature work. This cost calculation will often change based on the product life cycle stage, as well. During early product development, the opportunity offered by new feature development often far outweighs the potential cost associated with the risk of compliance issues. However, as a product matures and gains more customers, the risk increases and the benefit of new features may decline. This is important because we do not want to imply that all technical debt should necessarily be fixed. There are many times when the right business decision is to invest in other areas rather than in the remediation of technical debt. However, by applying economic calculations to decisions around technical debt, we can make a more informed and calculated decision based on data. 

    Risk as Technical Debt

    Of the different forms of technical debt, security and organizational debt are the ones most often overlooked and excluded in the definition. These are also the ones that often have the largest impact. It is important to recognize that security vulnerabilities that remain unmitigated are technical debt just as much as unfixed software defects. The question becomes more interesting when we look at emerging vulnerabilities or low-priority vulnerabilities.

    While most will agree that known, unaddressed vulnerabilities are a type of technical debt, it is questionable if a newly discovered vulnerability is also technical debt. The key here is whether the security risk needs to be addressed and, for that answer, we can look at an organization’s service level agreements (SLAs) for vulnerability management. If an organization sets an SLA that requires all high-level vulnerabilities be addressed within one day, then we can say that high vulnerabilities older than that day are debt. This is not to say that vulnerabilities that do not exceed the SLA do not need to be addressed; only that vulnerabilities within the SLA represent new work and only become debt when they have exceeded the SLA. 

    Organizational Debt

    Another type of technical debt often excluded from standard analyses is organizational debt and toolchain debt. These types of debt are often seen in mergers and acquisitions where the organizations are not fully integrated. If one portion of an organization does not adhere to the standard processes or organizational structure for the broader organization, this also can impact the ability of the company to deliver value to customers and represents another form of technical debt. This same paradigm can often be seen in toolchain disparity within companies. If one part of a company is using Slack to communicate while another uses Microsoft Teams, the lack of standardization can cause friction that impedes the organization’s ability to deliver. This is not to say that companies must all use the same toolset or that perfect organizational structure parity is the only way to operate—but understand that there is an ongoing cost for companies that do not standardize. Where standards have been set, they should improve efficiency. When these standards are not met, there is a loss of that efficiency; another form of technical debt.  

    If, for example, a company’s standard chat platform is Slack, that company has chosen to be standardized because they believe standard communication between all teams improves the flow of value to the customer and reduces cost through consolidated license purchasing. If that company purchases another company that uses Teams, that toolchain disparity should be considered technical debt until the platforms are unified. Alternatively, if a company is a portfolio company, there may be limited or no value in standardizing on a chat platform. It may be completely appropriate for that company to decide that the purchased company is free to use whatever chat tool they choose, as that difference will have no impact on the ability of either part of the company to deliver value.

    Mik Kersten discusses this type of debt in his book From Project to Product where he introduces the concept of “accidental complexity” which he says “includes all of the heterogeneity in the tool stack that does not improve the flow of business value. Tools inherited as a result of mergers and acquisitions, or independent selection of similarly functioned tools due to a lack of centralized governance, fall into this category.” He goes on to say that “[r]educing the accidental complexity of tools should be an ongoing effort, as this is a form of value stream debt.” 

    It is important to note that, while many mergers and acquisitions can bring with them significant technical debt, discontinuity in organizational structures or toolchains do not necessarily equate to technical debt. Whether this type of organizational misalignment is construed as technical debt depends, to some extent, on the company merger and acquisition strategy. For companies where there are standards around organizational structure and tooling where some of the benefits of the merger or acquisition are based on the assumption of integration, unintegrated processes and tools should be considered technical debt if they are not remediated within the timeframe projected by the integration business case. If, however, a company is a portfolio company and takes a federated approach, then it is completely acceptable to leave processes and tools unintegrated and these should not be considered technical debt. Organizational technical debt only occurs when tools and processes do not meet standards that align with organizational strategy. 

    By looking across the enterprise, we recognize that there are many forms of technical debt including software debt, organizational debt and risk-related debt. It is important to consider all of these forms of debt. Given this broader understanding of technical debt, we can re-define technical debt as: 

    Unmet demand for technical resources which reduces the ability to deliver value to customers or increases risk. 

    Calculating Enterprise Technical Debt

    With this expanded view of technical debt and the understanding that it can include software defects, organizational complexity and toolchain heterogeneity, we can begin to look at ways to measure technical debt. 

    Quantification of each type of technical debt presents unique challenges. However, the term technical debt itself provides a useful starting point for how to measure all of these. The term technical debt implies a cost and each of the types of debt we have discussed can be quantified in monetary terms—giving us an overall debt cost for a large organization. 

    Interest Rate of Technical Debt

    From a financial perspective, what we are looking at when we are calculating the cost of technical debt is the interest rate on that debt. That is, how much does the technical debt cost an organization over a given period of time? Ward Cunningham discussed this concept in his video on the debt metaphor when he says, “With borrowed money, you can do something sooner than you might otherwise, but until you pay back that money you will pay interest.” Organizational complexity may create delays in developing new products and these delays can be calculated as lost revenue. Unmitigated risks may cost the software due to security breaches and the associated impact on that brand. As time goes on, these costs continue to grow. 

    This is useful because it is also possible to calculate the cost to mitigate the debt. For example, the cost to mitigate a software defect is simply the cost to fix that defect. It is important to note that the cost of mitigation may change over time. The cost to fix a software defect goes up the farther away from creation we get, as familiarity with the worked segment decreases. So we must calculate this cost based on a specified point in time. If we understand the cost of mitigation, we can then look at the cost of debt and weigh that against the cost to mitigate that debt. That helps us to make informed business decisions about the return on investment for remediating technical debt. 

    Software Debt

    While this economic approach to measuring technical debt gives us a common framework for approaching debt, different types of debt still require different measurement approaches. The cost of software-related debt can include defects, unneeded complexity and lack of documentation as well as code that is no longer used but which remains in an application. All of these increase the cost of code maintenance and changes and that may also cause incidents and even outages. We can therefore calculate the cost for this type of debt relatively easily by looking at the cost of support activities related to the issue and adding that to the impact on productivity for the development of new code. The cost of the debt can also be easily calculated by looking at the time and resources required to fix the problem. 

    For example, at one large educational company, there was a bug in the code that caused load spikes. Once or twice a week, the entire team needed to jump on an incident bridge to remediate the load issues by restarting application servers in sequence. Because the application was old, the location of the defect was unknown and difficult to troubleshoot. However, it was easy enough to estimate that the cost to fix the issue would not exceed a couple of weeks worth of effort. As an estimate, let’s say this work would cost $10,000 to mitigate. Comparatively, due to the issue, the entire team, incident management and operations center took one to two hours per week remediating the impact. Let’s estimate this cost at $2,000/week. Based on this high-level analysis, it is clear that the ROI for fixing this defect would be realized five weeks after fixing the bug. 

    Risk Debt

    The cost of risk-related debt is somewhat more complicated to calculate but we can, again, use dollar values as a basis for our calculation. These sorts of debts include security risks as well as compliance-related risks. One of the reasons it is difficult to determine the cost of risk-related debt is because risks do not always become reality. That is, just because there is a risk of a security breach does not mean there will be a security breach. The very fact that it is a risk implies that there is a chance of something happening; it is not a certainty. However, risk analysis provides a very useful tool for calculating this sort of value in the form of annual loss expectancy (ALE). Annual loss expectancy is a means of measuring the expected financial loss during the period of a year. It is calculated by multiplying the expected loss by the probability that it will happen in a given year, or:

    ALE = SLE * ARO

    where SLE is the single loss expectancy, or the amount of loss if the risk occurs ,and ARO is the annualized rate of occurrence, or percent chance that the event will occur in a year. So, if the expected loss due to a given breach would be $100,000 and that type of breach is likely to occur once every five years—giving it a 20% chance of happening in any one year—the ALE for that risk would be $100,000 x 20% = $20,000. Because this number estimates the potential loss due to a risk, it serves as a valuable estimate of the interest for technical debt in the form of that risk.

    Similar to software defect-related debt, the value of the risk-related debt can be calculated as the cost to remediate the risk. This might include activities such as patching outdated systems, software changes to prevent hacking or even process changes to remediate compliance-related risks. For each of these cases, we can look at the cost of resources to remediate the issue and weigh this against the ongoing interest, expressed as ALE, to determine if the investment should be made to remediate the risk. 

    Organizational Technical Debt

    Organizational technical debt is the debt caused by organizational structure, processes and tools that do not align with the standards of the organization and which create delays in the delivery of value to customers. This is frequently seen in mergers and acquisitions where two separate organizations are brought together. Until and unless these organizational structures and tools are aligned, as in our Slack versus Teams example, there may be misalignment which makes it more difficult for the company to deliver as a whole. One way to think about this type of debt is the lack of alignment within tools and ways of working that impedes the flow of value. The “accidental complexity” described by Mik Kersten is a great example of organizational debt. 

    Organizational debt can be measured in terms of the cost of delays in delivering value to the customer. If, for example, handoffs between teams are taking a long time because they are using different development platforms, this delay can be measured and then quantified by the cost of delay. With this type of debt, it is also possible to look at the cost to overall productivity. If the productivity of the organization is negatively impacted because of a proliferation of communication systems that hampers the flow of information, then the cost of the impact to productivity should be calculated; this represents a lost opportunity and another form of technical debt. 

    As with development-related debt and risk-related debt, the cost of the debt itself can be estimated by the cost to remediate the organizational debt. This provides a useful mechanism for determining if the investment should be made to remediate the debt.

    Conclusion

    For large enterprises, the scope of technical debt is expansive. There are many types of technical debt and different methods to calculate the cost of the debt as well as the interest rate on that debt. Software-related debt must be calculated by the impact to the customer or impact to productivity of the people working on the software. Risk-related debt can be calculated by looking at the ALE of that debt. Organizational debt can be calculated by looking at delays in the delivery of value to the customer.

    All this information comes with a lot of uncertainty and a lot of calculations to consider. What is certain is that there are tremendous costs associated with technical debt. The interest on that debt is being repaid every day even if organizations are not aware of it. While it may not make sense or even be possible for enterprises to truly know the full extent of their debt or the amount of interest they are paying, by looking at the cost to remediate and the cost of the interest, we can at least quantify the impact and make calculated decisions based on return on investment for the work necessary to remediate technical debt in the enterprise.

  • Best of 2022: Microsoft Adds Raft of Tools and Azure Cloud Extensions

    Best of 2022: Microsoft Adds Raft of Tools and Azure Cloud Extensions

    As we close out 2022, we at DevOps.com wanted to highlight the most popular articles of the year. Following is the latest in our series of the Best of 2022.

    At its Build 2022 conference this week, Microsoft added a bevy of tools and platform updates as part of its effort to improve the productivity of developers and the DevOps teams that support them.

    A Microsoft Dev Box offering, for example, provides access to a set of pre-configured environments on the Azure cloud that enables developers to be onboarded in seconds, while a previously announced GitHub Copilot tool that makes use of artificial intelligence to make writing code faster will be made generally available this summer.

    Donovan Brown, a partner program manager in the Azure CTO Incubations team, told conference attendees the overarching goal was to enable developers to stay agile while they continue to work remotely in the wake of the COVID-19 pandemic.

    For DevOps teams, a Smart Maps capability was added to the Application Insights module within Azure Monitor. Smart Maps leverages machine learning algorithms to surface performance bottlenecks or failure hotspots and is also available as a technical preview.

    There is also a tool that discovers and groups dependent Hyper-V virtual machines (VMs) and physical servers to ensure all required components are identified and included during an Azure migration.

    Other updates include an App Service Landing Zone Accelerator that provides a combination of artifacts and documentation to automate application deployment using the Azure App Service 3.0 and preview tools for migrating ASP.NET web applications to the platform. In addition, Microsoft is previewing support for Google Remote Procedure Call (gRPC) on the Microsoft Azure App Service. A .NET Multi-Platform App UI framework allows for building natively compiled applications for iOS, Android, macOS and Windows using C# and XAML in a single codebase.

    Microsoft has also extended the reach of its cloud platform for managing application programming interfaces (APIs). The Azure API Management service now supports the Azure Private Link virtual private network (VPN) along with a capability that automatically renews certificates. In addition, Microsoft is previewing support for the GraphQL query language for APIs.

    In addition, a GitHub OpenID Connect (OIDC) with Azure Active Directory (Azure AD) workload identity federation capability has also been added to eliminate the need to store and access secrets outside of the Azure cloud platform.

    Microsoft, in partnership with VMware, is adding a managed Azure instance of the Spring framework for building Java applications and making previews of the Dynatrace observability platform and NGINX proxy software available as software-as-a-service (SaaS) applications on the Azure cloud.

    Finally, Microsoft has added a raft of tools to make it simpler to build cloud-native applications on Kubernetes clusters.

    It’s not clear how quickly developers are adopting these tools, but DevOps teams can expect to see the rate at which applications are developed on the Azure cloud increase in the months ahead. The challenge, of course, will be managing all those applications—both as they are being built and after they are deployed either in Azure or some other compatible IT environment.

  • Best of 2022: Day in the Life of a Site Reliability Engineer (SRE)

    Best of 2022: Day in the Life of a Site Reliability Engineer (SRE)

    As we close out 2022, we at DevOps.com wanted to highlight the most popular articles of the year. Following is the latest in our series of the Best of 2022.

    By now, most of us are familiar with the concept of site reliability engineering (SRE). The term was originally coined by Google and SRE has been gaining traction in recent years as a role dedicated to increasing the resilience of digital ecosystems. To accomplish this, a big part of the SRE doctrine is “automating your job away.”

    Observability and monitoring tools are extremely important to DevOps and for SRE. But what is the day-to-day experience actually like for your average SRE? What activities does an SRE do? And what percentage of their time does an SRE spend on particular tasks?

    I recently met with James Curtis, lead site reliability engineer at a large multinational company. According to Curtis, the SRE approach takes a certain type of person—someone with Ops in their blood and a passion for eradicating repetitive tasks. Below, we’ll use his input and the input from other sources to better understand what it’s like being in the shoes of an SRE.

    Understanding the SRE Role

    Organizations have interpreted the role in different ways. But in a general sense, SREs attempt to maintain high reliability and availability for software applications and respond to incidents as they occur. To aid their efforts, an SRE tries to streamline and automate as many operations as possible to remove opportunities for human error.

    A hallmark SRE goal is to reduce “toil.” Curtis defines toil as “tedious actions that really have no enduring value.” For example, say an admin must manually restart a service in Microsoft Exchange every time a triggering event interrupts the service—this action could certainly be automated away. SREs spend much of their time eliminating toil by coding automation and configuring internal tools to better interact with software infrastructure.

    SREs usually are also in charge of logs and setting benchmarks using a tool like Splunk or Datadog to observe and ingest data. Curtis himself uses Cribl, which offers an observability pipeline to parse and route log data. Since SREs oversee internal service-level indicators, they are typically in charge of normalizing behavior and setting SLOs and SLAs.

    Common SRE Activities

    An SRE juggles a lot of tasks. For proof, read the hour-by-hour day in the life by Yonatan Schultz, SRE at New Relic. Schultz’s average day is spent configuring infrastructure, jumping from project to project, and, of course, hopping into many meetings. Here are some other tasks an SRE might perform on a daily basis:

    Monitoring service-level indicators (SLIs). An SLI could be the number of successful requests out of total requests. Having a high SLI, in this case, would be a target. SREs track other metrics such as availability, uptime performance, latency, error count and throughput. Regularly monitoring systems is essential to ensure proper resource utilization of containers and to avoid out-of-memory (OOM) errors.

    Setting SLOs and SLAs and determining error budgets. Once you have determined baseline system performance, you can set service-level objectives (SLOs). These are typically internal targets like 99.99% availability. While SREs typically oversee functional metrics, some teams set goals for non-functional metrics, as well. SREs help determine service-level agreements (SLAs), which are more legally binding and typically partner-facing.

    Responding to incidents. On-call SREs will be tasked with finding the root cause of issues as they arise. When triaging an incident, it’s helpful to have all the necessary logs and tools immediately at hand. This is one area where automation can assist by pulling relevant details to instantly build a case, said Curtis.

    Writing postmortems. After an incident has been dealt with, it’s important to learn from it. Postmortems are common in cybersecurity practice and often fall under the responsibility of an SRE. These reviews seek to answer set criteria to get to the heart of an incident and identify the root cause(s) of an issue to prevent it from happening again.

    Automating other system tasks. SREs will spend significant time coding and building tools for engineers to interact with infrastructure. For instance, an SRE might generate reliability reports that consider performance over long time periods.

    Cross-department collaboration. SREs don’t tend to own application code. Instead, they support multiple software divisions. This means checking in with other developers, disseminating best practices and reviewing new architectures to represent the reliability side of the equation.

    As you can see above, the SRE role might blend many different activities, and keeping track of them all may be part of the job itself. Anika Mukherji, an SRE at Pinterest, shared that, at Pinterest, there is a weekly meeting where SREs share what they spent time on. For another helpful “day in the life” story, take this account of an average OpenShift SRE’s day. Nikita spends her day responding to open JIRA cards, handling incidents, pushing code to GitHub and syncing with SREs in other regions when shifts change.

    Time Well Spent

    So, how does an SRE allocate their time? As Curtis explained, the ideal goal is a 50/50 split between time spent in work mode and time trying to automate that work away. Of course, this is more like a sliding scale, he admits. When things are broken, attention naturally shifts toward more manual work, says Curtis. While there may be less automation upfront, the scale balances out the more you build.

    This is similar at other institutions too. For example, in an interview with DZone, Paul Greig explained that half of his time is spent on service reliability upkeep and half on toil reduction. John Turner, SRE at Squarespace, said some 70% of his time is spent writing code—much of it automation code.

    Who’s Fit to Be an SRE?

    As mentioned above, the SRE job requires a specific attitude toward solving operational problems. Curtis described this person as “someone who hates the monotony of doing something over and over.” This person should have a drive to continuously solve new problems because as soon as you’ve automated one thing, you move on to the next, he said.

    Don’t assume a role with high automation equals an easy paycheck. We’ve all heard the story of the system admin who secretly automated their job away and never told a soul. While this may fly in other roles, the job of an SRE is never finished. “You’re never going to run out of stuff to optimize,” Curtis said.

    Again, the SRE role is a blend of many different activities. You will likely interface with different developer teams, too. Since the role involves this type of interaction, communications skills and understanding are a must.

    Final Thoughts

    The SRE practice is wide and varied—each company may have its unique flavor. Looking to the future, there are plenty of use cases for machine learning to further empower SRE practices, said Curtis. This is especially relevant in security automation, where algorithms could be trained against real attacks to flag suspicious behavior. Or, with the right amount of data, predictive analytics could be applied to anticipate high CPU peaks, informing server utilizations.

    In both scenarios, Curtis stressed the importance of observability. “It really gives you the ability to look at data and to ask a question later you didn’t realize you needed to ask,” he said. This power hinges on easy data transformation to make things speak the same language. That’s why his team opted for Cribl, that allowed them to normalize that data on the fly and replay it later. “The ability to change data and morph it—that’s the power observability gives you.”

  • Best of 2022: Trends in Blockchain for 2022

    Best of 2022: Trends in Blockchain for 2022

    As we close out 2022, we at DevOps.com wanted to highlight the most popular articles of the year. Following is the latest in our series of the Best of 2022.

    Blockchain is hot right now. The industry has a staggering CAGR rate of over 69% between 2019 to 2025. Blockchain is no longer only relevant to cryptocurrency, either—blockchain-based applications are emerging to support many other scenarios that require decentralized data storage and accessibility.

    I recently met with Medha Parlikar, co-Founder and CTO, CasperLabs, to learn about some of the top trends that will likely impact this burgeoning sector in the coming months. According to Parlikar, we are still in the early stages of both blockchain and NFT development, and 2022 will see continued growth in the sector.

    These movements will likely include more significant investment into blockchain technologies and the emergence of nuanced business models that leverage NFTs. Proof-of-stake (PoS) models will likely become more critical to curbing negative climate impacts and more solutions will emerge to support the development and operations that sustain blockchain-based applications.

    Continued Blockchain Investment

    First, we will likely see greater investment in blockchain technology in the coming year. Unsurprisingly, finance will probably experience the most adoption, as the use of blockchain in the financial sector is anticipated to reach a value of 22.5 billion U.S. dollars by 2026. But outside of finance, blockchain investment will seep into innovative areas as well. For example, spending on blockchain in health care is expected to rise to $5.61 billion by 2025.

    Blockchain not only appeals to corporate interests—developers on the ground seem excited as well. “Developers are waking up to the opportunities that blockchain brings,” explained Parlikar. “We’ll see a big increase in developers moving into blockchain.” She pointed to a surge of developer interest in blockchain to construct tools that go beyond widgets to extract real business value.

    We’re also seeing a peak of interest in blockchain funding from nation-states. For example, in 2021, the European Union announced plans for a multibillion-euro investment into blockchain infrastructure.

    Uncertainties Around Regulation

    Blockchain is ushering in major economic shifts. But the cryptocurrency market is still a ‘wild west’ with little regulation. According to recent reports, it appears the U.S. Securities and Exchange Commission is gearing up to more closely regulate the cryptocurrency industry in 2022.

    “More investment in blockchain is bringing it into the mainstream, but what’s holding back a lot of adoption is regulatory uncertainty,” said Parlikar. Forbes similarly reports regulatory uncertainty as the biggest challenge facing blockchain entrepreneurs.

    Blockchain is no longer relegated to the startup domain, either; well-established financial institutions also want to participate in the massive prosperity, said Parlikar. This excitement is causing a development-first, law-later mindset, similar to the legal grey area that followed Uber as it first expanded its rideshare business.

    “[Blockchain] businesses are trying to hedge risk,” Parlikar explained. “We want to comply and aren’t doing nefarious things intentionally—there’s just a tremendous opportunity to innovate and streamline operations and increase the end-user experience.”

    New NFT Strategies to Emerge

    If you follow any artist accounts on Twitter, you’ve probably heard a lot about NFTs lately. Non-Fungible Tokens (NFTs) use blockchain technology to imprint ownership and value onto all types of assets.

    “I see NFTs as a great representation of the real world,” said Parlikar. Blockchain-fueled NFT technology is creating marketplaces out of ephemeral or purely digital things—such as 3D artwork—which could not be easily traded on typical markets.

    But, we’re likely only on the cusp of overall NFT usage. “The value of NFTs is still underutilized,” said Parlikar. Right now, NFT marketplaces are mainly just about asset ownership. But there’s no reason one couldn’t build, for example, an NFT rental system that shares access for a period of time, or one that offers subscriptions for recurring revenue, she suggested.

    “[There are a] lot of interesting markets that could emerge,” said Parlikar. For example, combining NFTs with the internet of things (IoT) could bridge the metaverse to the real universe, proving ownership for our physical assets.

    Proof-of-Stake (PoS) Response to Energy Backlash

    The dark side of cryptomining is that it takes significant computational power. Unnecessary computation is a drain on energy that exacerbates humanity’s environmental impact. And, if your say in a network is correlated solely with computational power, there will inevitably be an escalating computational arms race to one-up others in the network.

    This is why Parlikar is putting faith in proof-of-stake blockchains. “Proof-of-stake is definitely the way to go,” she said. In PoS networks, power is tied to the state of each actor in the network, which can be determined or modified by a consortium. Participants can agree on the governance model and set boundaries. This is opposed to proof-of-work (PoW), in which your power in the network is directly correlated to computational power.

    PoW systems become unsustainable for enterprises trying to spin up their own blockchain networks, or when attempting to lift up an entire industry, Parlikar explained, as well as increasing negative environmental impacts. Thus, PoS are a better fit to bring collaboration to enterprise-grade consortium networks.

    More Blockchain DevOps Solutions

    Overall, blockchain penetration is still relatively low. This is, in part, due to the fact that it’s still tough to use. As I’ve covered before, blockchain-based development is still relatively inaccessible. It requires nuanced theory to grasp and specific technical knowledge to implement. “It’s not trivial,” added Parlikar. “Dealing with private keys, obtaining keys—it’s still very complicated to get started.”

    Managing a blockchain could also use a dose of DevOps. Yet, blockchain’s immutability and decentralized nature create friction with traditional DevOps. “The majority of protocols will not allow you to put a smart contract under CI/CD,” Parlikar explained. Thus, organizations may opt for blockchain solutions that play nicely with test-driven development. These solutions will likely enable the ability to iteratively develop using established DevOps pipelines.

    Benefits and Challenges Ahead

    Interest in blockchain and distributed ledger technologies continue to surge. But, what are the end benefits of this blockchain and NFT-infused reality? According to Parlikar, we are heading toward “a world with greater trust and transparency.”

    Blockchain ensures provenance. To put it plainly, it validates that something was done at a certain point in time. With this knowledge, participants in a network could prove things beyond a shadow of a doubt.

    This has the power to avoid corruption and unlock more collaboration and value potential. “You don’t have to trust an individual—you can trust the information they are providing,” said Parlikar.

    It’s still early in the blockchain revolution. “Blockchain technology is still at the embryonic stage and facing many challenges,” wrote the European Commission in their report, Blockchain Now and Tomorrow: Assessing Multidimensional Impacts of Distributed Ledger Technologies. For blockchain to reach ubiquity, the community will have to overcome challenges, including lowering the energy footprint, improving developer usability and responding to impending legislation.

  • AWS Saves Ukraine’s Data | WPF ‘is not Dead’ (yet) | Devs Quit for Cash

    AWS Saves Ukraine’s Data | WPF ‘is not Dead’ (yet) | Devs Quit for Cash

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: Amazon S3 is keeping Ukraine’s data safe, we ask if Windows Presentation Foundation is dead, and developers tell us why they switch jobs. (more…)

  • Opvizor Extends Monitoring Reach to the Cloud

    Opvizor Extends Monitoring Reach to the Cloud

    Opvizor, a unit of Codenotary, has extended its monitoring capabilities to cloud computing environments starting with support for Amazon Web Services (AWS) and Microsoft Azure.

    Codenotary CTO Dennis Zimmer said Opvizor aggregates data to provide a more efficient approach to applying analytics to surface performance and risk issues.

    Previously, Opvizor was optimized for instances of VMware vSphere running in an on-premises IT environment. Now the unit of Codenotary, a provider of a service for generating software bills of materials (SBOMs), is extending those monitoring capabilities to the cloud.

    That transition comes as many organizations are looking to strike a balance between monitoring pre-defined metrics and observability platforms that make it possible to surface unknown potential issues that could adversely affect IT environments.
    It’s still early days as far as observability is concerned, so Opvizor is focused on meeting most IT teams where they are today as they further embrace cloud computing, said Zimmer.

    Observability, in general, promises to query not just logs and metrics but also distributed traces that are applied to specific processes. In that way, observability allows organizations to detect patterns that have not been predefined. In effect, an observability platform makes it simpler to debug application environments where the number of dependencies that exist between services are becoming too complex to track manually, much less decipher. Without the aid of an observability platform it can take weeks to discover the exact cause of an application’s performance degradation.

    Of course, those applications will need to be instrumented using agent software capable of generating the data observability platforms require. Fortunately, open source projects such as OpenTelemetry are helping to drive down the total cost of instrumentation.

    However, in the absence of observability, most IT teams will rely on existing monitoring tools. Opvizor is making a case for centralizing monitoring across multiple platforms to reduce costs, noted Zimmer.

    Heading into 2023, each IT team will need to decide how and when to rely on traditional monitoring tools versus observability platforms. Many observability platform providers are justifying the acquisition of a new platform based on cost and the ability to eliminate the need for multiple monitoring tools. However, it remains to be seen whether IT organizations will prefer to employ both monitoring tools to track predefined metrics alongside observability platforms.

    Regardless of approach, the reality is that IT environments are becoming more complex to manage. In addition to running workloads in multiple cloud platforms and on-premises IT environments, organizations are deploying emerging cloud-native applications based on containers alongside existing monolithic applications deployed on legacy virtual machines. As workloads are deployed on multiple platforms, the total cost of IT starts to rise. One way to contain those costs is to standardize on a set of tools that can be applied to multiple platforms.

    The pressure to rationalize toolsets only increases during any period of economic uncertainty. The challenge, of course, is convincing one team to give up their preferred tool in favor of another to reduce costs.

  • DevOps 2022: The Year That Was and Wasn’t

    DevOps 2022: The Year That Was and Wasn’t

    We will be wrapping up our DevOps coverage for the year by the end of this week. It’s hard to believe that another year has gone by. The gravitational lensing of COVID-19 makes it hard to see the progression of the last nine years here at DevOps.com. Things seem to get a bit hazy looking back. However, 2022 was certainly one for the record books in ways both good and bad. 2022 saw DevOps, our community our countries and our world pushed and pulled in directions we would not have imagined just a few short years ago. To understand what happened in DevOps in 2022, we need to account for several macro factors that played outsized roles this year. They are:

    1. COVID-19: It’s still here; still killing hundreds of people every day. While some in the U.S. and Western Europe have “moved beyond” it, COVID-19 is still draining our resources, sucking the air out of the room and impacting not just the economy but how we interact with each other individually. In the next few months, I am afraid we will see the 1.4 billion people of China experience a backlash as they move away from zero-COVID policies. Not to mention the other viruses and diseases that are circulating.
    2. War: When I was a young boy, I remember a popular poster/sticker that said something like, “War is not healthy for children and other living things.” That’s still true; it’s not healthy for the economy, either (unless you sell ammunition). The human toll of the war in Ukraine is heartbreaking in the present moment, and the toll that this war will take on the once-healthy tech sectors in Ukraine, Russia and the rest of Eastern Europe also will be felt for a very long time.
    3. Inflation: Again, I am transported back to my youth as inflation rears its ugly head and puts its foot on the throat of our economy. Sometimes I wonder if the Fed and other central banks won’t stop raising interest rates until the economy is good and dead, but their fear of inflation is almost genetic. While recent signs point to a potential peak, we still have a long way to go and the possibility of more interest rate hikes ahead of us. These will continue to make it harder to raise money and put a crimp in companies’ budgets. That means layoffs and budget cuts; projects cut or pushed out. We haven’t seen disruption like this in tech for more than 10 years. But for those of my vintage, we have seen cycles like this before. This too shall pass.
    4. Hate: Is it me or has the whole world gone crazy with tribalism, nationalism, racism, antisemitism and a bunch of other “-isms”? We seem to be ripping ourselves apart from the inside out. I don’t have a clue how to combat this on a macro level, but I pray every day that things get better.

    With these four horsemen of the apocalypse riding among us, you may think that I am not very bullish on DevOps now and in the future. But you’d be wrong. Like the little boy in a room full of horse manure, I remain very excited because I know there must be a pony around here somewhere.

    DevOps has continued to evolve, continued to mature and has become an indispensable part of today’s modern software factories. Like their own version of Marvel’s Avengers or DC’s Justice League, DevOps, along with cloud-native, SRE, Agile, CI/CD, DevSecOps and dare I say it, platform engineering, are wielded by IT teams around the globe to continue to push forward as we digitally transform our businesses and the very fabric of our lives. That is what gets me up and out of bed every day and makes me excited to come to work.

    While some marketers are busy running around printing stickers and shirts proclaiming that DevOps is dead, nothing could be further from the truth. Look around—as far as the eye can see, DevOps is everywhere. As we explored in our DevOps Experience virtual event in October (the sessions are still there for you to watch on demand), DevOps is everywhere. It is at the edge; it is in the “connected things”; it is inherent in technologies like WebAssembly (Wasm) and DevOps is distributed.

    In 2023, I think DevOps at the edge, the DevOps of things and Wasm will be huge areas of growth. I realized that some of these folks claiming that DevOps is dead are trying to say platform engineers will replace DevOps engineers. But like Darth Vader trying to strike down Obi-Wan, all they are doing is making DevOps stronger than ever. You see, DevOps was never about DevOps engineers; it’s not about one role or one individual. In fact, some of my best friends helped kick off the DevOps movement and they’d react quite loudly when they heard the term DevOps engineers. DevOps is about building, maintaining and managing our software and applications. It’s a way of thinking about how we can go faster, be smarter and achieve better quality. It’s not about a particular tool you use or the role you play on the team, it’s about the fact that you have a team.

    So, what does 2023 have in store for DevOps? More of the same, I think. DevOps, along with all of the associated frameworks and technologies, will continue to allow us to do more and do it faster and better. As I think about the coming year, I am very excited about the potential for ML and AI and I think they are going to have an outsized role in allowing us to be more successful. I think the novelty of these technologies has worn off and we will get to the productivity phase. I remember interviewing Luke Kanies years ago; he told me that one day software would write software. I asked him when he thought this might happen, and he said maybe 2035 or so. I think we are on track. Much like ChatGPT and Jasper’s impact on copywriting, AI won’t take the place of developers. But it will free them of some of the mundane, repetitive chores they do now.

    I think we’ll also see an emphasis on feedback loops and iteration. I think automation, ML/AI will contribute to this in a big way. We have put so much of our DevOps emphasis on the development, integration and deployment of software. But observability allows us to see patterns and use cases as never before. We need to incorporate those findings into the next revision of our software. This is going to be a big story in 2023. That old infinity loop model for DevOps will rise to the top once again.

    Thank you for all of the support for DevOps.com this year and over the past few years. I hope this holiday season finds you and yours healthy and happy. Merry, merry. Happy, happy. See you in 2023.

  • Rezilion Updates Open Source MI-X Tool to Better Secure App Development

    Rezilion Updates Open Source MI-X Tool to Better Secure App Development

    Rezilion has updated its open source MI-X vulnerability discovery tool to include mitigation and remediation recommendations.

    In addition, the tool can now produce machine-readable output in either a JSON or CSV format.

    Finally, the company added Windows support for Heartbleed and SpookySSL vulnerabilities in Windows environments.

    MI-X makes it possible to use a command line interface (CLI) tool to identify and establish the exploitability of more than 20 well-known common vulnerabilities and exposures (CVEs).

    Yotam Perkal, director of vulnerability research for Rezilion, said Rezilion created the tool as part of its efforts to create a platform that enables organizations to better secure their software supply chains. The primary purpose of the platform is to prevent vulnerabilities from being introduced into applications in the first place by providing developers with the context required to understand the potential severity of a vulnerability, he added.

    There’s no doubt that developers need tools that enable them to easily identify vulnerabilities while writing code. However, shifting responsibility for vulnerabilities entirely left toward developers is not a panacea for application security. DevOps teams need to embrace DevSecOps best practices to ensure that the build created by multiple developers is secure and that the runtime environment where applications run is also free of malware.

    The challenge is that achieving that goal needs to go beyond adding additional tools to a DevOps workflow. The biggest challenge is narrowing the historic cultural divide between developers and cybersecurity teams tasked with discovering vulnerabilities. Most cybersecurity professionals don’t understand how applications are developed, so every instance of vulnerability found in an application looks the same to them—regardless of whether the actual code is externally accessible.

    DevOps teams, meanwhile, are trying to strike a delicate balance between patching innumerable vulnerabilities and building and deploying additional applications. More than anything, those teams need to be able to prioritize their patching efforts based on the actual versus theoretical severity of a vulnerability, noted Perkal.

    It may never be possible to address all the application security debt application environments have accrued over the years. However, as apps are built and application security issues are addressed more proactively, the state of cybersecurity should improve. The issue, of course, is that cybercriminals are now targeting software supply chains; inserting malware into a DevOps workflow creates a potential opportunity to compromise any number of downstream applications. Organizations are now racing against time to secure their software supply chains.

    The good news is that no developer intentionally sets out to build an insecure application. It’s just that far too many of them have been conditioned to think of security as an obstacle they need to overcome rather than address. However, the more tools they are provided to remediate as many vulnerabilities as possible, the better off everyone concerned will be.