devopsy.kinsta.cloud

Author: Bill Doerrfeld

  • Introduction to Cloud Custodian

    Introduction to Cloud Custodian

    Organizations continue to increase their footprint in the cloud. Yet, applying governance and enforcing policies across multiple cloud environments at scale is a challenging endeavor. Organizations typically have governance policies written in English, but translating them into enforceable code has historically required some custom workarounds. In the same vein as infrastructure-as-code (IaC), DevOps engineers now seek a similar GitOps approach to implement shared policies across cloud-based workloads.

    Cloud Custodian is one such toolset that can manage and enforce cloud policies in a standardized format. Using Cloud Custodian, a Cloud Center of Excellence has the building blocks to create policies for security governance, development guardrails and cloud cost optimizations. The free, open source project, which has been steadily evolving for years, recently gained its incubation status with the Cloud Native Computing Foundation (CNCF) and to date boasts over 350 contributors on GitHub.

    I met with Kapil Thangavelu, Cloud Custodian creator and maintainer and CTO at Stacklet, to learn more about Cloud Custodian and what features we should anticipate from the project in the near future. Below, we’ll provide a comprehensive introduction to Cloud Custodian, exploring its history, how to use it and its core benefits. We’ll also analyze a few sample policies to showcase how Cloud Custodian functions in practice.

    What is Cloud Custodian?

    Cloud Custodian got its start as an internal tool developed within Capitol One toward the end of 2015, explains Thangavelu. At the time, engineers were grappling with the newfound realities of managing the cloud at scale and realized that having one-off ad-hoc policies across teams was causing friction. “It’s really hard to manage and enforce policies at scale,” says Thangavelu. So, engineers toyed with creating a more GitOps approach to policies around governance and security that could be run as an internal SaaS solution.

    Thus, Cloud Custodian was born. The YAML DSL tool allows you to easily define rules to manage cloud infrastructure. It can offer real-time enforcement and produce event-based responses. For example, you can create policies that require tags on resources or one that turns off instances during off-hours. Or, a policy could prevent a developer without the proper privileges from accidentally making an open load balancer. Cloud Custodian supports AWS, Azure, and GCP and policies are specified to a resource type, whether it’s EC2, ASG, Redshift, CosmosDB or a PubSub Topic.

    Thangavelu describes Cloud Custodian as a bunch of Lego bricks that developers can pick up to build guardrails in arbitrary ways. Instead of shipping with canned rulesets, “Cloud Custodian makes customization itself a first-class citizen,” he says. Capital One open sourced the project and contributed Cloud Custodian to CNCF in August 2020.

    Example Policies

    Cloud Custodian comes with a library of actions and filters. Let’s take a look at a few example policies across the three main cloud providers. But first, to install Cloud Custodian from PyPI, you can initiate the following command:

    $ python3 -m venv custodian
$ source custodian/bin/activate
(custodian) $ pip install c7n

    Let’s consider an example policy that would improve security hygiene on AWS. Below is one such policy that creates a CloudWatch Event to be triggered anytime a user logs in from an invalid IP address. This could be used to ping a security administrator to investigate. With this rule, it’s also possible to turn on an auto-remediation function.

    policies:

  - name: invalid-ip-address-login-detected
    resource: account
    description: |
      Notifies on invalid external IP console logins
    mode:
       type: cloudtrail
       events:
          - ConsoleLogin
    filters:
      - not:
          - type: event
            key: 'detail.sourceIPAddress'
            value: |
               '^((158\.103\.|142\.179\.|187\.39\.)([01]?[0-9]?[0-9]|2[0-4][0-9]|25[0-5])
               \.([01]?[0-9]?[0-9]|2[0-4][0-9]|25[0-5]))|(12\.([01]?[0-9]?[0-9]|2[0-4][0-9]|25[0-5])
               \.([01]?[0-9]?[0-9]|2[0-4][0-9]|25[0-5])\.([01]?[0-9]?[0-9]|2[0-4][0-9]|25[0-5]))$'
            op: regex
    actions:
      - type: notify
        template: default.html
        priority_header: 1
        subject: "Login From Invalid IP Detected - [custodian {{ account }} - {{ region }}]"
        violation_desc: "A User Has Logged In Externally From A Invalid IP Address Outside The Company's Range:"
        action_desc: |
            "Please investigate and revoke the invalid session along
            with any other restrictive actions if appropriate"
        to:
          - CloudAdmins@Company.com
          - SecurityTeam@Company.com
        transport:
          type: sqs
          queue: https://sqs.us-east-1.amazonaws.com/12345678900/cloud-custodian-mailer
          region: us-east-1

    Next, let’s consider a policy that would add some degree of governance for developers working on Azure. One simple yet popular policy involves tagging the ownership details to individual resources. This policy, therefore, will tag all resource groups with the creator’s email address:

    policies:
  - name: azure-auto-tag-creator-resource-groups
    resource: azure.resourcegroup
    description: |
      Tag all existing resource groups with the 'CreatorEmail' tag; looking up to 10 days prior.
    actions:
     - type: auto-tag-user
       tag: CreatorEmail
       days: 10

    Finally, let’s consider a policy that could aid in cost optimization on GCP. Below is a sample policy to enforce minimal CPU utilization for autoscalers on GCP. Implementing such a policy could help avoid underutilization to combat rising cloud costs.

    vars:
  min-utilization-target: &min-utilization-target 0.8
policies:
  - name: gcp-autoscalers-enforced
    resource: gcp.autoscaler
    mode:
      type: gcp-audit
      methods:
        - v1.compute.autoscalers.insert
    filters:
      - type: value
        key: autoscalingPolicy.cpuUtilization.utilizationTarget
        op: less-than
        value: *min-utilization-target
    actions:
      - type: set
        cpuUtilization:
          utilizationTarget: 0.8

    Benefits of Cloud Custodian

    By automating a lot of the tedious policy management away, Cloud Custodian could reduce risk and accidents through more streamlined cloud governance. “It solves the natural problems when infrastructure is in everyone’s head,” says Thangavelu. By aggregating ad-hoc scripts and unifying policies across an organization, you could immediately instigate new rules without manually reminding all members of an organization, which could take years.

    For those familiar with Open Policy Agent (OPA), you may notice some overlap in the objectives, as both are engines for enacting cloud-native policies. Compared to OPA, Cloud Custodian has some developer experience perks. For one, you don’t have to use Rego, as the policies are written in YAML, which is a familiar configuration language for DevOps engineers. Cloud Custodian uses abstractions on event runtimes for each cloud provider. Furthermore, compared to OPA, you don’t need to bind the engine for a particular problem domain, says Thangavelu, as Cloud Custodian is specifically bounded to cloud governance and management.

    Cloud Custodian is also battle-tested. Many big names are using it in production, including Capital One, HBO Max, Intuit Inc, JP Morgan Chase & Co, Siemens and Zapier. Across its adoption, the tool tends to either be utilized by a Center of Cloud Excellence, security teams to conduct reporting or reveal real-time incidents or the CFO to direct cost optimizations.

    Future of Cloud Custodian

    Cloud Custodian maintainers describe the project as a consistent firehouse. Since the project tracks changes in all cloud provider resources in real-time, it’s quite a dynamic toolset. In addition to monthly updates, users can expect many additional updates on the horizon, including support for Tencent Cloud as well as additional Kubernetes support. Additional work is being done around shift-left capabilities to meet the developer where they are, says Thangavelu.

    For more information, check out the Quickstart Guide to Cloud Custodian, or view the project roadmap here.

  • Four Causes of Technical Debt in DevOps

    Four Causes of Technical Debt in DevOps

    Ideally, DevOps should retain a lean footprint, but avoiding technical debt is easier said than done. As such, over half of IT leaders report technical debt is a big or critical problem. Without routinely addressing technical debt, DevOps teams can easily face inconsistencies during deployments. Versioning can get out of hand without consistent upgrades and aging legacy components can be a roadblock to development fluidity.

    Simply put, letting too much technical debt pile up isn’t all that great.

    I recently met with Kurt Andersen, SRE architect at Blameless, and Matt Davis, SRE advocate, to gather insights on the sources of technical debt in the field of DevOps. According to Davis, “Tech debt is something that you really want to keep chipping away at.” Below, we’ll highlight some common causes of technical debt and suggest ongoing practices to help tame it.

    1. UI-Based Deployment Patterns

    UIs are great for quickly configuring DevOps, but if your deploys aren’t codified, this reliance on UIs could come back to haunt you in the form of technical debt. A lot of the time, explained Davis, infrastructure teams have a tendency to just use the GUI of their given cloud provider to initiate deployments. Perhaps the team is wrapped up in feature velocity or a deadline to release.

    For example, you could use a UI to quickly create a database, but this is likely not a repeatable and robust process. Especially as an organization scales its cloud adoption and uses multiple clouds, you start to produce “snowflake” deployment patterns across teams, says Andersen. This could lead to inconsistencies around different deployment options per region.

    It’s not easy to catalog a configuration if it’s not written in code, and the more undocumented UI-based deployment patterns are used, the more silos of knowledge you create. “If not codified with Terraform, you have to document what you did—code gives you that documentation,” said Davis.

    Solution: Rely less on the GUI and codify your deployment patterns with infrastructure-as-code (IaC).

    2. No End-of-Service Life Cycle

    Development is typically more concerned with feature velocity than planning for service deprecation. Yet not envisioning the entire life cycle or setting an off-ramp from the start can produce more technical debt later on. “No one pays attention to planning for retirement, and at the end-of-life, services are hard to retire,” described Andersen.

    What you end up with are these “senile services” that aren’t doing that much but remain critical to the business’s operations. These services may be challenging to migrate. Or, they may be the product of unknown shadow or zombie APIs. For example, a marquee customer may depend upon a certain legacy endpoint, making the service difficult to evolve.

    Not planning for the end of the service lifecycle can hold up your development from more efficient ways to work. For example, if one “senile service” has never used a particular commit strategy before, it could stall agility when revamping the deployment pipelines, says Andersen. Thus, it’s best practice to set clear deprecation timelines and plan for the future.

    Solution: Plan the end-of-service life cycle from the upstart.

    3. Lengthy Dual-Track IT

    Another area that can produce technical debt in DevOps is dual-track IT. This is when engineering teams maintain one core stack in addition to an experimental structure and shift toward adopting the experimental structure over time. Development is constantly overseeing migrations to new technologies, yet they rarely plan for the coexistence of both stacks.

    While splitting an old and new stack into distinct units sounds like a temporary measure, some of these large transitions simply take longer than you think. For example, SoundCloud’s journey to refactor its monolith into microservices was an eight-year process.

    Solution: Realize dual-track IT might take longer than you think and plan accordingly.

    4. Intermittent Version Upgrades

    Third-party software is constantly upgrading to new versions. Especially if you’re integrating many open source dependencies into your stack, you’ll find the cadence of version upgrades to be quite fast. These upgrades are often imperative to plug new vulnerabilities as they emerge in the software supply chain.

    But upgrading versions and base images and having a patch management scheme that works from environment to environment requires a lot of work, Davis cautioned. Kubernetes version upgrades, for example, can take a lot of time. Yet, if your upgrades are not matching the pace of change, added Andersen, they can quickly pile up and leave you left behind with a mountain of work. “If you’re not upgrading monthly, you’re in danger.”

    Solution: Have a continual upgrade process in place.

    Technical Debt: Keep Paying, Cleaning, and Pruning

    Technical debt is something you need to grapple with before it gets out of control. But it’s not always apparent where it lives. Looking toward the future, automated tech debt scanning could help discover where technical debt is and provide remediation points as part of ongoing software stack auditing.

    You don’t want to get to the point where you accumulate and accumulate technical debt, only to burn it all to the ground in one disastrous fell swoop, said Davis. Similar to the way you slowly pay off a mortgage each month, technical debt is something you must continually keep an eye on and make reducing part of your team’s continuous workflow.

    Or, consider the old camping mantra of always leaving the campground cleaner than you found it. You pick up a little bit of extra trash and take it out with you, making the environment better for everyone. Or, consider how a gardener must prune the branches of a fruit tree to help it grow. Pick the analogy that works for you and see if you can apply it to continually reduce tech debt in your DevOps processes!

  • GraphQL: Security by Obscurity Just Isn’t Enough

    GraphQL: Security by Obscurity Just Isn’t Enough

    The debate about how to secure GraphQL rages on. Many organizations are hesitant to adopt GraphQL for public-facing APIs as there is no precise method to handle authorization concerns as of yet. Without a role-based access layer to enable fine-grained permissions for each field (and underlying services that GraphQL might wrap), the query language can be prone to access control issues.

    One method to hamper GraphQL access is to turn off introspection features altogether. Some may assume that hackers can’t perform introspection on an endpoint if it’s not visible. But this isn’t entirely true—even though an “internal” endpoint may not be fully documented, any endpoint on the public web can be searched and introspected quite easily by brute-forcing queries or through field suggestions.

    I recently met with Karim Rustom, cybersecurity engineer, Escape Technologies. According to Rustom, “making something obscure isn’t enough to make it secure.” Leveraging a single slip-up in access control could be used as a doorway into these systems. Although it’s a best practice to limit introspection, it’s likely not enough to protect production GraphQL endpoints, he said. Below, we’ll consider some common GraphQL threats and alternative solutions to retain a safe GraphQL implementation.

    Common GraphQL Threats

    Having run thousands of scans on production GraphQL endpoints, Rustom found common trends emerge around GraphQL threats. First, he noticed the potential for denial-of-service attacks to be quite high. GraphQL, by design, is intended to query whatever data the client requires. But sometimes, these queries can return a mountain of data.

    If GraphQL owners don’t place a limit on what can be requested in a single query, the platform can easily fall victim to denial-of-service attacks and service interruptions. And limits aren’t only necessary for bulk read-only operations—although GraphQL is primarily used for fetching data, some methods might initiate intense server-side processing, which should be limited in some form. According to Rustom, such data manipulation methods are especially pervasive in FinTech, food delivery or health care-related applications.

    Another rampant GraphQL threat lies in access control issues. This comes as little surprise, as access control is ranked as OWASP’s most common web application security risk. Often, when creating new objects or new queries, a developer might forget to add the correct authorization, said Rustom. This can become a huge problem and lead to privilege escalation.

    Finally, too many production GraphQL implementations rely on security by obscurity. “Folks think that disabling introspection is a layer of security—we highly stress that it’s not,” said Rustom. He recalled performing penetration testing for a certain client; even though they had disabled introspection, the team was able to fetch ample open paths and discover an unsecured doorway to divulge secret internal knowledge.

    Solutions and Best Practices

    To avoid some of the vulnerabilities presented above, it’s recommended to first place request limits on overly complex GraphQL calls to prevent denial-of-service attacks. One nuance here that Rustom recommended is to consider request complexity—not in terms of the query size, but the overall processing required. He suggested assigning scores for every query and permutation to help inform the decision about whether or not to process the incoming call.

    Furthermore, it’s recommended to disable field suggestions as the feature could be prone to abuse. Using field suggestions, hackers can leak a lot of information to build a partial or complete introspection. These tactics are similar to the word-busting techniques black hats use to perform introspection on REST endpoints.

    Although it’s a good practice to limit the possibility of surveillance, hackers can still discover systems and leverage other common vulnerabilities like broken access control. Rustom suggested other tactics, such as luring attackers into a honeypot or logging them into a fake account. Or, creating error messages that are intentionally misleading for a attacker.

    To retain a safe GraphQL implementation, organizations can use pre-made free and open source tools to audit their surface area and scan for known vulnerabilities. For example, GraphQL.security quickly scans your GraphQL URL endpoint to discover vulnerabilities. Or, Graphinder can examine a subdomain to discover GraphQL endpoints, which could aid auditing purposes. There’s also middleware that wraps a GraphQL service with additional security, such as GraphQL Armor and GraphQL Shield.

    Future of GraphQL in the Market

    There appear to be many flavors of GraphQL throughout the market. We’ve seen it emerge as a mechanism to power backend microservices. We’ve also witnessed the technology adopted as a layer to centralize multiple REST services with a more usable data retrieval mechanism for frontend engineers. Some GraphQL endpoints also run on the public web to serve end clients and external partners.

    “GraphQL is a beautiful technology, just lacking in security,” said Rustom. And unfortunately, his outlook for API security looks grim—amid political turmoil and economic instabilities in regions like Europe, some companies may seek to cut cybersecurity investments to stay lean. For companies that haven’t yet suffered a breach, security is easy to ditch to slim budgets, but it should remain a priority, said Rustom. “Keep securing GraphQL.”

  • How to Prevent Turnover in DevOps Teams

    How to Prevent Turnover in DevOps Teams

    Turnover is all too common in software engineering. The Bureau of Labor Statistics 2021 report found that the average software engineer turnover is 57.3%. It’s a programmer’s market, and these sought-after folks might jump ship for a higher salary, better work-life balance, different team dynamic or perhaps even for a company with a better ESG footprint. Yet, constant changeups in a development team can stifle progress when there’s pressure to meet new digital demands and rapidly deliver new features.

    IT skills shortages can impede innovation across the board. But turnover can be especially problematic in the field of DevOps, as the practice relies on collaboration between teams and a shared sense of operational standards. Here, a gap in shared knowledge can impair ongoing platform maintenance.

    I recently met with Nathan Sutter, global VP of engineering at CoderPad, to examine the core reasons behind engineering turnover and look at specific ways to prevent it in DevOps teams. In a nutshell, he advocated for reducing context switching, giving both engineering and product teams a seat at the table and not underestimating your teammates’ flexibility.

    Reduce Context Switching

    Engineers like having ownership and pride around what they’re creating—take that away, and it can lead to burnout and even an exit from the company. The more you context switch, describes Sutter, the less you focus on building a core project and the less attached you feel to the code or architecture you’re creating. “Engineers like to build cool stuff, ship it, and use it,” he said.

    Context switching may occur for a couple of reasons. First, it may just result from poor planning, in which case engineering time is wasted. Secondly, context switching might arise due to legitimate emergencies, such as incident response or necessary bug fixes. Furthermore, enterprises traditionally ship operational concerns to a production team, thus increasing the number of applications a developer oversees and decreasing their role in ongoing maintenance.

    Instead, Sutter is a proponent of allowing developers to own everything about an application end-to-end. Essentially, the DevOps or infrastructure team builds the scaffolding and the application teams own the rest, such as writing Terraform configurations, Docker containerization, monitoring and alerts, and on-call assistance. In other words, you operate what you build and own the component throughout its entire lifecycle. This practice has already been embraced by full-cycle developers at Netflix.

    Specifically, in DevOps, the on-call cycles can be challenging when you have so many things to own. “When you have 25 apps to look after, something is always going off,” said Sutter. “This is an “exhausting, thankless and horrific way to work.” This imbalance can affect the work-life balance and decrease your time to focus on upgrading core infrastructure.

    Don’t Separate Engineering and Product Teams

    Another setback that might manifest itself in turnover has to do with inefficiencies caused by siloed teams. For example, suppose a product team spends months outlining a complex software design in a black box. In that case, they will most likely encounter many issues once the schematics are finally sent to engineering teams. Without collaboration between the two groups earlier on, organizations can increase time-to-market for new features and encounter frustration between teams.

    Instead, Sutter believes product and engineering teams should counterbalance each other and collaborate more in the product design phase. Whereas product teams can “skate where the puck is going” and identify trends in the market, engineering teams figure out how to technically implement these designs and build a sequence of work that’s scalable for the long-term. Of course, the other side rings true, too—engineers can overplay their hand and plan minute details too far into the future, described Sutter. This underpins a need for ongoing collaboration where both sides have a voice at the table.

    Don’t Underestimate Flexibility

    Even though software engineers like to have a sense of ownership, we shouldn’t discourage flexibility—people easily become bored working on the same thing for years and years. There’s also the fallacy of sunk cost to keep in mind, which states that we tend to value things more because we’ve put more time and effort into them. Thus, providing flexibility to pivot when it makes sense can increase overall satisfaction and output.

    Accordingly, flexible management is also crucial to embrace pivots when they are necessary. For example, if a project is well underway but an engineer identifies a new solution that is more elegant, team leads should be open to recognizing and acting on changes. But to realize this sort of relationship, trust and openness must be bidirectional, said Sutter. If engineers can’t express their ideas or are afraid to tell their boss they’re wrong, these important conversations can’t happen.

    A flexible structure is also necessary to attract talent that prefers more modern work-life balance. As big tech companies with multimillion-dollar commercial leases now want workers to return to physical offices, small-to-medium companies now have an unprecedented upper hand to attract and retain talent if they embrace flexible remote-hybrid working options, added Sutter.

    What Matters Most?

    The majority of developers working on open source projects are open to changing jobs in 2022, according to the EDB Open Source Talent Survey. Job satisfaction is challenging to wrap your head around, and each engineer may be motivated by various reasons.

    But having assembled and led many software engineering teams over the years, Sutter recognized an interesting change in the attitudes of what matters most to software engineers in their day-to-day job. Pre-COVID, software engineers were definitely interested in higher compensation, he said. After that, it was building something they’re proud of, team camaraderie, the worth of a brand name and a positive work-life balance. Post-COVID, compensation and other benefits are still just as valid. Yet, hybrid flexibility is often table-stakes—some are even willing to take a pay cut to obtain these benefits. This underscores the high importance of flexibility in managing modern software engineering teams.

    Executives and engineering leads have their work cut out for them when it comes to obtaining and retaining top talent. Yet interestingly, many roadblocks are not so much technical problems but people and organizational problems. As we’ve covered, reducing context-switching and placing more ownership on singular products could increase satisfaction. One of the tenets of DevOps is also de-siloing teams to increase collaboration. Reducing toil could also go a long way toward increasing satisfaction. Whatever your chosen strategy, it’s always good to “display openness and understanding to hear people when they’re not happy,” said Sutter.

  • Exploring the Emerging Role of FinOps

    Exploring the Emerging Role of FinOps

    Cloud overspending is becoming a top-of-mind budget item for the modern CFO. To direct resource utilization in the cloud, some organizations are turning to increased automation and more diligent cloud procurement processes. As part of this momentum, new roles are emerging to oversee cloud financial operations.

    Amid rising cloud costs, FinOps is emerging as a helpful discipline. It is not always a specific role but more often a culture that blends engineers, finance and business areas to participate in dynamic cloud cost management. But getting all these players on the same page isn’t always easy — new findings say influencing developers to actually make optimizations is a top challenge.

    The FinOps Foundation recently released The State of FinOps 2022 report, a deep exploration into the role and the state of FinOps adoption across 1,000 companies worldwide. Below, I’ll highlight the key insights from the report to understand current trends and consider how organizations might implement similar tactics to better operationalize their cloud impact.

    Who Is Practicing FinOps?

    What type of person is involved in FinOps? Well, folks come from diverse backgrounds, spanning cloud financial management, IT or DevOps or consultants. Many are also new to the practice in general—the average career tenure is roughly three years. FinOps tends to be a well-paying field—45% of practitioners report salaries of over $100,000 annually.

    FinOps also tends to be associated with larger enterprises—45% of respondents came from organizations with 10,000 or more employees. In terms of industry, it is most common in financial services and IT companies, although studies show the practice is proliferating in many other sectors too. Geographically, FinOps is most adopted in primarily adopted in North America and Europe.

    Team Dynamics

    The average team size is five people, although many feel they’re a “one-person show.” Interestingly, more mature adoption tends to involve a larger team, upwards of 20+ members for established teams. There’s also a notable increase in FinOps within organizations managing between $50 million and $500 million of cloud costs.

    FinOps must interface with many levels of the company, sharing projections with both development and executive leadership. Most FinOps teams report directly to the CTO or other C-suite members. This makes sense, as an executive directive was found to be the most common way to promote a FinOps culture, followed by organic and grassroots adoption.

    Technology and Tools

    In general, FinOps tends to be achieved through cloud awareness and dedicated tooling. The most mature practitioners are “all-in on cloud.” The largest subgroup (24%) runs 1,000 to 5,000 computing instances in the public cloud, hosting technology like virtual machines, containers and serverless functions. Simply put, the average FinOps practitioner has a lot on their plate.

    In terms of tooling, there’s not one cloud cost management suite to rule them all. Teams use an average of 3.7 tools. A variance of tooling here is likely due to rising hybrid multi-cloud architectures and the need to manage costs separately within each cloud environment.

    As such, FinOps tooling includes a mixture of native cloud provider tools and other middleware. Surprisingly, after AWS Cost Explorer, the second most common tool is a homegrown tool. After that, Cloudablity, CloudHealth and Azure Cost Management rank as popular primary tools.

    Top Challenge: Collaboration

    The top barrier has to do with interdepartmental collaboration. Getting engineers to take action on cost optimization was the biggest challenge, cited by 30% of respondents. Other common challenges included accurately forecasting cloud spending (29%), convincing the organization to adopt FinOps (22%), enabling automation (20%) and reducing waste or unused resources (19%).

    Developers in the flow likely don’t want to be burdened by new cost optimization rules, so FinOps may encounter an uphill battle trying to affect low-level change. It will probably take a combination of shift-left IDE-based cost projection tooling and a decree from upper management to jumpstart this change.

    Besides getting everyone on board, FinOps is often tasked with chargeback models—48% of respondents report using chargeback to report cloud costs, the report found. Yet, correlating cloud costs to a single department or team is challenging with so much shared infrastructure. As such, shared costs remain a challenge; so much so that many practitioners don’t even split them up.

    There is also the need to resolve unexpected anomalies and increases in cloud costs, and it can be hard to distinguish between the two. As a result, 53% of respondents said it takes days for their FinOps teams to respond to unexpected cost increases. In the future, finding ways to shorten this timeframe will help save the company money.

    Next Goal: Getting Engineers to Take Action

    The vast majority (94%) of organizations are wasting money in the cloud, according to the 2022 HashiCorp State of Cloud Strategy Survey. Idle or overprovisioned resources can unnecessarily increase the cloud budget, making the need for FinOps evident. Investing in cloud optimization is also another tactic to meet sustainability pledges.

    FinOps is an emerging role that can help forecast spending and reduce waste. But the practice does require time and resources to reach an ROI. For example, a mature team only has a 5% deviation comparing forecasted spend and actual spend, whereas a less mature team can have as high as a 20% deviation. To improve, teams will require automation for evaluating and communicating cloud projections and more established anomaly management best practices.

    But above all, organizations must grapple with the number-one problem: Getting engineers to take action. Perhaps this is due to the fact that most respondents share cost data with technical teams monthly. Instead, more frequent cost transparency would match the pace of agile development and might empower more engineers.

    The State of FinOps 2022 surveyed 1,056 respondents on their FinOps roles. The yearly study is conducted by The FinOps Foundation, a Linux Foundation program. For more details, you can read the entire report here.

  • Study: Real-Time Data Has A Transformative Effect

    Study: Real-Time Data Has A Transformative Effect

    We all know that data is the heart of the modern digital business. And real-time data is arguably the next evolution—having up-to-date data can empower energy savings and optimizations to reduce cost. Real-time data processing also aids fraud detection and improves health care responsiveness.

    Especially in DevOps, access to real-time data is necessary to reduce incident response time and inform ongoing SRE objectives. Leveraging performance data can also empower future decision-making and open new revenue opportunities. In essence, real-time data can have a transformational effect on the operations of a digital business.

    DataStax and ClearPath Technologies recently released the 2022 State of the Data Race Report. The study found that the majority of data leaders are using real-time data to create additional value. The report also suggested that working with cutting-edge real-time data technologies increases developer experience, which is good to maintain amid ongoing talent shortages.

    Below, I’ll review the key findings from the study to infer how organizations can effectively use these technologies and how they can mitigate roadblocks that arise throughout the process.

    The State of the Data Race

    Simply put, customers expect real-time experiences. And it’s not only the end consumers—developers are also increasingly accustomed to building in this modality. This is evidenced by the recent rise of asynchronous, event-driven architectures. The survey depicted how fundamental real-time data is to modern software, finding that 78% of respondents agreed that it is a “must-have,” not just “nice-to-have.”

    The investment appears to be paying off. A full 71% of all respondents said they can directly tie revenue growth to real-time data. Not only does it seem to have a transformative impact on revenue growth, but it enhances productivity, too—66% of real-time data-focused companies agreed that developer productivity has improved. Naturally, the report found that developers are the ones most likely to be working with real-time data.

    Although modern databases and real-time data processing tools are more readily available these days (and open sourced), the maturity of data handling is not homogenous. Based on the classifications of this report, “data leaders” made up only 10% of respondents.

    Roadblocks to Value

    The road to data maturity has a few obstacles that could limit adoption. First is data complexity—33% said this is one of the biggest barriers to leveraging real-time data. This was closely followed by controlling data costs (32%), data accessibility (30%) and available skillsets (28%).

    Most notably, a skills gap continues to be a commonly cited hindrance throughout industry reports and across technology niches. Interestingly, a lack of in-house talent rises slightly higher in organizations with a high real-time data maturity. Those surveyed in this category said that their top barrier is finding the appropriate skills in their business unit.

    Cybersecurity is also a top concern—51% say security challenges are a big concern when it comes to the use of real-time data. As the report surmised, the business-critical nature of real-time data raises its worth as a commodity and area for attack. Furthermore, amid new data sharing regulations, IT departments are now under more pressure than ever to uphold data security and avoid the leakage of sensitive personal information.

    Unlocking Real-Time Data

    This area has a lot of potential; 86% of developers at organizations with a strategic focus on real-time data said that the “technology is more exciting than ever,” the report found. Real-time data is beginning to be applied at the edge, where AI/ML can be trained to give a more responsive advantage to modern applications. Yet, organizations must prioritize its adoption and mitigate the roadblocks mentioned above to unlock the benefits of real-time data.

    Prioritizing real-time data will take more than tools—it will take vision. According to the study, those who created more value with real-time data initiatives were found to:

    • Have clear product owners
    • Have business unit accountability of data
    • And have department representatives working together in a cross-functional team.

    As you can see, these foundational goalposts have less to do with technology and more with lucid leadership and collaboration. That being said, an investment in the appropriate tooling is undoubtedly required to achieve real-time data.

    For example, this tooling might encompass NoSQL databases optimized for real-time or Apache Cassandra for high throughput. The transition to real-time will likely require streaming and messaging technologies as well. Less obvious, however, is an API layer to sit between the applications and the database. This element could greatly help open data accessibility in a standardized fashion.

    The State of the Data Race: Final Thoughts

    Working with real-time data can be a differentiator and have a transformative impact. As a result, most developers and leaders see real-time data as an exciting area. Yet, organizations must reduce data siloes to open up data access to realize its benefits. It will also take collaboration between data scientists and business divisions to construct meaningful outcomes.

    The State of the Data Race report surveyed 500 leaders across various industries. Above, we’ve summarized the key points from the survey. Readers are encouraged to pick up a copy for more nuanced data points and further commentary.

  • 90% Of Orgs Say Multi-Cloud Helps Achieve Business Goals

    90% Of Orgs Say Multi-Cloud Helps Achieve Business Goals

    Multi-cloud is the new normal. And thankfully, this new normal isn’t just happenstance—it’s driving positive outcomes.

    New findings show that most organizations benefit from using multiple cloud providers. Multi-cloud adoptions are helping achieve business goals, such as improving reliability and scalability and increasing overall security and governance. However, nearly all organizations (94%) admitted that they have some avoidable cloud spending and skills shortages continue to hold back a fully-realized multiple-cloud success.

    HaschiCorp recently released the 2022 State of Cloud Strategy Survey: Making Multi-Cloud Work. In addition to showcasing the positive outcomes of multi-cloud adoption, the report found a trend toward automation frameworks that help operationalize multi-cloud. Below, I’ll explore the report’s key takeaways to see what CloudOps and platform teams can garner from current multi-cloud trends.

    Multi-Cloud is Achieving Business Goals

    First off, the use of the public clouds is increasing dramatically—the majority of public cloud workloads will likely double in the next two years. To support this influx, more organizations are moving toward multi-cloud. According to the HashiCorp survey, 60% are already multi-cloud and 21% plan to use multiple clouds within the next year.

    The good news is that a multi-cloud strategy appears to be working. A full 90% of respondents said multi-cloud is helping their organizations achieve their business goals. This figure is an impressive jump from HashiCorp’s 2021 survey, in which only 53% of organizations said multi-cloud was helping them achieve their business goals.

    So, what exactly are companies extracting from multi-cloud adoption? Well, the topmost business goal driving multi-cloud is reliability, at 46%. This is followed by digital transformation (43%), scalability (42%), security and governance (41%) and cost reduction (39%). As we can see, multi-cloud equally benefits various areas.

    Centralized Automation Supports Multi-Cloud

    With the rise of multi-cloud, CloudOps is becoming its own discipline. As such, it’s common for an organization to build a dedicated Cloud Center of Excellence (CCoE) or strategy team to manage cloud operations—86% said they relied on a cloud platform team. Such a team is often in charge of standardizing cloud services, developing cloud best practices, creating operational policies and related duties.

    To aid cloud platform teams, cloud automation is becoming increasingly important. The survey found that 99% see automation as important for operationalizing multi-cloud infrastructure. Multi-cloud automation could help solve inconsistencies across varying cloud workflows. Respondents also found it can be applied to deliver more flexible infrastructure, improve security and governance and better utilize cloud resources. Improving security is especially important here, as 89% said security is a key driver of cloud success.

    So how are these automations built? Well, folks are rarely developing cloud automation tools from scratch—only 27% do so. It’s far more common to either use open source or commercial tools for cloud automation. It’s pretty equal across the board whether these tools are run ‘as-a-service’ or are self-hosted.

    Regardless of the tooling type, companies are experiencing a broad set of benefits from using middleware to manage their multi-cloud operation strategy. 49% rank scalability as a benefit. This is followed by infrastructure provisioning (42%), cost optimizations (40%) and more flexible deployment options (39%).

    (Nearly) Everyone is Overspending on Multi-Cloud

    Unsurprisingly to some, the study found that 94% of organizations are wasting money in the cloud. The following were the top cited areas that contribute to cloud overspending and waste: Idle or underused resources (66%), overprovisioning resources (59%), a lack of needed skills (47%) and manual containerization (37%).

    The cloud overspending dilemma necessitates forethought into optimizations and reduction of idle resources. As I’ve previously covered, some organizations are even tinkering with employing a cloud economist, or Cloud CFO, to keep escalating cloud fees in check. To help this mission, 43% say consistent and automated tooling has or would lead to better utilization of cloud resources.

    Although wasteful spending is pervasive, the majority of companies are forecasting their cloud spending correctly. More than half (57%) said that in 2021 their organization’s projected cloud spend aligned with their actual cloud spend.

    Getting the Most Out of Multi-Cloud

    Multi-cloud is becoming more common across all IT divisions—as is middleware to operationalize and standardize logic between multiple clouds. And, widespread business success with multi-cloud signals increased maturity in this area.

    Yet, multi-cloud doesn’t come without its challenges. Skill shortages rank as the highest factor impeding an organization’s ability to operationalize multi-cloud. Businesses must face other challenges, such as siloed teams, compliance and risk management and lack of training in order to fully realize multi-cloud. Standardized operating models, along with shared workflows and policies, are possible ways to combat these roadblocks.

    The HashiCorp study surveyed 1,000 technology practitioners and decision-makers from around the world. HashiCorp also partnered with Forrester to provide additional analysis of the findings. To read the results for yourself, check out the full report here.

  • Understanding the Importance of Developer Experience

    Understanding the Importance of Developer Experience

    If you’re creating a developer tool, you’re bound to run into this dilemma—you know the inner workings of it really well, but others don’t. And without solid documentation and samples to follow, it can be challenging for another person to get started. Friction with the tools we use can contribute to burnout, inhibit creativity and slow down innovation.

    Developer experience (DX) is becoming integral for any software company to consider. I recently met with Christine Spang, CTO and co-founder of Nylas, to consider the benefits of investing in developer experience. Below, we’ll take a look at what exactly DX is and consider some best practices for implementing it.

    What is Developer Experience?

    Developer experience (DX) is similar to user experience but for developer-facing tools. DX considers the developer’s journey while developing with a certain framework, language or platform. “Developer experience is the amount of friction a developer experiences between having clarity on what they want to implement and getting to the point where you have created that thing,” said Spang.

    For example, consider integrating with an API to solve a common use case like payments integration. There are many steps involved here, from reading the reference documentation to getting an API key, testing calls and understanding how the service responds. After developers initially integrate the API into their application, they may also have to make continual updates as the service versions. Throughout this journey, DX can be thought of as “how hard it is to figure out how to translate from idea to output,” said Spang.

    It’s also important to note that DX isn’t only relegated to public-facing platforms. Having a quality developer experience is just as crucial for internal software. The urgency to improve internal developer experience is increasing with the surge of APIs being developed, especially since the majority of them (58%) are private APIs, according to Postman’s 2022 State of API Report. Depending on the size of a company, internal DX can encompass many areas.

    Best Practices for Investing in Developer Experience

    So, what are some methods to improve developer experience across the board? Well, having up-to-date documentation, code samples, better examples and standardized documentation rank high as positive developer experience traits. Spang also shared some helpful best practices to improve DX:

    Provide scaffolding for the most common cases. Measure patterns around the kind of things people are building with the platform and offer scaffolding to support the common use cases. This might equate to a getting started guide for a common scenario.

    Offer sample code libraries. It’s a lot easier to edit code than write it from scratch. With that in mind, create pre-baked environments where developers can play around. Offering working apps, use case templates and glue code can be a practical means to help developers get going more seamlessly.

    Start with the reference manual. Walkthroughs and sample code should never replace a complete reference manual. It’s essential to have up-to-date documentation to cater to advanced users from the get-go. From here, you can generate Postman Collections or interactive sandboxes built around the reference specification.

    Measure DX success and iterate. One way to monitor developer experience metrics is to record the onboarding flow and measure conversion rates. Where APIs are concerned, this can be tracked by asking how many people get through to making calls? What is the mean time it takes to make the first call? Knowing such metrics can inform how effective your tool’s self-service features are.

    One point that’s important for DX is walking the fine line between too much custom work and focusing on the core project. On that note, Spang recommended “making sure whatever you’re building maps to the overall vision of the company.” In essence, if you’re not a consulting shop but a product company, you should stay true to that objective.

    “You want to make the path for what people commonly want to do very easy,” said Spang. “Yet, you don’t want advanced users to feel constrained by the platform.”

    Examples of Quality Developer Experiences

    For those creating developer-facing services, it’s helpful to have a framework to model. So, what are some examples of quality developer experiences?

    Well, top public APIs are great examples of where the bar is set for developer expectations these days, said Spang—services like Twilio or Stripe demonstrate relatively seamless developer journeys. Spang also pointed to companies with a strong DevOps and CI/CD background, like Etsy, who have for years invested in usable backend infrastructure that accelerates internal development.

    For other sources of quality developer experience, consider the high-ranking developer-favorite technologies from the latest 2022 Stack Overflow survey. Engineers love working with technologies such as PostgreSQL, AWS, Phoneix and Docker. The newer, trending languages tend to be increasingly usable, too. For example, 86% of developers said they love working with Rust while 76.65% dread working with Objective-C.

    The Overall Benefits of DX

    An investment into developer experience can help improve agility and developer satisfaction, said Spang. For one, an easier path for development increases the velocity in how you ship and evolve projects over time. Secondly, developer experience equates to less burnout and less turnover, both of which can have real business impacts.

    The software development industry is growing like wildfire, bringing in waves of entry-level developers each year. Interestingly, the average experience level is decreasing, increasing the need for more abstracted tools and toolkits, said Spang. In this environment, developer experience undoubtedly shines as a competitive advantage.

    So, if you’re creating developer-facing tools, consider the experience others will have when using them. Increasing developer experience may just be a defining factor in the project’s success and the long-term happiness of your developer community.

  • Open Standards Are Key For Realizing Observability

    Open Standards Are Key For Realizing Observability

    Observability has quickly become a major focus of enterprise DevOps. Yet tool sprawl and complexity can hold some observability initiatives back. The 2022 CNCF Microsurvey on observability trends found that 23% of organizations use between 10 and 15 tools for monitoring, metrics and gathering logging and tracing data. A full 50% also noted that engineers and teams using multiple tools presented a top challenge to observability.

    I recently met with Dotan Horovits, principal developer advocate for Logz.io, to get his take on improving how modern development teams handle observability. According to Horovits, open standards and open source projects are the principal drivers to realizing interoperable observability industry-wide. Below, we’ll consider the issues facing observability and introduce OpenTelemetry, a CNCF project quickly gaining traction to unify telemetry data from various sources.

    Open Standards For The Win

    Although proprietary monitoring agents have a lot of intelligence, they inherently use tight coupling between telemetry generation and the backend storage analytics, said Horovits. This can cause vendor lock-in and lead to formatting variations in telemetry data between toolsets. “The open standards have the power to converge our industry to prevent vendor lock-in, even the playing field, and allow us to get any telemetry for any source,” said Horovits.

    Without an open standard, supporting the evolving suite of technologies at the heart of modern polyglot software stacks is challenging. Organizations constantly add to their tech stack, from SQL to NoSQL, Kafka, API gateways and serverless cloud services, and these modules produce varying data formatted in unique styles. What’s more, companies are typically using more than one cloud vendor, further necessitating the need for a decoupled method to represent multi-cloud metrics. “As soon as your IT stack becomes this complex, there is a need to aggregate all telemetry into one unified observability standard,” explained Horovits.

    Introducing OpenTelemetry

    OpenTelemetry is described as a “high-quality, ubiquitous and portable telemetry to enable effective observability.” Using OpenTelemetry (sometimes abbreviated as OTel), engineers can generate, collect and export metrics, logs and traces. OpenTelemetry works with many popular libraries and is completely vendor-neutral and supported by a rising number of tools in the space. OpenTelemetry was accepted to CNCF on May 17, 2019, and is an incubating project.

    The OpenTelemetry registry boasts hundreds of libraries and plugins to export and transform telemetry data from many different environments, from legacy to contemporary sources. By decoupling the data collection and export process, OpenTelemetry is “one agent to rule them all,” Horovits explained. Of course, he added, vendors who adopt OpenTelemetry may still provide their own logic behind it.

    The Benefits of More Open Telemetry Data

    At the time of writing, live public repository data shows OpenTelemetry is the second-most active CNCF project, behind Kubernetes. So, why is the observability community so excited about OpenTelemetry? Well, for one, the interoperability it affords could equate to efficiency and time savings. “OpenTelemetry gives you the freedom to integrate throughout the stack,” stated Horovits. Part of this is because its open source roots make it a force multiplier. “You can never achieve such vast coverage with closed source as you would with a wide-open community,” Horovits said.

    Arguably, an open telemetry toolset has also become necessary due to the sheer complexity of cloud-native architectures. Most organizations are shifting to a distributed microservices architecture supported by a dynamic, elastic container-based infrastructure. In this world, a single service might span multiple clusters. Therefore, engineers might need to consider clusters, pods, nodes and namespaces when filtering for data.

    In this complex reality, there are many dimensions, multiplying what needs to be monitored. And in general, traditional monitoring systems become ineffective at this level. Tool sprawl and lack of consolidation are problems that arise when trying to cater to many environments simultaneously. Thus, a unified, decoupled observability platform makes a lot of sense.

    Future of OpenTelemetry

    We’ve seen vast adoption from cloud vendors and solution providers around OpenTelemetry. And, the project has been moving swiftly—OpenTelemetry has had a distributed tracing specification released for about a year. The project almost has a release candidate for metrics and one for logs is in the works, said Horovits.

    Beyond that, one feature that Horovits is excited about is continuous profiling, which could help users continuously analyze a function and track usage trends across instances. Another exciting potential development is eBPF. Since it’s language-independent and runs at the kernel level, eBPF could extract observability signals without requiring application-specific hooks or integrations. This could increase support for more environments.

    For more information, you can visit the OpenTelemetry site to peruse the documentation or read up on the feature roadmap for OpenTelemetry. Horovits also gave this helpful presentation on OpenTelemetry at KubeCon 2022.

  • Leverage Empirical Data to Avoid DevOps Burnout

    Leverage Empirical Data to Avoid DevOps Burnout

    Burnout is possible in any industry. But it seems to be especially rampant in tech, a space bursting at the seams with inflated expectations and ever-accelerating trends. Nearly 60% of tech workers said they were currently feeling burned out at their workplace, found a study by TeamBlind.

    Limiting burnout and supporting employee mental health is essential, not only for the workers’ sake but for the business as well. During this period, which commentators call the Great Resignation, if employees don’t feel respected, they will likely seek new opportunities elsewhere. Ensuring the needs of engineers are met is a vital target amid an ongoing cloud skills drought, a talent gap to which the DevOps space is certainly not immune.

    I recently met with Eric Mader, senior solutions architect, AHEAD, to understand the root causes of burnout as it relates to DevOps. According to Mader, burnout can be caused by many factors, including work-life imbalances and conflicting egos in the workplace. But above else, burnout is most often caused by management setting unrealistic targets that aren’t based on empirical team performance data.

    What Factors Contribute to DevOps Burnout?

    So, what can cause burnout in DevOps? First, the workplace culture can significantly contribute to burnout. In most remote working models, explained Mader, it isn’t easy to ascertain the correct work-life balance. For example, there may be an unwritten expectation to immediately respond to Slack messages or quickly get back to emails after-hours or over the weekends.

    An “always on” attitude can lead to unhealthy work habits. And while this isn’t necessarily unique to DevOps, it is sometimes unavoidable due to the high connectivity required to perform the job. Another potential contributor to burnout is the team dynamic, noted Mader. “Every team seems to be unique in its own way,” he said.

    Although Agile and DevOps involve very technical work, at times, ego can get in the way of a healthy team dynamic. For example, in many tech teams, there is often a “guru,” such as a senior developer or tech lead, that everyone goes to for advice, Mader said. This person might end up doing most of the work (both technical and interpersonal) and usually feels the most burned out. Or, other wanna-be leaders might take on an unnecessary command-and-control role.

    Personality differences with coworkers could contribute to burnout, but other factors like dealing with toil or managing legacy technical debt can be just as tiresome.

    ‘Move Fast And Break Things’

    According to Mader, perhaps the biggest contributor to burnout is the insistence from executives that tech workers consistently outperform their previous output—especially when past team performance data does not support these goals. Setting achievable goals is great, but when deliverables are out of scope, it’s a recipe for burnout.

    Part of the pressure to increase velocity is the constant illusion that “faster and more is better,” said Mader. Value is often attributed to the number of items achieved in a sprint instead of the quality of each point. For example, a team might deliver 200 story points of work in a sprint, but only 30 of these points carry a significant amount of value. Or, a leader might see a team accomplish 60 points of work in a sprint and then encourage 200 in the next one.

    These sorts of high expectations suffer from basic capacity problems. And the added stress quickly realizes itself as a burden, contributing to burnout, explained Mader. He encourages technical leads not to force more work but instead, “focus on the highest-value work based on the empirical data you have.”

    Five Tips To Avoid Burnout

    Mader shared some tips on how to avoid burnout in DevOps teams. Here are some simple yet effective ideas to maintain happier engineers and a more productive, healthier team relationship.

    Snuff out the always-on attitude. Organizations need to walk the talk when it comes to mental health awareness. Therefore, ensure employees know that their life boundaries are respected. Encourage this by honoring time off for holidays, paternity/maternity leave or personal vacation. Discourage emailing on the weekends and encourage turning on ‘Do Not Disturb’ alerts on team chats and work email accounts.

    Prioritize what gets automated. There’s a misnomer that any automation is good automation. In fact, over-prescription of software development automation can lead to some negative consequences. “I’ve seen many DevOps teams that focus so much time on automating low-impact business processes,” said Mader. Instead, prioritize automation projects that affect more people and those that would be continually used.

    Try out the ‘Fist of Five.’ Ensuring everyone’s voice is heard helps avoid a toxic working environment. One tactic to quickly gauge reaction in team meetings is to use what Mader calls The Fist of Five: One if an idea is totally off base and five if an idea makes total sense. This simple practice could ensure everyone gets to contribute input on new projects.

    Take the quietest person in the room aside. Only the squeaky wheel gets the oil, Mader noted. With that in mind, take aside the quietest person post-meetings to gather their input—you might be surprised. Junior folks who may be less apt to speak their minds typically have a less-institutionalized mindset and could suggest more innovative ideas.

    Use empirical data to inform future goalposts. Finally, Mader encouraged DevOps teams to “use empirical evidence to back up on where you’re going in the future.” Specifically, in the DevOps sphere, empirical data takes the form of metrics like mean-time-to-recovery, or SLAs, SLIs and SLOs.

    Remember that even if a team did accomplish a certain number of goals in a sprint, it doesn’t mean they can produce the exact same results the next time around—they could have been pushed to their limit to produce past results. Thus, managers should aim to move the needle on metrics progressively, not in giant strides that upset morale.

    Take an Opinionated Approach to DevOps

    Nowadays, DevOps teams are spoiled for choice—there are countless tools on the market and an endless stream of best practices and methods. Even within a single company, the DevOps toolset may vary considerably from team to team. To help make sense of it all, Mader encouraged DevOps leads to simplify the transformation journey by setting guardrails and adopting a more opinionated approach around what tools and processes you take on.