devopsy.kinsta.cloud

Category: IT as Code

  • Survey Surfaces Multi-Cloud Computing and Cost Challenges

    Survey Surfaces Multi-Cloud Computing and Cost Challenges

    A survey of 360 CIOs and IT leaders in the U.S. and the United Kingdom found that, as the number of cloud platforms an organization employs expands, the number of tools they are required to deploy and master does, too.

    The survey, published this week by Virtana, a provider of a platform for managing cloud migrations, found nearly two-thirds of respondents (63%) reported their organization relied on at least five separate tools for migration, cloud cost optimization, integrated performance monitoring, application performance management and cloud infrastructure monitoring. A full 83% are expending some level of manual effort to consolidate data from all of these tools, or are simply using them in isolation from one another. Only 17% said data integration is fully automated.

    Nearly three-quarters (73%) also noted that siloed efforts limit their ability to realize the full potential of the cloud.

    It appears that multi-cloud computing only makes managing IT more challenging. Not to mention the increased pressure a multi-cloud approach puts on DevOps and developers that already juggle an overwhelming number of tools for development, DevSecOps and more. And, of course, the risk of cloud misconfiguration grows alongside the number of cloud platforms an organization uses.

    According to the Virtana survey, a total of 82% of respondents have a multi-cloud strategy, with more than three-quarters (78%) deploying workloads in more than three public clouds. A full 59% now run more than half of their workloads on public clouds. More than half (51%) also plan to increase the number of public cloud instances they support this year, with more than a third (34%) planning on using five or more cloud platforms.

    More challenging still, more than three quarters (76%) are in the early stages of implementing serverless computing frameworks. Overall, more than half (52%) will continue to maintain a hybrid environment as part of their core IT strategy. A full 96% of respondents see value in workload portability, but only 71% are in the early stages of achieving that goal. Top benefits anticipated include maximized cost savings (58%), reduced cloud service provider risk (46%) and increased business agility (43%). However, nearly three-quarters (75%) have either no or few cloud governance capabilities (34%) or have stitched together disparate tools to achieve that goal (41%).

    Jonathan Cyr, vice president of product management for Virtana, said that as the global economy becomes more challenging, it’s only a matter of time before sensitivity to cloud costs start to rise. Each time the overall IT environment expands, increases in complexity drive the total cost of IT higher.

    It’s not clear to what degree that sensitivity to cloud cost might drive more IT organizations to centralize the management of IT environments. In theory, a single control plane could be employed to manage multiple cloud computing environments, but today most clouds are still managed in isolation from one another.

    IT teams also find themselves inadvertently locked into a cloud platform. The more developers invoke proprietary application programming interfaces (API), the more difficult it becomes to migrate a workload from one cloud platform to another.

    It’s all but certain there will be a cloud management reckoning in the months ahead. Development teams may prefer one cloud platform over another for many reasons but, as the cost of IT continues to increase, there will soon be tougher cloud cost questions emanating from the finance office.

  • What SREs Can Learn From the Atlassian Outage of 2022

    What SREs Can Learn From the Atlassian Outage of 2022

    What happens when the tools and services you depend on to drive site reliability engineering turns out to be susceptible to reliability failures of their own? That’s the question teams at about 400 businesses presumably asked themselves in the wake of a major outage in Atlassian Cloud. The incident offers a number of insights for SREs about reliability risks within reliability management software itself—as well as how to work through complex outages efficiently and transparently.

    What Caused the Atlassian Cloud Outage?

    The outage, which began on April 4 and finally resolved April 18, affected about 400 Atlassian Cloud customer accounts. Atlassian Cloud is a hosted suite of popular Atlassian products including Jira and OpsGenie. The outage meant that affected customers could no longer access these tools or the data they managed in them.

    According to Atlassian, the problem was triggered by a faulty application migration process. Engineers wrote a script to deactivate an obsolete version of an application. However, due to what Atlassian called a “communication gap” between teams, the script was written in such a way that it deactivated all Atlassian Cloud products, not just the obsolete application.

    To make matters worse, the script was apparently configured to delete data permanently, rather than mark it for deletion, which was the intention. As a result, data in affected accounts was removed permanently from production environments.

    The Good, the Bad and the Ugly

    The Atlassian Cloud outage may not be the very worst type of incident imaginable—failures like Facebook’s 2021 outage were arguably worse because they affected more people and because service restoration was complicated by physical access issues—but it was still pretty bad. Production data was permanently deleted and hundreds of enterprise customers experienced total service disruptions that have lasted several days and counting.

    Given the seriousness of the incident, it’s tempting to point fingers at Atlassian engineers for letting an incident like this happen in the first place. They seem to have written a script with some serious issues, then presumably deployed it without testing it first—which is exactly the opposite of what you might call an SRE best practice.

    On the other hand, Atlassian deserves credit for responding to the incident efficiently and transparently. Although the company was silent at first, it ultimately shared details about what happened and why even though those details were a bit embarrassing to its engineers.

    Crucially, Atlassian also had backups and failover environments in place, which it used to speed the recovery process. The major reason why the outage lasted so long, the company said, is that restoring data from backups to production requires integrating backup data for individual customers into storage that is shared by multiple customers, a tedious process that Atlassian apparently can’t perform automatically (or doesn’t want to, presumably because it would be too risky to automate).

    Unfortunately for impacted customers, it does not appear that any fallback tools or services were made available while they waited for Atlassian to restore operations. We imagine this poses more than minor issues for teams that rely on tools like Jira to manage projects and OpsGenie to handle incidents. Perhaps those teams stood up alternative tools in the meantime – or perhaps they crossed their fingers, hoping their project and reliability management tools would come back online ASAP. 

    Takeaways for SREs From the Atlassian Outage

    For SREs, then, the key takeaways from this incident would seem to be:

    • Always perform dry runs of migration processes in testing environments before putting them into production. Presumably, if Atlassian engineers had tested their application migration script first, they would have noticed its flaws before it took out live customer environments.
    • Back up, back up, back up—and make sure you have failover environments where you can rebuild failed services based on backups. While this outage is bad, it would be 100 times worse if Atlassian couldn’t restore service based on backups and data had been lost permanently.
    • Ideally, each customer’s data should be stored separately. As we noted above, the fact that Atlassian used shared storage seems to have been a factor in delaying recovery. That said, it’s hard to fault Atlassian too much on this point; it’s not always practical to isolate data for each user due to the cost and administrative complexity of doing so.
    • SRE teams would do well to think about how they’ll respond if their reliability management software itself goes offline. For example, it might be worth extracting and backing up data from your reliability management tools so you can still access it if your tool provider experiences an incident like this.
    • Over-communicate with your customers often and early. In this case, there was initial radio silence that left customers in the dark. Most of this chatter eventually took place in public forums.

    Conclusion

    The Atlassian Cloud outage is notable both for its length and for the fact that, ironically, it took out software that teams use to help prevent these types of issues from happening at their own businesses.

    The good news is that Atlassian had the necessary resources in place to restore service as quickly as possible. A shared data storage architecture led to slow recovery, which is unfortunate; again, it’s hard to blame Atlassian too much for not setting up dedicated storage for each customer.

  • New Relic Expands Scope of Observability Reach

    New Relic Expands Scope of Observability Reach

    At its Futurestack conference, New Relic announced it expanded the integrations and tools it provides for its observability platform and added its first cybersecurity tool.

    The company now provides more than 470 integrations with cloud services, open source tools and other enterprise technologies, with support for offerings from Akamai, Atlassian, CircleCI, Cloudflare, Netlify, PagerDuty and Postman now being added to its Instant Observability platform. The company has also added the ability to now collect logs alongside metrics and traces within its Application Performance Monitoring (APM) offering.

    Finally, New Relic has deepened its cloud ties with Microsoft, added a guided user onboarding interface, support for a wider range of instrumentation methods, tighter integration with Kubernetes and, for the first time, is branching out into the realm of cybersecurity via a New Relic Vulnerability Management service that is being added to its software-as-a-service (SaaS) portfolio.

    Ishan Mukherjee, group vice president for product go-to-market at New Relic, said New Relic Vulnerability Management is the start of a larger effort to enable IT teams to leverage the observability data they already collect via the New Relic platform to drive the adoption of DevSecOps best practices. Legacy approaches to vulnerability management create yet another silo of data that only further aggravates cybersecurity blind spots, he noted. New Relic Vulnerability Management aggregates data collected via its agent software and integrations with a wide range of platforms to provide a unified view of the overall security posture of an IT environment, Mukherjee added.

    It’s still early days as far as adoption of observability platforms is concerned, but it’s apparent that platforms that unify the collection of metrics, logs and traces across both applications and the IT infrastructure they run on are transforming how IT is managed. Most DevOps teams today are able to continuously monitor IT environments using tools that track a set of pre-defined metrics; observability platforms make it possible to aggregate data so that DevOps teams can launch queries that help them uncover the root cause of an IT issue. That capability is increasingly becoming critical as IT environments—thanks, in part, to the rise of microservices and cloud computing environments—become too complex for IT teams to manage using legacy tools.

    The challenge is that most IT teams are not quite sure what questions they should ask to get the most value from investment in an observability platform. Observability platforms that accurately surface issues without requiring much intervention from IT teams are likely to gain the most traction, as the competition among providers of these platforms intensifies.

    In the meantime, organizations might want to start reevaluating roles within their IT teams as it becomes simpler to collaborate around a single source of data. The days when IT teams had to triage issues using disparate tools that surfaced conflicting data may finally be coming to an end. As such, it should become possible for a team of IT professionals to manage a wider range of processes across an extended enterprise IT environment.

    Naturally, it may be a while before observability platforms drive wholesale reorganizations of IT teams, but it’s now more a question of when rather than if that day will come.

  • Is Your Future in SaaS? Yes, Except …

    Is Your Future in SaaS? Yes, Except …

    The growth of SaaS—including IaaS—was phenomenal when it came and it has become somewhat pervasive. Today, there are SaaS tools in use at just about every business, large or small. From sales to marketing to customer management, there is a SaaS tool that does it affordably and well. For now. The same is true of cloud vendors. Cloud prices are kept low by two mechanisms—volume and profit. Volume; much like the theory that mass transit is cheaper, by putting many customers on a single server reduces cloud vendors’ operating expenses and thus reduces what the cloud vendor must charge to make a profit.

    But as costs rise and growth slows in the current environment, it is important that your IT department, as a customer, pays attention to the cost/benefit equation of all SaaS offerings. Some will remain beneficial because of the work they offload, even as prices increase for your organization. Others, you’ll see, will quickly outgrow the value they bring as costs rise and there is a third class that just isn’t worth it—maybe it never was and you haven’t bothered to pay attention.

    Look for those items. So that this is less vague, here’s an example: A company I am involved with has been paying for a SaaS service that simply emulates an on-site functionality. The organization could (and previously did) drop a couple of boxes in and achieve largely the same result. Recently, we looked more closely at items like this and found that, in this particular case, (though not in most SaaS cases the org is using) the cost of dropping two boxes in and letting them do the job was cheaper (assuming those boxes were upgraded/replaced every other year) than continuing with the SaaS solution at the current monthly price. So much cheaper, in fact, that a contractor could be hired to deal with any issues that came up with the physical boxes and the only increased work for IT would be maintenance updates to keep the system secure. If, instead of appliances, the solution consisted of two Windows or Linux boxes, even that maintenance cost would be minimized.

    Service will be the same or better—the devices are right there if a change is needed—and there will be one less link in the chain of tools/vendors for attackers (the physical solutions would be on a private subnet, SaaS cannot be). Assuming the solution is a quality product, there really is no downside here. And yet, for several years this company has used the SaaS solution. These are the low-hanging fruit that people will point at and call “retrenchment” or whatever. Ignore them. Do what’s best for your org—be it SaaS or local—and make that decision on a per-tool/per-project basis.

    And keep kicking it. You’re building apps and infrastructure that keep the company running. Don’t hesitate to take pride in what your team(s) have built and don’t hesitate to cut costs on things that don’t matter so you have more funds for things that do. And once again, thank you—from all the employees in your organization that don’t realize they should say thanks.

  • Increasing Use of SLOs to Enable Observability

    Increasing Use of SLOs to Enable Observability

    Observability is a growing discipline among most IT and operations departments. To release stable software faster, operators need continuous visibility into metrics like performance, uptime and availability. As a result, engineers are increasing their use of service-level objectives (SLOs) across the board—a recent study found that 82% of companies are increasing their use of SLOs.

    SLOs enable deep visibility into specific applications’ performance and are often used by site reliability engineers (SREs) to ensure quality and avoid service interruptions. Not only that, but various teams are correlating SLOs to aspects of the business to reduce cost and help direct decision-making. But while many environments have visibility, there are still gaps present.

    Nobl9 recently released a global survey of IT professionals and executives that tracked the state of service-level objectives in 2022. Below, we’ll look at the key findings from the report and consider what they mean for the state of SLOs and observability in general.

    State of Observability and SRE

    In general, site reliability engineering is still maturing across organizations. Though only 31% of companies have adopted SRE, it’s slated for much future growth, as 46% said they plan to embrace SRE in the future.

    These operators are now faced with many cloud-native observability tools, which are producing a sea of data in the form of metrics, logs and traces. A full 39% of companies use anywhere from six to 10 observability and monitoring tools and 35% use more than 11.

    With so many tools now deployed, who is using this observability and monitoring data? Namely, at 74% of companies, observability data support operational requirements. Operations teams are the ones most likely to use SLOs to monitor uptime, performance and overall efficiency. After operations teams, security teams also use this observability data (71%), which makes sense as SLOs can inform incident response. Other areas that follow are customer support, compliance and capacity planning.

    One interesting finding is that only 42% of companies use SLOs for service-level agreement (SLA) adherence. This indicates that SLOs are most often applied to internal optimizations and decision-making.

    Hybrid Environments Complicate Visibility

    Companies are mostly tracking SLOs to increase their visibility into networks (83%), databases (76%) and applications (75%). Other top areas include private cloud environments and legacy computing arrangements. But although observability is trending, organizations still lack complete visibility across the entire stack.

    Of note is that nearly half (46%) of respondents said their monitoring and observability tools don’t provide full visibility into all of their company’s IT assets. For example, only 45% of companies have visibility into their containers and just 35% have visibility into their microservices architecture.

    A lack of full-stack visibility may be complicated due to rising hybrid and multi-cloud conditions, as 78% say a hybrid cloud environment makes monitoring infrastructure more difficult.

    Benefits of Tracking SLOs

    The report defined SLO as “a performance and availability target set for a given system, application [or] service over time.” And the benefits are readily apparent for organizations following such targets—they can help increase performances, direct decision-making and help avoid outages.

    As a result, 70% of companies are currently using SLOs in some fashion. Here are some benefits of tracking SLOs:

    • Increase microservice performance: 87% said using SLOs for microservices architecture would improve service performance.
    • Enable full-stack observability: 58% said some of their company SLOs are mapped to business operations.
    • Improve business decision-making: 91% agreed using SLOs can help drive better business decisions.
    • Prevent service interruption: 67% said their company prevented business interruptions thanks to SLO thresholds alerts.
    • Reduce expenses: 90% indicated SLOs saved their company money.

    Final Thoughts

    More teams are tracking service-level objectives than ever before. And most companies (71%) that aren’t using SLOs now plan on adopting them soon. The data demonstrated that the observability market is an evolving area with room to grow. And, the same can be said about the SRE role.

    Yet, it’s good to point out that not all companies will adopt precisely the same roles or monitoring procedures. Therefore, there will likely continue to be variance in who is using SLOs and how they are applied. Regardless, the study suggested that investigating SLOs has the potential to benefit the software life cycle in many ways.

    The study, “Adoption of SLOs Grow to Increase Visibility and Drive Business Improvements” surveyed 309 participants around the globe in various sectors. The study was sponsored by Nobl9 and conducted by Dimensional Research. For more information, you can download the report here.

  • 15 Ways Software Becomes a Cyberthreat

    15 Ways Software Becomes a Cyberthreat

    Software is an integral part of private and commercial life; there is no way around it. You need software to do your taxes, book a flight or browse the internet. Software has made our lives much easier in so many ways. However, as we become more reliant on software we also become more vulnerable to cyberattacks.

    This article will explore 15 different ways that software can end up becoming a cybersecurity threat. By understanding these risks, you can take steps to protect yourself and your business.

    Lack of Security Features

    Many popular software programs either lack basic security features or the features that are present lose functionality, leaving users vulnerable to attack. For example, Adobe Reader and Microsoft Word have both been found to have security vulnerabilities. If you use these programs, make sure you keep them up to date with the latest security patches.

    If you are worried about having to deal with downtime while installing new software updates, you can always schedule them for a time when you know you won’t be using your computer. This way, you can be sure that your software is always up-to-date and secure.

    Poor Password Security

    One of the most common ways users’ accounts get hacked is through the use of weak or easily guessed passwords. If you use a program that requires a password, make sure to use a strong one that would be difficult for someone to guess.

    Additionally, if you use the same password for multiple accounts, an attacker only needs to figure out that one password to gain access to all of your accounts. This is why it’s important to use different passwords for different services. If you can’t remember all of your different passwords, you can use a password manager to help you keep track of them.

    Consider using both a password manager—to take advantage of the more robust security these applications typically have—as well as a password randomizer, which will generate long, random passwords for you. Many password managers include a randomizer feature.

    Phishing Attacks

    Phishing attacks can impact any type of software program. In a phishing attack, an attacker will try to trick you into giving them your password or other sensitive information by masquerading as a legitimate website or program. Be very careful about any emails or messages you receive that ask for personal information, and never click on links in these messages unless you’re absolutely sure they’re legitimate.

    Phishing that takes place through software programs can also happen through messaging programs, like Skype or Facebook Messenger, or through email programs, like Gmail. Be sure to be cautious about any links or attachments you receive through these programs, as well.

    Keyloggers

    Keyloggers are malicious programs that are installed on your computer without your knowledge. Once they’re installed, they can track every keystroke you make, which means they can easily steal passwords and other sensitive information. Keyloggers can be installed through email attachments, malicious websites or even infected USB drives.

    To protect yourself from keyloggers, never install software from untrustworthy sources. Be very careful about what emails you open and what websites you visit. If you do get a keylogger on your computer, make sure to run a malware scan as soon as possible to remove it.

    Drive-by Downloads

    Drive-by downloads are another type of malicious software that can be installed on your computer without your knowledge. These types of programs are usually downloaded when you visit a malicious website or click on a malicious link. Once they’re installed, they can do things like steal your passwords or track your web browsing.

    To protect yourself from drive-by downloads, be careful about what websites you visit and what links you click on. If you think you may have downloaded a malicious program, run a malware scan as soon as possible.

    Malware

    Malware is malicious software that can infect your computer without you even knowing about it and, once it’s there, it can do things like steal your passwords or data or give an attacker remote access to your machine. From there, attackers can potentially access your organization’s IT infrastructure. Be very careful about what email attachments you open and what websites you visit, as these are two of the most common ways malware can end up on your computer.

    The software you use can also be used to introduce ransomware into your system. A malicious actor could create a fake version of a popular software program and distribute it online. When users download and install the fake software, they unknowingly grant the attacker access to their system. Or, an attacker could exploit vulnerabilities in software programs to gain access to a user’s system. Once inside, the attacker could install ransomware and encrypt the user’s files.

    Unsecured Wi-Fi Networks

    If you use public Wi-Fi networks to connect to the internet, you may be putting yourself and your organization at risk. These networks are often unsecured, which means that anyone else on the network can snoop on your traffic and see what you’re doing. If you need to use public Wi-Fi, make sure to connect to a secure VPN first so that your traffic is encrypted and private.

    Social Engineering

    Social engineering is a type of attack where an attacker tries to trick you into doing something that will give them access to your account or data. For example, they may pretend to be a customer support agent for a program you use and ask you for your password so they can “fix” an issue with your account.

    Or, they may send you an email that looks like it’s from a trusted source but actually contains a malicious link. Be very careful about any communications you receive, even if they appear to be from a legitimate source.

    Unsafe Browser Extensions

    If you use a web browser like Google Chrome or Mozilla Firefox, you may have installed some extensions to add additional features or functionality. But some of these extensions can actually introduce security risks. For example, an extension might have access to all of the websites you visit and the data you enter into them. Be very selective about which extensions you install, and only install ones from trusted sources.

    WordPress Plugins

    If you have a WordPress website, be careful about which plugins you install. Some plugins can introduce security risks, for example, by giving attackers access to your website or database. Install only trusted plugins, and make sure to keep them up-to-date. Also, while browsing new plugins, it is important that you ensure you are downloading and installing plugins that are regularly updated. Take a look at the last time the software was updated before deciding to install it.

    Adware

    Adware is a type of software that displays advertising on your computer, often in the form of pop-up ads. While not all adware is malicious, some forms of it can track your online activities and even collect sensitive information like your passwords or credit card numbers. Be careful about what programs you install, and always read the EULA before agreeing to anything.

    Google Docs

    Google Docs is a popular cloud-based word processing application that is part of Google Workspace. While it’s generally safe to use, there have been some reports of malicious actors using it to spread malware or launch phishing attacks. If you use Google Docs, be sure to only open documents from trusted sources and never click on any links in a document unless you’re absolutely sure they’re safe.

    Third-Party App Stores

    If you use an Android device, you may be tempted to download apps from a third-party app store instead of the official Google Play store. While there are some legitimate app stores out there, many of them are full of malware and other malicious programs. It’s always best to stick to the official app store for your device to avoid these risks.

    Outdated Software

    One of the most important things you can do to keep your computer safe is to make sure all of your software is updated to the latest version. Software developers regularly release updates that patch security vulnerabilities, so it’s important to install these updates as soon as they’re available. You can typically set your software to update automatically or you can check for updates manually on a regular basis.

    Commandeering For Loops

    Commandeering for loops is a type of attack where an attacker takes control of your computer by sending malicious commands through the command-line interface. This can happen if you accidentally download and run a malicious program or if you visit a website that has been compromised by an attacker. To protect yourself from this type of attack, be careful about the programs you download and run and only visit websites that you trust.

    Conclusion

    Software, in its myriad forms, is crucial to running both our businesses and enhancing our personal lives. There is almost no way to avoid using several, if not dozens, of different programs on a daily basis. But as we’ve seen, even the most innocuous-seeming software can pose a serious security risk if it’s not used properly. But understanding the attack vectors ahead of time can help protect yourself and your organization from these risks and keep your data safe.

  • Nobl9 Shares SLO-as-Code Methodology

    Nobl9 Shares SLO-as-Code Methodology

    Nobl9 has released version 1.0 of an open specification for defining service level objectives, dubbed OpenSLO, and, in addition, has defined a repeatable SLO methodology.

    Kit Merker, Nobl9 COO, said the Service Level Objective Development Lifecycle (SLODC) methodology represents an effort to create a set of best practices for achieving and maintaining SLOs that are implemented as code within an application environment.

    A survey of more than 300 IT managers and executives conducted by Dimensional Research on behalf of Nobl9 found only 29% of respondents had no plans to implement SLOs. A full 94% of respondents that have or plan to implement SLOs intend to map them directly to business operations, with 91% reporting they expect that effort to improve decision-making. More than 80% also said their organizations are planning to increase the use of SLOs, with 87% indicating SLOs should improve overall microservices performance.

    The challenge is that most organizations have limited visibility into their IT environments. The survey, for example, found less than half of respondents (46%) had visibility into all their IT environments. Only 45% and 35% claim to have visibility into containers and microservices, respectively. More than three-quarters (78%) said hybrid clouds make observability more difficult.

    Ironically, 45% of companies reported they already employed 11 or more observability and monitoring tools. On the plus side, 31% have hired site reliability engineers (SREs), while nearly half (46%) planned to create that role.

    Nobl9 is trying to spur greater adoption of SLOs by making available an open source SLO specification that defines a common interface for constructing SLOs across a Git-based workflow. Since OpenSLO was initially launched, more capabilities have been added including a DataSource object that makes it easier to reuse connection details and which makes creating SLOs less verbose and three alerts have been defined: AlertCondition, AlertPolicy and AlertNotifcationTarget.

    Nobl9 is also releasing an OpenSLO-to-Nobl9 converter to transition OpenSLO YAML files into Nobl9 YAML when required.

    SLOs, of course, are not a new idea. They have been employed as a metric to track the performance of IT services for decades. However, as more microservices-based applications are built and deployed, it’s becoming more challenging to maintain SLOs across applications that have many more dependencies than legacy monolithic applications. Ultimately, each IT team needs to provide some sort of objective benchmark that assesses their overall effectiveness at delivering application services. SLO-as-code is intended to make it simpler to gather the metrics that confirm whether service levels are being achieved. Those SLOs are then tracked and integrated across everything from financial operations to supply chains, noted Mercer.

    It’s not clear to whether reliance on SLOs has waned over the years or if application environments simply became too complex to track meaningful metrics. However, as applications are increasingly viewed as services, it’s now only a matter of time before SLOs become more widely adopted across a modern application environment. The challenge, of course, is that defining an SLO is a lot easier than maintaining it—especially within today’s highly dynamic application environments where services tend to come, change and eventually go unexpectedly.

  • Container Storage for DevOps: New Options With NVMe

    Container Storage for DevOps: New Options With NVMe

    The DevOps world is ripe for a storage paradigm change, both in how it operates within the enterprise and how it implements applications. 

    NVMe, especially NVMe-over-fabrics (NVMeoF), is both a tool and a host target that can provide such a change—especially in the new world of microservices and containerization. NVMe can drastically improve application performance over classic implementations and, given the right tools, transform the process of writing those implementations to be faster, more robust and more flexible. 

    I/O-Intensive Work

    A quick recap shows why this is so. The DevOps build toolchain, especially for large codebases, has always been heavily I/O-dependent and storage access has classically been the biggest bottleneck for building an image. The ability to cluster small, inexpensive Linux x86 servers connected to SANs twenty years ago was a first step in separating storage hardware from compute, but I/O latency was pretty awful. Build/link operations could actually take longer than the compilation of object files. 

    Virtualization helped make hardware utilization better and cheaper but didn’t really improve throughput. Cloud infrastructures such as OpenStack and CloudStack made those infrastructures simpler and more elastic to manage but, again, the IOPs bottleneck remained. 

    Containerization for DevOps made it even simpler from the applications level—just deploy the app and ramp up container instances as needed—no more dedicated infrastructure needed. And it’s cheaper for incremental functions. 

    However, not much was done to clear the I/O bottlenecks for largescale storage and increasing overall performance. 

    When the new nonvolatile memory technology of NVMe came to market in 2014 for the single server world, it delivered two game-changing capabilities. 

    1. An increase of 5x disk access I/O speed. This was, arguably, the biggest single performance improvement since multicore processors with onboard cache (think Opteron) emerged. With Linux adding support for standard, open source NVMe drivers in 2021, the options increased beyond Windows and Mac users.However, SANs and their protocol stack were still an issue. SATA- and SCSI-based SANS performance was gated to the 1.2GB limitations of those technologies and big iron to head-end the SAN still dictated high-cost infrastructure.
    2. Unlike SATA or iSCSI, NVMe has the native capability for TCP/IP or fibre channel addressing. This allows NVMeoF to be an autonomous point for data storage in a normal network. A user can now keep control of their data but use network-attached compute as a utility with roughly 6Gb speed for I/O—greatly lessening the big penalty for disk I/O.  

    Applying Containers

    The dramatic uptick in container deployments and the debut of container attached storage (CAS)—software that includes microservices-based storage controllers that are orchestrated by Kubernetes—represented another industry milestone. These storage controllers can run anywhere that Kubernetes can run, which means any cloud or even bare metal servers or on top of a traditional shared storage system. Critically, the data itself is also accessed via containers as opposed to being stored in an off-platform, shared scale-out storage system. 

    Since container-attached, cloud-native storage enables the use of small, disaggregated architecture without the need for large-scale SANs or storage controllers dovetails nicely with DevOps work. 

    Container-based applications can benefit enormously from NVMe-based storage. As experts have pointed out, since DevOps applications are all about scalability, the parallelism embedded in the NVMe spec fits nicely to support them. 

    While SANs are good for large-scale contiguous operations, they’re expensive chunks of capacity and not nimble to deploy. They’re also bound to the slower I/O storage protocols, which can negate performance gains possible by adding NVMe to that array (as opposed to discretely).

    Container-based storage with NVMe gives DevOps teams the opportunity to add storage at low cost with deterministic management of that stateful storage—in theory, of course. In practice, the temptation to add like infrastructure to current infrastructure is powerful, even with all its limitations.

    To make that theory a reality, a control plane with the fast data plane is necessary—and the market is beginning to address this need.  

    As one example, OpenEBS recently introduced an upgraded engine, MayaStor (now part of DataCore Software), which enables use of NVMeoF at impressive speed: Benchmarks with Intel SPDK architecture show a 5x read/write speed improvement over SATA.

    OpenEBS debuted a few years back as a containerized storage solution for Kubernetes-based applications. With an optimized storage engine, Kubernetes has the opportunity to perform like other, more specialized hosting and development solutions. 

    Impact for DevOps

    What does this mean for DevOps teams? Firstly, using NVMe in containerized DevOps applications speeds up the time required for large builds, thereby speeding iterations of writing software. It also removes the need for complex and expensive SANs to host build environments. This double whammy of benefits means that development is quicker and cheaper and, in turn, applications can deploy faster while using more flexible tools.

    Second, NVMe greatly mitigates I/O concerns for how application code is designed and how environments are implemented. No more being beholden to the storage trunk—NVMeoF drops the performance penalty for I/O ops and distributes traffic across a normal TCP/IP network. This is now a triple whammy—faster, cheaper and more flexible containers built atop a fast, flexible virtualized infrastructure.

    Backed by innovations in container-attached storage, DevOps teams can achieve the agility and flexibility that are paramount to its mission—and act as a proving ground for the rest of the IT team to consider alternative paths to greater efficiency and performance.

    Join us for KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain (and virtual) from May 16-20—the first in-person European event in three years!

  • Red Hat Adds Managed Ansible Service on Azure

    Red Hat Adds Managed Ansible Service on Azure

    Red Hat today announced general availability of a managed instance of the Red Hat Ansible Automation Platform on the Microsoft Azure cloud.

    Announced at the Red Hat Summit 2022 event, this offering makes it possible to deploy the control plane for Ansible on the Microsoft Azure cloud as an alternative to existing support for Amazon Web Services (AWS) or an on-premises IT environment.

    In addition, in version 2.2 of the Ansible Automation platform, Red Hat is previewing support for content signing technology that automatically validates content to improve overall security.

    Thomas Anderson, vice president of Ansible for Red Hat, said this managed service enables IT teams to take advantage of integrated Microsoft billing in addition to the latest features and content provided for the platform and other capabilities that have been added to optimize Microsoft Azure environments.

    The open source Ansible platform has been gaining traction as an automation framework that enables IT teams to declaratively automate tasks. It requires less programming expertise than rival automation frameworks, which Anderson said is the primary reason so many third-party vendors have adopted Ansible to automate a range of IT tasks.

    The challenge many organizations encounter today is managing multiple islands of automation across multiple clouds and on-premises IT environments. Ansible presents IT teams with the ability to begin unifying all those islands under a single framework, said Anderson.

    It’s not clear to what degree DevOps teams that have embraced rival automation frameworks might be willing to replace them with Ansible. However, as more IT teams embrace automation to make up for shortages of IT personnel, it’s apparent that Ansible is being used across a wide range of IT disciplines spanning everything from DevOps to network management.

    Regardless of the approach, the need to automate as many tasks as possible is becoming critical as IT environments grow increasingly more complex. Many of the scripts that IT teams created on their own simply don’t scale. More challenging still, they are not often documented, so every time the IT professional that created that script leaves the organization it becomes difficult to maintain that scripting code.

    In the longer term, of course, automation frameworks will be increasingly infused with machine learning algorithms capable of learning the unique nuances of individual IT environments. It’s not likely those algorithms will replace the need for IT professionals any time soon. However, rather than resisting automation, more IT professionals have concluded that the tasks required are simply too challenging to manually perform. Many will conclude they simply prefer to work for organizations that have a consistent approach to IT automation that makes it feasible for them to manage a wide range of IT tasks.

    One way or another, IT automation is becoming a standard capability that now goes well beyond the scope of a few isolated DevOps workflows. The issue is finding a way for IT professionals with varying skill levels to collaborate across a common framework that is both easily accessible and highly extensible.

  • Does GraphQL Introduce New Security Risks?

    Does GraphQL Introduce New Security Risks?

    The GraphQL query language is an excellent tool for increasing the ease of data sharing. The premise is that you request the fields you need in a single bundled request, avoiding multiple network calls. Due to its usability, GraphQL is a growing area of development. Yet, as with any new technology, it comes with caveats.

    REST APIs function as a request-and-response paradigm in which the receiver accesses one resource at a time. On the other hand, GraphQL exposes many fields at once, requiring additional rate-limiting and time-outs to avoid nefarious queries. Additionally, the ability to traverse the entire schema could be abused by attackers to expose intricate data linkages. There are also some usability features with GraphQL that could be abused if misconfigured.

    GraphQL is a friendly alternative to REST. But how do we handle the security implications of GraphQL? I recently chatted with Mike Benjamin, vice president of security research, Fastly, to dive deeper into the security repercussions of adopting GraphQL. Below, we’ll consider the unique insecurities associated with GraphQL and suggest specific steps to limit the impact of these risks.

    Four GraphQL Security Vulnerabilities

    1. Introspection

    GraphQL provides built-in documentation that tells developers what the fields are, providing a schema of everything you can query. You’ve essentially given the entire dataset schema to a developer, meaning they can perform introspection to capture attack surface information. Hackers will most commonly look for injection or mutation capabilities that they can exploit, explained Benjamin. Or, they might seek to leverage the interlinkages between data to map a way through to classified information.

    To mitigate this problem, Benjamin cautioned against over-exposing all documentation publicly. Instead, he suggested using a restrictive security model for external viewers. “Provide appropriate user documentation in a different method,” he says. This may end up looking more like the standard REST API view of documentation.

    2. Field Suggestions

    Another usability benefit in GraphQL could also be a potential security concern—field suggestions. Field suggestions occur when you don’t enter the correct field name or enter a partial input and the system returns a hint. You might type in 0 or user, and the system makes a suggestion to fill out the rest of the field.

    This user interaction improves the developer experience, but, unfortunately, it could be abused by a hacker, said Benjamin. Thus, he recommended disabling hints after initial development. Or, if you decide to keep it in, he recommended placing a limit on its use. For example, if you try this more than five times in a second, the system could flag it as a bot.

    Right now, this attack vector is pretty limited, but Benjamin anticipated that, over time, hackers could automate this introspection with mass scanners to divulge more information.

    3. Denial-of-Service-Oriented Attacks

    Without the proper guardrails, all web APIs are prone to denial-of-service (DoS) attacks. Such an attack may pummel a service with a million queries per second to shut it off completely. In the case of GraphQL, these queries can dive very deep in a nested data relationship. A social network, for example, might expose friends of users, friends of users’ friends and so on.

    Or, hackers might utilize GraphQL’s bulk query ability to perform reconnaissance, brute-forcing the enumeration of data to query hundreds of user IDs to see which are valid. If you’re allowing unlimited data retrieval bundled within a single request, you could be inviting hundreds of bundled queries that could expose large swaths of the underlying database in one fell swoop.

    To counter these actions, Benjamin recommended limiting the testing depth. This could involve placing limits on how long a query can run, which would cause a time-out for requests that require suspiciously high degrees of processing. Another option is to limit how deep the bundling can get, said Benjamin. For example, perhaps the system restricts going three levels deep into a graph database, as that could expose millions of interconnected endpoints.

    4. Lack of Object-Level Authorization

    Another vulnerability that GraphQL endpoints may be prone to is a lack of object-level authorization, which hackers could leverage to get at a datapoint through its edges, said Benjamin. Since the idea is that you can access any data from one all-encompassing endpoint, authorization rules are often too abstracted away from the actual data. Queries may be able to access nested data or query relationships between data, too, which may unintentionally divulge information.

    Benjamin recommended placing authorization closer to your objects or data to rectify this issue. Use authorization-level controls that align permissions structure and roles with users. “Everyone needs a common authorization model,” said Benjamin.

    Future Evolution: GraphQL Security Matures

    As I’ve written before, the greatest strength of GraphQL is also its greatest weakness. Using GraphQL with default settings is a security trade-off in favor of usability. But according to Benjamin, we’re seeing more GraphQL servers disabling some introspection features, which he sees as a very positive move and a sign of maturity of the base configuration.

    GraphQL might suffer from similar threats that all APIs are vulnerable to, such as XSS, injection, DDoS attacks and broken authorization. However, Benjamin stressed that the added complexity of GraphQL makes it a unique case. “The most important thing for folks to consider is that GraphQL is more complicated than a REST endpoint—much more complicated,” said Benjamin. “GraphQL is unique in that it has linkages of data which contributes to complexity. It takes a little more time to think about how that’s confirmed, how it’s presented and how it would be attacked.”

    In other words, with great power comes great responsibility.

    Although exploits in public GraphQL APIs are still rare, we should stay vigilant. Benjamin encouraged developers to continue reading about, understanding and thinking about these attack vectors when designing their interfaces, as these risks will become more prominent as GraphQL usage grows and hackers become more familiar with these specific frailties.