devopsy.kinsta.cloud

Category: DevOps and Open Technologies

Editorial Calendar for February 2016

  • Some Change is Bad

    Some Change is Bad

    Some time toward the end of the year, I start considering the volume of change we, as practitioners, have been under for … literally years, even decades. And I start to try and think of ways to keep you all from drowning in change. And then I write something that I hope helps you all focus on good change, and keeps moving your environment forward.

    This year I’m going to say “consider the reasons for the change.” I will attempt to find examples that do not require mentioning a specific product or vendor, so apologies if these read a bit convoluted as I attempt to illustrate this point without castigating.

    There are three basic reasons that a market or marketers want you to consider a change. There are, of course, variants and cases where it is a bit of this reason and a bit of that, but most demands that we change what/how we are doing things can be lumped into “mostly one of these”. They are:

    1. A great idea that is really a game-changer for some subset of the market.
    2. A shifting of who is in control.
    3. Locking customers in or expanding revenue.

    All of these happen at the vendor/project level, and all of them happen at the market level.

    Number one is best illustrated by any of the waves of server virtualization that have come through. From virtual machines to containers, each has solved problems and changed the way we do business. Notably, I’m only talking about server virtualization efforts here. Most of the other virtualization attempts have really not fit the “great idea that is a game-changer” category, though some have limited usefulness. Personally, I think the new version of storage virtualization will be in the game-changer category but am waiting to see it in use.

    Number two is one to watch out for and is most obvious when big-name OSS projects are forked. It is rarely about the technology being implemented (after all, a fork is a copy of the original; at first it is not at all about the technology), but rather about who is calling the shots. There is a bit of this in every public cloud vendor implementing their own version of container management (though this example is more about lock-in, in my opinion).

    Number three is multi-pronged. If your vendor went from selling you hardware or software to selling you a service, they want that recurring revenue and they want you to be linked into that service so deep that you can’t walk away. The much-reviled 1990s mantra of “embrace and extend” is the poster child for this kind of change. There might be a value-add in there somewhere and sometimes the added functionality makes the move a no-brainer. Often, though, the change is simply to the vendor’s advantage, not their customers. As most major Linux distros have shown, open source is not immune to this one, as most of them are making OSS users the de facto beta testers for their paid offerings.

    None of this is to say that the change in front of you is bad—even if the reason it was originally spawned is bad. It is reminding you that, after a constant stream of change, we’re more open to it than maybe we should be and you should remember the only important thing about new changes—do they make our environment more secure, faster, more agile? Everything else is just noise. Yes, all the cool kids are switching to coding in [insert flavor du jour], but does it offer your organization anything new? Yes, it seems everyone is using [insert current CI/CD flavor], but does it address the weaknesses of your current toolchain?

    You all are nailing it. Keep doing it. And don’t be afraid of change, but make sure you know if that new OSS database system is really new, or just a lot of installing the same code under a slightly different name. Because someone didn’t like the new owners.

  • WhiteSource Adds SBOM Tool That Lists Vulnerabilities

    WhiteSource Adds SBOM Tool That Lists Vulnerabilities

    WhiteSource has added a software bill of materials (SBOM) tool to its portfolio that, in addition to capturing the components of an application, also surfaces vulnerabilities that should be addressed.

    Many organizations are becoming more rigorous about making sure SBOM are attached to every software development initiative in the wake of an executive order issued by the Biden administration that requires federal agencies to employ them. While not every organization builds applications for federal agencies, the fact that the federal government is requiring them in the wake of a series of high-profile breaches has led other organizations to review their software supply chain processes.

    An SBOM facilitates those reviews by making available a formal, machine-readable inventory of software components and dependencies that can be used to track their supply chain relationships and identify dependencies and hierarchical relationships.

    Ori Bach, executive vice president for product at WhiteSource, said WhiteSource SBOM takes the concept a step further by also identifying open source libraries. It will also automatically update the SBOM when changes are made to make it easier to identify unintentional or malicious content being installed during application builds.

    Finally, WhiteSource SBOM also provides remediation guidance that ensures updates won’t break the build.

    Bach said it’s already apparent cybercriminals have significantly increased their effort to compromise supply chains as part of an effort to compromise as many downstream applications as possible. In many cases, cybercriminals are now specifically targeting open source projects that create software that tends to be employed across a wide range of use cases.

    An SBOM in and of itself doesn’t make applications more secure. However, it can provide visibility into the components of an application that can then be used to inform DevSecOps best practices, which organizations are implementing more broadly as responsibility for application security continues to shift left toward developers.

    There are two primary specifications that organizations can use to create an SBOM. The Software Package Data Exchange (SPDX) specification is an open standard for communicating SBOM information including components, licenses, copyrights, and security references. Defined by the Linux Foundation, SPDX is now recognized as the ISO/IEC 5962:2021 international standard. The second is the Software Identification (SWID) format that is today widely used by publishers of commercial software.

    Regardless of the format used to construct the SBOM, Bach said there’s an opportunity to more closely align the increased visibility it enables with the application security goals of the organization building the application.

    The primary reason SBOMs are not as widely used as they might be by now has as much to do with inertia as it does with technology. However, as organizations start to expect an SBOM to be made available, it’s clear application developers and builders will need to find a way to automate the SBOM process as applications are continuously updated. The challenge, of course, is that applications are now being updated with greater frequency—thanks mainly to widespread adoption of DevOps best practices—that now need to be integrated with any tool for creating and maintaining an SBOM.

  • Logz.io Extends Open Source Observability Platform

    Logz.io Extends Open Source Observability Platform

    Logz.io has enhanced its observability platform to include both unified dashboards and support for distributed tracing to monitor service performance enabled by the open source Jaeger agent software being advanced under the auspices of the Cloud Native Computing Foundation (CNCF).

    At the same time, Logz.io is making available a preview of an anomaly detection capability that leverages machine learning algorithms to build an artificial intelligence (AI) model to monitor data.

    The company has also added support for OpenSearch, a fork of the open source ElasticSearch software project that is no longer available via an Apache license. OpenSearch was created by Amazon Web Services (AWS) and Logz.io as an alternative search engine.

    Finally, Logz.io has added a security event management capability that makes it easier to classify, prioritize and collaborate on mitigation of cyberattacks as they are discovered.

    The Logz.io observability platform is entirely based on open source software. In addition, the company has made contributions to enhance both Jaeger and the OpenTelemetry project, including contributing an OTEL Span Metrics Processor that makes it easier to aggregate metrics.

    Logz.io CEO Tomer Levy said that, in general, it’s still early days as far as observability is concerned. As a concept, observability traces its lineage to linear dynamic systems. Observability at its most basic level measures how well internal states of a system can be inferred from knowledge of its external outputs. The overall goal is to make it easier to query machine data in a way that enables DevOps teams to proactively discover the root cause of issues before they cause further disruption. That’s a major advance over monitoring platforms that typically only provide predefined metrics to identify when a specific platform or application is performing within expectations. The issue, however, is most IT teams really don’t know what questions to ask to take full advantage of an observability platform, said Levy.

    However, as more organizations are exposed to observability platforms, it will become easier to advance both DevOps and DevSecOps as more platforms continue to leverage AI to provide remediation capabilities and automate processes based on the massive amounts of data collected via these platforms, said Levy. There is simply no way the average IT team will be able to manage all the microservices being strewn across a distributed computing environment without the aid of an AI model trained using observability data, he added.

    As DevOps continues to evolve, it’s more than likely that the best DevOps engineers are not going to want to work for organizations that have not invested in some form of observability. The odds of being consistently successful as a DevOps engineer decline sharply as IT environments become more complex.

    On the plus side, it should soon become a lot easier as the overall level of application instrumentation increases to manage those complex environments and to do so at a level of scale that doesn’t require a small army of DevOps engineers.

  • Top 10 DevOps Things to Be Thankful for in 2021

    Top 10 DevOps Things to Be Thankful for in 2021

    Thanksgiving has been observed in the U.S. since 1861 and was officially designated as an official holiday in 1863 by president Lincoln. Of course, the tradition itself traces its lineage to the early days of the Pilgrims.

    This past year, of course, has been one of recovery from a pandemic that caused the deaths of millions of people around the world. Vaccines have saved the lives of countless more, so there is much to be thankful for despite all our recent trials and tribulations. In that spirit, there are more than a few things for DevOps professionals, in particular, to be thankful for this year starting with:

    Digital business transformation: Organizations of all sizes have had to adjust their business models because of both major changes in buyer behavior and an ongoing supply chain crisis. DevOps teams are at the core of any major digital business transformation initiative, so the demand for DevOps expertise continues to outstrip the available supply. Going forward, it’s been predicted we’ll see more digital transformation in the next 10 years than in all the last 40 years combined. The vast majority of those initiatives would never see the light of day if it were not for the expertise of developers and engineers working hand-in-hand to build and deploy applications faster than ever.

    Low-code platforms: As the demand for applications to drive digital workflows spiked in the wake of the pandemic, professional developers relied more on low-code platforms to decrease the time required to build an application. While many complex applications will still be built using procedural code, it’s clear not nearly as many applications could have been deployed without relying on low-code platforms to build them. It remains to be seen how many of the next generations of low-code applications will be built by so-called citizen developers that may soon be flooding DevOps pipelines with application code.

    Microservices: As an architecture for building applications the core concept of employing loosely coupled services together to construct an application goes all the way back to when service-oriented applications (SOA) were expected to be the next big thing in the 1990s. Microservices have, of course, been around for several years themselves. However, with the rise of containers as a software artifact for building microservices, it’s only now becoming apparent that building an application constructed using microservices not only takes less time but is easier to maintain and also more resilient.

    Observability: As a concept, observability traces its lineage to linear dynamic systems. Observability in its most basic form measures how well the internal states of a system can be inferred based on knowledge of its external outputs. In the past year, a wide range of IT vendors introduced various types of observability platforms. These make it easier for DevOps teams to query machine data in a way that enables them to proactively discover the root cause of issues before they cause further disruption. That’s a major advance over monitoring platforms that typically only provide predefined metrics to identify when a specific platform or application is performing within expectations.

    Biden’s executive order: In the wake of a pernicious attack against IT service management software from SolarWinds, the Biden administration published a sweeping executive order that requires federal agencies to implement a wide range of security best practices. Foremost among them is guidance concerning the security of software supply chains; which resulted in many organizations reviewing their entire software supply chain to make sure similar breaches will not occur.

    DevSecOps: As an extension of any DevOps best practice, DevSecOps generally describes an overall shift left in terms of making developers more responsible for application security. While still in its relative infancy, DevSecOps is starting to gain traction as DevOps and cybersecurity teams look to bridge a cultural divide that is as wide as any in IT. The ultimate goal is to deliver more secure applications without reducing the speed at which they are being built and deployed.

    GitOps: At its core, GitOps is an operational framework that takes DevOps best practices used to automate application development and extends them to infrastructure. The basic idea is that the configuration software created to manage infrastructure-as-code is just another software artifact to be stored in a Git repository. The approach provides a means for IT teams to achieve a continuous delivery goal that, more than a decade after DevOps best practices were defined, remains elusive. It’s still early days as far as GitOps best practices are concerned, but this has been the year when GitOps become part of the mainstream DevOps conversation.

    Kubernetes: Simultaneously one the most powerful and most complex IT platforms ever to come down the proverbial enterprise IT pike, Kubernetes has emerged as a de facto standard for building and deploying container applications that can more efficiently scale up and down as required to consume IT infrastructure. The challenge (and the opportunity) now is to make Kubernetes much more accessible to mere mortal IT administrators that typically don’t have the programming skills of a site reliability engineer (SRE) and which are both hard to find and retain.

    GraphQL: Employed as a query language for application programming interfaces (APIs), GraphQL provides a more efficient means of accessing data than the REST APIs that have been employed by developers in the past. Originally developed by Facebook, GraphQL adoption is still in its infancy but it’s clear APIs based on GrapQL will one day be as plentiful as REST APIs. That’s especially critical for microservices-based applications that rely on APIs to access data across a highly distributed computing environment that requires a more efficient means for transferring data.

    Venture and Wall Street capital investments: Investment firms are helping fuel DevOps advances as they continue to make capital available for companies vying for their share of a market that is expected to see compound annual growth rates of 24.7% through 2026. There will, of course, eventually be a wave of consolidation as merger and acquisition activity inevitably ensues. In the meantime, however, there is a wide range of DevOps technologies that might not otherwise be as broadly adopted if it were not for all the cash being poured into DevOps companies.

    The most important thing, of course, is that so many DevOps professionals are still here to build and deploy applications. It’s been a trying year for all concerned; however, as the pandemic crisis continues to recede, we can all be thankful for the opportunities that lie ahead.

  • Nvidia/ARM Wavering | Google Outage Outrage | Backblaze IPO on Fire

    Nvidia/ARM Wavering | Google Outage Outrage | Backblaze IPO on Fire

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    Nvidia Can’t be Allowed to Buy Arm

    First up this week: Nvidia’s faltering attempt to buy Arm: Regulators around the world are alarmed by the idea. And now the UK—the legal home of the company—has taken the next step, which could nix the deal.

    Analysis: RISCy home for ARM

    This matters because ownership of the ARM ISA and the core designs would be a huge strategic stick to beat Nvidia’s competition with. Arm doesn’t make chips—it licenses technology to Nvidia and many of its competitors—AMD, Apple, Broadcom, Samsung and Qualcomm, to name but five. So this deal really shouldn’t go through, regardless of Nvidia’s fluffy PR reassurances.

    ARM chips aren’t just found in 99% of smartphones, but are an increasing fixture in the datacenter—especially where “performance per Watt” is important. They’re even found in exascale iron, including the Riken Fugaku, the world’s fastest supercomputer.

    Helen Thomas: Nvidia faces tough Arm wrestle

    It is one thing for a regulator to give your deal a thorough going-over. It is another for it to imply publicly that your assurances [aren’t] credible. … Nvidia got both from the UK’s Competition and Markets Authority. … The CMA’s full phase 1 report doesn’t make for easy reading.
    …
    The transaction, which was announced in September 2020, is bogged down in regulatory reviews around the world: … Arm’s “neutral” business model has always made it a difficult acquisition. … Arm licenses its designs to other chipmakers, providing scope to restrict rivals’ access.
    …
    Arm was a political talking point in 2016, when its sale to SoftBank … was waved through with embarrassing ease. There is now a much more muscular political take on UK national security concerns with a particular focus on tech.


    But if it’s owned by a Japanese company, why is the UK getting involved? Glad you asked—Aighearach answers:

    What did they buy? Shares of a UK company. … Then it is a UK company! … Buying shares is buying ownership, but it doesn’t move the company to another jurisdiction.
    …
    This blathering about “sold to a Japanese firm” is getting old.


    Nvidia claims, “The transaction will … boost competition and innovation.” Falmari finds that hilarious:

    Did Nvidia’s spokesperson previously work as a stand-up comedian? Because as jokes go, “boost competition” is a whopper. [Do you think] that’s what a company like Nvidia wants—to boost the competition?

    Google Cloud Outage has Ripple Effect

    Google’s load balancers all went offline for two hours on Tuesday. Aside from the knock-on Google Cloud failures, DevOps tools such as BigCommerce, Datadog, Deno, Discord, Egnyte, Evernote, Linear, Netlify and Toggl were reported to be affected. But perhaps the worst impact was felt by the poor, unfortunate users of Pokémon GO.

    Analysis: Paralysis

    In theory, putting all your eggs in one basket is okay—so long as that basket can reliably stop your eggs smashing on the pavement. But this isn’t the first time we’ve seen Google Cloud issues such as this; I can’t imagine it’s the last.

    Steven J. Vaughan-Nichols: Google glitch triggers major internet outage

    The result? Many of us have seen 404 errors when trying to get to some of our favorite web pages. … There appears to be more major sites having problems than not.
    …
    The specific problem seems to be with the GCP load balancers. These, working with Google’s Cloud Delivery Network (CDN) [are] designed to stop website failures and Distributed Denial of Service (DDoS) attacks by putting your website behind a single anycast IP.


    Are you feeling a touch of déjà vu? jeffbee, too:

    This is a sort of repetitive outage for Google. They’ve wiped out the GSLB configs before.

    A year ago there was also that big outage where they blanked out the whole Gmail delivery config and started rejecting all mail. … Config safety is not their strong suit.


    The #3 IaaS provider won’t get to #2 if it keeps this up. Here’s u/totopapa:

    I’m trying to convince my company to switch from AWS to GCP but an outage like this one is definitely a showstopper.

    Backblaze IPO Burns Brightly

    Backblaze is now NAS:BLZE. In its first week, the stock jumped 60%.

    Backblaze is well known for its tiny-margin consumer backup product and its open-source “storage pod” hardware—but is less well known for its DevOps-focused “B2” cloud storage service.

    Analysis: Can it Avoid Death-by-Packrats?

    Backblaze is trying to grow this newer DevOps business, without losing its classic consumer chops. But many of its former competitors have essentially dumped their “unlimited” offerings—it’s the old tragedy-of-the-commons problem.

    Jordan Novet: Backblaze [goes] public on small valuation and minimal funding

    In an era of blockbuster market debuts preceded by astronomical private financing rounds, Backblaze … a cloud service for backing up data … is a throwback to a time when sub-billion-dollar companies routinely went public to raise capital and build their profile … to win the trust of customers.
    …
    Backblaze shares climbed 24% in their debut on Thursday and another 12% on Friday, lifting the price to $22.31 and giving the company a market cap of around $650 million. … By the standards of its peers, Backblaze is a micro-cap.
    …
    One of Backblaze’s competitors is Amazon, the world’s fourth most valuable U.S. company. … Backblaze currently stores two exabytes of data — that’s 2 million terabytes — and has a half-million paying customers.


    Donovan Jones: Backblaze Targets $100 Million IPO

    The firm provides cloud-based storage services to small businesses and individuals worldwide. BLZE has grown revenue and gross profit impressively through the 2020 pandemic period and the IPO appears fairly valued, so is worth a close look.
    …
    The firm’s financials show strong revenue and gross profit growth. … The firm’s dollar-based net revenue retention rate for 2020 was 114%—a solid result. … The market opportunity for providing cloud-based storage services to small businesses and individuals is large and expected to grow at a material rate. … As for valuation, compared to partial competitor Dropbox, management is seeking the same Enterprise Value/Revenue multiple. [However] the combined revenue growth rate and EBITDA percentage rate … was 3% as of June 30, 2021, so the firm needs improvement in this regard.


    But there’s a long way to go, opines John Kozubik (aka rsync):

    Of the ~500,000 customers, 70,000 are B2 customers and 430,000 are “backup” customers—which are flat-rate “unlimited” users. … Roughly 85% of the user-base are paying $7/mo for “unlimited backup.”
    …
    These flat-rate users are very fickle and price-sensitive. … Roughly 85% of the user-base are paying $7/mo for “unlimited backup.” … It’s a fools errand to pursue this market and I assume the pivot to B2 was to build something more sustainable with a much higher price and usage ceiling.

    The Moral of the Story: Success is a journey, not a destination


    You have been reading The Long View by Richi Jennings. You can contact him at @RiCHi or tlv@richi.uk.

    Image: Kristin Wilson (via Unsplash)

  • Building on Terraform: Evolution, not Revolution

    Building on Terraform: Evolution, not Revolution

    Terraform is widely adopted among both enterprises and service providers who embrace infrastructure-as-code (IaC) as their model for infrastructure provisioning automation in DevOps environments. And while the technology has limitations, innovations seeking to unseat Terraform are highly fragmented and have seen tepid adoption. This points to a market that’s settling on a de-facto standard, so rather than trying to replace Terraform with the next great innovation, infrastructure operators should instead find ways to extend and augment Terraform to suit their unique needs. Let’s explore several of the better-known shortcomings of Terraform, some drivers for the rise in new alternatives to the technology, and what users need to consider when contemplating a different path.

    Without a doubt, Terraform is big. Across the big three public clouds, the number of Terraform provider installs tells the story: At the end of August, there were 457 million provider installs for AWS, and around 72 million each for Azure and GCP. Yes, compared to longer-in-the-tooth infrastructure configuration management tools like Ansible, Chef and Puppet, IaC as a category is still early. But, the trend is clear: IaC is becoming widely used at scale, and Terraform is the IaC platform most enterprises go with.

    It’s also no secret that there are shortcomings to Terraform. A quick search on Reddit and Hackernews unearths the litany of complaints, some more valid than others: 

    • It’s tricky creating a Kubernetes cluster and then adding a resource to it
    • Challenges passing values through a config or dumping a pure representation of an elevated config
    • Cluster deletion is a pain
    • An overall lack of visibility into what’s happening during the provisioning process
    • The need for glue code to drive everything is a drain on developer productivity
    • Lingering challenges with backward compatibility
    • Continuous update challenges and the need to refactor
    • Scaling across multiple workspaces and multiple cloud accounts
    • Rollbacks can be tricky, especially when dealing with immutable infrastructure

    Terraform’s parent company, HashiCorp, of course, steadily works to address these and other pain points. But the whole thing highlights an interesting question: Why is Terraform so popular?

    Terraform is the only IaC solution that meets the needs of developers who want an IaC tool that preserves state, fits well into their CI/CD pipelines, is intent-based and gives them the right level of granularity. Terraform, even with its challenges, is better than the alternatives in these regards.

    Here’s why: 

    Terraform was, from the start, designed for and targeted to developers. It maintains state and supports iteration, making IaC a powerful model for automating infrastructure. PaaS solutions, by contrast, are not robust enough for most developers who do not need or want all of the infrastructure decisions abstracted away. Most cloud management platforms suffer from the least common denominator problem, and even though the technology has improved in this regard, it’s a solution that developers generally abhor for its granularity; it’s geared more to the needs of IT management. Configuration management tools, as an alternative, are not intent-based, so flow becomes too complicated.

    It’s interesting to note that state (maintaining the mapping between the resources that you’ve represented in your definition files and the actual physical resources that are in the cloud provider) was added into Terraform, helping resolve the issue of infrastructure drift. Terraform also realized that developers don’t necessarily need a UI. These tangible examples of the developer-centric design ethos at HashiCorp point to why it is so widely adopted.

    So, what’s driving the interest in new approaches to infrastructure automation? The proliferation of domain-specific use cases and an ever-present need to reduce time-to-value are the leading reasons. Domain-specific automation options are becoming more dominant over OOB solutions because they take too long to get to automation; one tool does not fit all. Domain-specific automation addresses issues such as security automation, AIOps, DevSecOps, security compliance, integration with container scanners—not just generic orchestration. People are looking for value very fast, which can be found only with domain-specific automation—and therefore, fragmentation happens as different solutions emerge to address those cases.

    There are several interesting alternatives to Terraform. Terragrant is used along with Terraform to expand its capabilities to multiple workspaces. It allows you to use the same template across many environments. Pulumi is another alternative that has achieved a measure of adoption and visibility. It provides native code support which addresses the problem with YAML file development complexity. Another more ambitious undertaking is the Crossplane open source project. It takes a Kubernetes-centric approach to creating a universal, cross-cloud control plane. It looks good conceptually, but it’s a fundamentally different approach, and adoption is nascent.

    Turning to one of the Terraform alternatives isn’t necessarily a bad idea, but is fraught with landmines. In the end, this choice is less about technology and more about adoption and ecosystem momentum. Vendor and professional investments in tools, training, and skill sets surrounding Terraform is a formidable moat, and it cannot be easily replicated. Tools that would win over Terraform would have to win adoption, and achieving adoption means delivering all the enterprise-grade features and support that Terraform users currently enjoy. Chef and Puppet are examples of how this could happen. Both were successful technologies, but they were slow to react to the challenge presented by Terraform. Ultimately, these projects suffered from a legacy mindset and were overtaken by Terraform’s more focused, simple solution.

    Most of the issues with Terraform lie in managing it at scale. When you have many workspaces and templates, it becomes hard to manage; it’s also a challenge that templates cannot be generalized among environments, resulting in a lot of template redundancy. Also, Terraform still needs help handling Day 2 operations and visibility into drift detection. 

    When considering ways to extend Terraform to address a broader set of use cases, you should first consider how well things will interoperate across your environment. If you have multiple clouds, multiple service layers and multiple cloud vendors, this is probably the most important factor. Also, investigate any new technology’s ability to support the things about Terraform that are critical to your Day 2 operations. Things like secrets and state come to mind. You’ll want enough management and monitoring in the new solution without it being a new burden to carry. You should be able to use your existing Terraform files, and you should be able to support all the versions of Terraform that you run. Compliance with governance and audit are key, and the new solution should support Terraform for both your on-premises/air-gapped environments and your cloud environments. Ideally, anything you add to Terraform would be open source, eliminating new complications from cross-vendor lock-in.

     

    Terraform use case 

    Architecture for how 8451 evolved its Terraform deployment

    For all its shortcomings, Terraform remains the best option on the market for DevOps organizations, and there are ways to manage it with fewer headaches and without developers wasting so much time writing glue code or reinventing the wheel over and over again. One way this happens is by integrating Terraform with meta-orchestration tools that allow Terraform to become more flexible in managing environments and workspaces at scale. The open source Cloudify project is one example.

    Yevgeniy Brikman gave a talk two years ago at HashiConf that has aged well. At the time, DevOps was still in the Stone Age. Today, it’s probably in the Bronze Age. Those of us building IaC today are being asked to deploy infrastructure that looks like modern cities, but the tools we have to work with remain relatively primitive. What we’re building takes too long to provision and it looks bespoke and temperamental at best or kludgy and fragile at worst.

    It’s a story repeated over and over: Teams try to build their own, homegrown solution first as a wrapper on top of Terraform. Ultimately, they realize that it’s unsustainable as a lack of engineering resources becomes a bottleneck. We need solutions that augment—not replace—what they’ve already built and what works. That’s almost always Terraform. With Terraform, DevOps organizations get a proven IaC option that’s targeted to the needs of developers, with a substantial ecosystem and market uptake. When considering its shortcomings, it’s best to embrace an evolution strategy, and not start from ground zero.

  • Era Software Adds SaaS Edition of Log Management Platform

    Era Software Adds SaaS Edition of Log Management Platform

    Era Software has unfurled a software-as-a-service (SaaS) edition of its platform for collecting and managing log data that is based on a time-series database it built.

    At the same time, the company has revamped the user interface of the underlying EraSearch log management platform that IT teams can deploy themselves to make it simpler to troubleshoot IT issues.

    Stela Udovicic, senior vice president of marketing for Era Software, said the EraCloud edition of the log management platform provides a more accessible way to access a log management platform based on an object storage system that can handle any type of log data.

    There’s also an indexing engine based on machine learning algorithms that enables the platform to scale better than rival platforms, she noted.

    Available via the Amazon Web Services (AWS) marketplace, EraCloud collects, indexes and analyzes log data from cloud applications, Kubernetes and IT infrastructure platforms to provide a centralized means of collecting data at a lower cost. In addition to being able to continuously monitor billing costs, an EraSearch cluster can now be provisioned in minutes.

    Era Software, previously known as EraDB, earlier this year raised an additional $15.25 million in financing to drive development of a log management platform that provides an alternative to open source platforms, such as Elasticsearch, that don’t scale as well. The result is a higher total cost as the volume of log data that is less efficiently managed via Elasticsearch steadily compounds. As part of that effort, the company also supports a subset of the Elasticsearch application programming interface (API) to facilitate transitioning from Elasticsearch.

    Many organizations today already have terabytes of log data they are trying to manage and store, noted Udovicic.

    It’s not quite clear how many IT organizations are considering alternatives to log management platforms. However, as the amount of log data that IT teams want to be able to store over a longer amount of time increases—in part to facilitate greater observability—the pressure to contain the cost of retaining all the data is also rising.

    Observability, in general, requires combing logs, metrics and traces—a specialized form of logging—in a way that makes it easier to identify the root cause of an issue by querying data. It differs from monitoring in that observability is achieved using a predefined set of metrics that track whether a platform or application is performing within expectations.

    It may be a while before most IT teams generally achieve observability, but the journey typically starts with finding a more efficient way to collect log data in both proprietary and open source formats. Rather than having to normalize all that data, Era Software is making a case for analyzing that data in whatever format it happens to be available.

    One way or another, a much larger percentage of the applications deployed by IT teams in production environments will have to be instrumented to achieve observability. The issue now is determining how best to make all the data collected from those applications available to DevOps teams at a reasonable cost.

  • What Developers Need to Know About Edge Computing

    What Developers Need to Know About Edge Computing

    More and more companies are living “on the edge” as they move from “I’ll think about edge computing” to “I don’t think our business can optimally grow and compete without edge computing.” Developers will play a key role in helping organizations use the edge, but their effectiveness will depend on having edge-aligned processes, products and people in place.

    Edge computing takes place near the physical location of either the user or a data source. It’s one way that organizations can use and distribute a common pool of resources across multiple locations. The edge model provides users with faster, more reliable services while expanding the potential of hybrid cloud computing for organizations. Customers and businesses alike will benefit from the way in which edge computing optimizes the use and application of artificial intelligence and machine learning by running data-intensive applications closer to the data source.

    While it is in many ways a question of architecture, applications are key to effectively exploiting the edge. Here are some of the things developers—and their organizations—need to know.

    Edge computing and cloud computing are not mutually exclusive. The best solutions will leverage both platforms. For example, centralized computing might be used for storage and compute-intensive workloads, while edge computing would be used to exploit data at the source, in near-real-time. Infrastructure and company culture need to be in place to support this, but developers must be thinking in terms of building applications that can cross and bridge computing platforms.

    Consistent tooling is essential. Developing for legacy IoT was very specific and required specialized tools and skills. The modern edge, in contrast, is part of a technology continuum, and developers need tools that they can use across that continuum, including:

    • Provider/enterprise core: The “non-edge” tiers, owned and operated by public cloud providers, telco service providers or large enterprises.
    • Service provider edge: Tiers are located between the core or regional data centers and the last mile access.
    • End user premises edge: Tiers on the end-user side of the last mile access can include the enterprise or the consumer edge.
    • Device edge: Standalone (non-clustered) systems that directly connect sensors/actuators via non-internet protocols. This represents the far edge of the network.

    Containerization enables flexibility, scale: Linux container technology wasn’t built for the edge, but the model fits perfectly with edge requirements for modularity, segregation and immutability. 

    Keep an “open” mind. Open APIs open up new opportunities. Developers using open APIs can access real-time data programmatically, making it possible to build applications that can leverage data as it’s “born.” Likewise, a focus on open source development tools and platforms will help to ensure that organizations can continue to leverage the edge as the model evolves. 

    What’s different is the devices. Developing for the edge is just like developing for any other platform–except that developers must write consistent and secure code that can be seamlessly copied to all device types. 

    Edge computing may not start with developers, but developers will play a huge role in organizations’ ability to leverage the edge. A focus on flexibility, openness and the future will help ensure that developers build applications that fully exploit and expand the edge advantage. 

  • Grafana Labs Expands into Incident Management Realm

    Grafana Labs Expands into Incident Management Realm

    During its online ObservabilityCON 2021 conference, Grafana Labs this week previewed an incident management platform in addition to making available an on-premises edition of its distributed tracing tool.

    Tom Wilkie, vice president of product for Grafana Labs, said the Grafana OnCall incident management platform is a natural extension of the observability tools the company already provides using the widely employed open source Grafana dashboard. The platform, which is expected to be generally available in 2022, is based on software Grafana Labs gained by acquiring Amixr earlier this year.

    Grafana OnCall unifies all the alerts created by Grafana, Prometheus and the Alertmanager tool the Prometheus community created to streamline the number of alerts generated by the platform. The goal is to reduce the alert fatigue a DevOps team might experience when dealing with an IT issue that can be traced back to a single root cause, noted Wilkie.

    Grafana

    At the same time, Grafana Labs made Grafana Enterprise Traces available for on-premises IT environments. It is based on Grafana Tempo, a cloud service that makes it possible to store and then analyze distributed traces using inexpensive object storage resources. The latest version of the core open source Tempo project has also been updated to add service graphs and the ability to search recent traces.

    Grafana Labs this week also updated Loki, an open source log management platform, to add support for out-of-order log lines, custom log retention policies per log stream and per tenant and a deletion API that makes it easy to scrub log data. Version 2.4 of Loki also adds additional high-availability capabilities.

    Other updates to the Grafana Labs platform include support for recorded queries, a set of pre-built dashboards that enable users to quickly drill into their Prometheus metrics data, the ability to consume metrics from the AWS CloudWatch service provided by Amazon Web Services (AWS), improvements to its ability to monitor Kubernetes clusters and updates to the open source load testing tool the company gained by acquiring k6 last year.

    Grafana Labs is also committing to expanding its integration with Microsoft Azure cloud service alongside its integrations with AWS, noted Wilkie.

    While competition among providers of observability platforms is already fierce, Wilkie said an observability platform based on an open source dashboard that can already collect data from more than 100 sources provides a unique advantage. At the core of the Grafana Labs platform is a centralized framework for collecting data generated primarily by the open source Prometheus monitoring platform being developed under the auspices of the Cloud Native Computing Foundation (CNCF).

    While Prometheus is used most often in Kubernetes environments, it is starting to gain traction in monolithic application environments as well; it provides an open source approach to defining a de facto standard for collecting metrics and other data that can be consumed by an observability platform.

    It’s not clear yet to what degree IT teams are distinguishing between monitoring and observability. Monitoring focuses on predefined metrics to identify when a specific platform or application is performing within expectations. Observability combines metrics, logs and traces to instrument applications in a way that makes it simpler to troubleshoot issues without relying solely on a limited set of metrics that are pre-defined and monitor a specific process or function.

    Regardless of whether IT teams view observability as being distinct from monitoring, it’s clear that the ability to manage IT environments is improving at a rapid rate. The challenge, of course, is the IT environment itself is becoming more complex with each passing day.

  • Automated Refactoring with Moderne

    Automated Refactoring with Moderne

    Jonathan Schneider and Olga Kundzich are co-founders of Moderne, a Seattle-based startup. Moderne received $4.7 million in seed funding in July 2021 to commercialize OpenRewrite, an open source automated refactoring tool for code (initially Java) that Schneider started while at Netflix. Moderne’s technology reduces the tedium and time it takes for software remediation; a big problem in today’s world of cloud-native development. The video is below, followed by a transcript of the conversation.

    Announcer: This is Digital Anarchist.

    Alan Shimel: Hey, everyone. Welcome to another segment of TechStrong TV. We have a new company to detail for you, and we have two cofounders here to tell you all about it. It’s a great open source story as well. Let me introduce you to – and I’m gonna do my best not to mess it up – Olga Kundzich, Kundzich, Kundzich – three times the charm – and Jonathan Schneider. I think I got that one on the first try. Olga, Jonathan, welcome to TechStrong TV. 

    Olga Kundzich: Great to be here. 

    Jonathan Schneider: Thanks. It’s good to be here. 

    Alan Shimel: Thank you. So the only rules is only one of you can talk at a time, but both of you can talk as much as you’d like. But why don’t we start off with maybe a little bit about each of you. Olga, if you don’t mind, why don’t you go first and maybe share with the audience a little bit of your background and how you came to be here today. 

    Olga Kundzich: Great. So my background, I was working at Pivotal on a Spinnaker continuous delivery solution, and that’s where Jonathan and I had met. We worked together there for about a year and a half. Prior to that, I worked at EMC in engineering for, in different functions for a number of years. So when we were working together on Spinnaker, we talked to a lot of people in the industry. There was a lot of interest around that product, and kind of we realized that everyone had sort of the same problem. We would start a conversation about some old feature, and it will come back to, well, all my application is still stuck on this old, aging framework, and let me talk to you in a year when I migrate all of this back.

    So and the problems were highly repeatable and kind of same framework, same third-party vulnerabilities, same libraries, APIs, so after reflecting on this we realized that this is really not a coincidence. It’s a consequence of modern application development practices. Our software right now is 90 percent third-party dependencies, open source, APIs, so the problem’s kind of highly repeatable. And that’s kind of led, what led us to, back to technology that Jonathan developed earlier at his time with Netflix and kind of take it, develop a lot of more functionality into that sort of large-scale distributive factoring that will help us solve these type of problems, the framework migrations, API migrations, CVE patching that cross all of these open-source and third-party dependencies. 

    And I will let Jonathan speak about his background and a little bit on the open-source framework.

    Jonathan Schneider: Yeah, so Netflix had this engineering culture called freedom and responsibility, so product engineers were sort of, had the freedom to do whatever they wanted really. They could use their, a different build tool, they could use their particular Java style, they could use whatever libraries they wanted, and so forth. And so as a central team, if you’re on an engineering productivity team or you’re on a platform team and you’re trying to effect some change inside the organization, we couldn’t put up gates in front of product engineers that would, say, break their build if they didn’t comply with a certain thing. Instead, you tried to build tools that would help bring them along on a journey with you, and I kept hearing the same thing over and over again from engineers which is, hey, I got a lot on my plate. Happy to move forward to this new pattern that you want to see, if you do it for me. If you just do the work for me, I’m happy to commit it. 

    Alan Shimel: Nice of them. 

    Jonathan Schneider: And of course, you know, you think there’s two of us and 700 of them, and that doesn’t scale super well. Right? So that’s where we started working on a way of actually automating the source code transformation. When we developed that technology, one of the consequences of that freedom and responsibility is that there was not a consistent style of the code or consistent patterns, necessarily. So that refactorization we built had to make transformations that were, looked like they were a developer writing in that code base stylistically consistent, hundred percent accurate changes. And so we got some interesting results out of that, so when Olga and I were working together and you would hear, “Oh, I’m still stuck on Spring Boot 1 and I’m trying to get to Spring Boot 2,” or, “I’m stuck on JUnit 4. I’m trying to get to 5,” we just found another application of this at a larger level in a highly repeatable problem space. 

    Alan Shimel: Fair enough. You know what, I’ve spent the last 20 plus years talking to founders. I’ve founded my own companies. This is what happens. You see a real problem. Right? See a real-world problem. There’s got to be a better way we could solve it. Sometimes you find out solving that problem is not as easy as you thought. Sometimes it is, very rarely. But you work through it and that’s what startups are all about. In your case, though, I mean, look, Spinnaker, which is now part of CDF, right, was managed by CDF, but Google, Netflix were instrumental in the development of Spinnaker. But Spinnaker is an open source tool, and you guys went with an open source route in what you’ve developed. Why don’t we talk about that a little bit? 

    Jonathan Schneider: Sure, yeah. There’s really two parts to our product offering. There’s OpenRewrite, which is GitHub slash OpenRewrite. OpenRewrite is that core refactoring technology we’ve been talking about that can make stylistic automatic and consistent changes to a code base. On OpenRewrite we built recipes, so we have a recipe to migrate you from Spring Boot 1 to 2, from JUnit 4 to 5, to do code cleanup things, to do security vulnerability fixes. We want those recipes to always be Apache licensed and open source. Framework authors are going to develop the recipe, so the Spring team will develop the recipe that will move you from Spring Boot 2 to 3, or the Quarkus team at IBM or Red Hat would develop the recipe that’d move you from one version to another.

    What Moderne does commercially is we take those recipes and we run them at massive scale, so I could take that recipe and run them against tens or hundreds of millions of lines of code, and you can see that, the impact of that recipe running against your whole organization’s code base. 

    Alan Shimel: Fair. And so you mention – how did – it’s not Modern. It’s Moderne. Correct? 

    Jonathan Schneider: Yeah, we call it Moderne. Yeah.

    Alan Shimel: Moderne. So that’s the name of the company.

    Jonathan Schneider: It’s the name of the company, yeah. 

    Alan Shimel: And OpenRewrite is the open source aspect of it or –

    Jonathan Schneider: That’s right. 

    Alan Shimel: – owner of the solution. So let’s get into the mind of the entrepreneur here a little bit, Olga. Having worked at Pivotal and EMC for years before this, you saw this issue. You saw this problem. There’s got to be a solution. When did you say, you know what, this is something worthy of starting a company, right? When I was starting companies, one of the big questions is, is this a feature or a product? Right? Is it just a feature of someone else’s product, or is it truly a product in and of itself? And that makes a big difference in, obviously in what you’re building and what your aims and all of that. Olga, I’d like to hear from you. Feature, product, how did – what was kind of your thought process in saying, you know what, this could be its own company?

    Olga Kundzich: I think this, something like this is essential for managing modern cloud-native applications. These are becoming, the applications sort of become a glue code teaching all of these third-party components. They evolve and change at their own pace, and developers really right now are just restitching those things back together, upgrading CVs, upgrading versions, and it takes a really significant amount of time just to – sort of anecdotally we hear from people that 20 to 30 percent of their time, engineering time, is just now dedicated to these tedious migrations. So if we can automate this, we can unlock innovation, productivity of engineering organizations, and help a lot of businesses actually innovate.

    Alan Shimel: Got it. Jonathan, what about from your end? Well, let me ask you both. Is this your first time starting a company? 

    Jonathan Schneider: It’s first time starting a company, yeah. I think it’s real special to me. I think one thing I love about this business is that we’re really after helping organizations fix their code, and our outcome and our future success is tied directly to how much code we’re able to fix, how much help we’re able to provide. We’re not about reporting and saying like, you know, here’s the problems that you, engineering team, need to fix. When we partner with an organization, we’re kind of in the trenches with them getting these issues off their plate.

    We talked to one organization recently – and this is why I think it’s a product and not a feature. We talked to one organization recently and they say we just look at our top ten issues reported by a series of static analysis tools they have in-house, and it says, well, it’s gonna, these tools estimate it’ll take 1,700 developer days to fix these top ten issues. So that’s like 4.8 years’ worth of developer time to fix those top ten issues that they have, and those aren’t false positives. Those are just literally top ten critical problems they have right now. So we developed recipes for a lot of those, and their engineers said, you know, “We estimate it would take us five days with these recipes to apply that change and be done with it.”

    When I think about 1,700 developer days or 4.8 years of time, what that really means to me is infinity, or like it means that the engineering team is never going to get through that backlog of issues. Five days seems like it is possible, and that’s, there’s this line we have, the, basically software grows until it can no longer be maintained, and I think our business is in keeping that software in a maintainable state so that the business can continue to innovate and thrive. 

    Alan Shimel: Absolutely. So companies launch now – we didn’t even mention the website. Shame on –

    Jonathan Schneider: Yeah.

    Alan Shimel: What’s the website? 

    Jonathan Schneider: Yeah, so the website is moderne.io, so it’s modern with an E.

    Alan Shimel: Can you spell that, Jonathan? M-O- –

    Jonathan Schneider: Yeah, it’s M-O-D-E-R-N-E, so modern with an E, dot I-O.

    Alan Shimel: Dot I-O.

    Jonathan Schneider: Yep, that’s right. We do have a restricted beta going on right now that you can just sign up for that on that website, and we’ll do an invite to you and you can see how we can apply these recipes across tens of millions of lines of code. 

    Alan Shimel: Absolutely. Are you working with CDF and the Spinnaker community?

    Jonathan Schneider: We’re really in an adjacent product space now. I think there’s a lot of value to operational automation, what Spinnaker is doing, what different APM vendors are doing. I think we were working in that and we saw just an adjacent pain point that we just decided to go after instead. 

    Alan Shimel: Good stuff. Well, guys, I want to wish you nothing but a lot of success with this. It’s always, you know, I gotta be honest with you. I always enjoy interviewing new companies coming out of stealth and beta, first-time entrepreneurs, ’cause you make the world go around. And good luck. Keep us posted. Come back on and tell us more as you, as the story continues to unfold here. 

    Jonathan Schneider: Yeah, look forward to talking again soon. 

    Alan Shimel: All right. 

    Olga Kundzich: Thank you.

    Alan Shimel: Jonathan and Olga from Moderne, M-O-D-E-R-N-E, dot I-O, here on TechStrong TV. We’re gonna take a break. We’ll be right back.

    [End of Audio]