devopsy.kinsta.cloud

Category: DevOps and Open Technologies

Editorial Calendar for February 2016

  • Best of 2021 – 7 Popular Open Source CI/CD Tools

    Best of 2021 – 7 Popular Open Source CI/CD Tools

    As we close out 2021, we at devopsy.kinsta.cloud wanted to highlight the most popular articles of the year. Following is the fourth in our series of the Best of 2021.

    DevOps is a software development strategy that incorporates agile practices for fast, efficient product creation and release. It focuses on integration of development and operations teams, continuous integration/continuous delivery (CI/CD) and automation of tasks and processes.

    Typically, DevOps teams use pipelines to streamline and standardize processes. DevOps pipelines are toolchains that teams can use to automate tasks and provide visibility into the software development life cycle. In this article, we’ll cover seven popular open source CI/CD tools.

    How DevOps Will Change in the Near Future

    Although DevOps has been around for just over a decade, it has continued to evolve. New DevOps technologies and changing infrastructure have caused teams to adapt and technology advancements that will continue to drive change. Below are a few of the ways you can expect DevOps to evolve in 2021 and beyond.

    AI and Data Science

    Artificial intelligence (AI) and machine learning have significant potential to improve current DevOps processes. Teams can use AI to identify inefficiencies in pipelines, optimize testing or evaluate user needs. Increasing amounts of data about environments, customers, markets and individual habits can all be applied to increase productivity and, ultimately, revenues.

    Serverless Architectures

    Serverless platforms, such as the AWS serverless ecosystem, including AWS Lambda and Fargate and the Azure serverless platform, including Azure Functions and Azure App Service, are becoming increasingly popular. These platforms enable DevOps teams to focus on just their code and eliminate many other variables, such as concern for host operating systems. These architectures provide scalability, deployment speeds, and user experiences that are difficult to match. This value will further secure FaaS in the coming years.

    Cloud-Native

    The growth of cloud-native applications has both driven and been driven by widespread adoption of cloud services. The containers and microservices that make up many cloud applications are used in many pipeline deployments and environments. Efficiently developing cloud-native applications require agility and automation that are difficult to meet without DevOps strategies.

    For the above reasons, DevOps has become almost synonymous with cloud-native. As cloud-native technologies change, tooling and processes adapt, as well. Additionally, since cloud growth isn’t slowing down, you can expect that DevOps will continue to focus on and refine cloud-native development.

    DevSecOps

    When developing applications in an automated CI/CD pipeline, security cannot be an afterthought. Especially in complex microservices architectures, security must be built in from the planning and development stages. Everyone must be aware of security requirements – developers, DevOps engineers and infrastructure teams.

    Transitioning to a DevSecOps paradigm requires a culture change, but also a new technology stack. Shifting security left requires tight integration of application security tooling into the development life cycle, and implementation of cloud security solutions that can help gain visibility, and apply automated security controls, to complex cloud environments.

    Which CI/CD Tools will You Use in 2021 and Beyond?

    As technologies change, so does tooling. While you can continue to use the same tools you started with, you should at least consider if other tools can increase your efficiency. Below are a few popular, open source tools you might want to integrate.

    Argo CD

    Argo CD is a CI/CD tool for Kubernetes development. It is an open source project which is currently in the incubation status at the Cloud Native Computing Foundation (CNCF). It uses Git repositories to store the state of Kubernetes applications, monitors applications and can resync clusters to the desired state, as represented in the git configuration.

    This innovative approach also allows you to store multiple desired states of a Kubernetes application, using branches, tags, or by pinning manifest versions using a Git commit. This provides a flexible environment for managing Kubernetes configurations during the development process.

    CircleCI

    CircleCI is an open source CI/CD tool. It includes features for job orchestration, resource configuration, caching, debugging, security and dashboard reports. CircleCI integrates with a variety of tools, including GitHub, Heroku, Slack and Docker.

    CircleCI is available in three tiers, one of which is free. You can use it in the cloud or on-premises with Linux, Mac or Windows machines.

    GitHub Actions

    GitHub Actions are a recently released open source tooling feature you can use to automate workflows. It enables you to build, test and deploy code directly from GitHub. You can base your automation on any GitHub events, including push or issue creation.

    GitHub actions include features for matrix workflows, hosted runners for all major operating systems, built-in secrets storage and live log reviews. It supports all common programming languages.

    GitHub actions includes hosted runtime environments that are free to use for the first 2000 minutes of use per month. If you need more time for testing, you can purchase time by the minute or in bulk. You can use GitHub actions by including them in your YAML files.

    GoCD

    GoCD is an open source build tool that lets you build CI/CD development pipelines. At its core, it is a continuous integration (CI) server, which lets you work with complex release workflows. It goes beyond classic CI to let you construct a full continuous delivery (CD) pipeline, all the way from automated and secure deployment to production.

    GoCD defines pipelines using code checked into source control – this infrastructure-as-code (IaC) approach lets you test, manage and reuse pipelines across multiple projects. Pipelines can be represented by JSON or YAML templates.

    Another feature of GoCD is the ability to run pipelines both sequentially and in parallel, with configurable dependencies. You can view entire workflows from development to production and track a feature from initial commit to production deployment, using value stream mapping.

    GoCD is extremely flexible, providing an ecosystem of plugins and the tool also allows you to develop and contribute your own.

    Gradle

    Gradle is an open source build automation tool that has been around for about a decade. It continues to be well-adopted and supported. Although it is not a new tool, it can be a good alternative if you’re looking to replace Maven as your build management solution. Gradle is more extensible and faster than Maven, and can be used for native development of C/C++ applications.

    Gradle includes features for incremental builds, build caching and parallel execution. It supports a variety of languages and platforms, including Java, Scala, Android and C/C++. You can integrate Gradle with a range of tools, including Eclipse, IntelliJ and Jenkins.

    You can use Gradle on-premises or with a virtual machine in the cloud. Gradle is free to use; however, if you want enterprise support or training, you need to pay an additional fee.

    Helm

    Helm is an open source Kubernetes package manager. You can use it to define, install and upgrade Kubernetes-orchestrated applications. Helm includes a wide range of pre-built charts (sets of distributable Kubernetes resources) that you can use to deploy services faster and more easily. You can use Helm on Linux, Mac and Windows devices.

    Jenkins X

    Jenkins X is an updated version of the well-known Jenkins. It builds on the existing features of Jenkins with capabilities tailored to Kubernetes and Docker workflows. It includes features for pre-built pipelines, integrated GitOps support, automatically generated preview environments and feedback integration.

    Jenkins X integrates with Tekton, Prow, Helm, Knative and Skaffold. You can use it on Linux, Mac or Windows machines.

    CI/CD and Automation Go Hand-in-Hand

    Agile teams need automation tooling, especially when creating and working with CI/CD pipelines. DevOps environments are dynamic, built for fluid workflows and constant change. Automation helps DevOps teams maintain a high level of visibility and reduce time spent on repetitive tasks. When adding AI and data science into the mix, DevOps teams gain the ability to make data-driven decisions, in addition to automated alerts and response.

  • How Log4j Becomes a Serious DevOps Problem

    How Log4j Becomes a Serious DevOps Problem

    The recent discovery of the Apache Log4j vulnerability has wide-ranging implications for anyone who develops software, especially for those in the DevOps realm. What’s most troubling about the vulnerability (CVE-2021-44228) is how prevalent the use of Log4j is. The vulnerability is reported in a vast array of applications and directly impacts numerous Apache projects, including Druid, Dubbo, Flink, Flume, Hadoop, Kafka, Solr, Spark and Struts. There are also numerous Cisco programs impacted by the vulnerability. The list of impacted applications and libraries seems to be almost infinite, extending to Elastic LogStash, GrayLog2, Neo4J, Steam, Twitter and even the VMware Tanzu family of programs. 

    Simply put, any application that uses Log4J for logging and tracing is subject to the identified family of attacks known as Log4Shell. Those attacks prove easy to exploit and can be used to take control of vulnerable servers.

    Why DevOps Practitioners Should be Concerned

    The world of DevOps is all about reiteration, where features and improvements to applications are slipstreamed into production environments. To accelerate development, the QA process is usually limited to the most recent changes.

    In other words, developers, via their pipelines, can often establish a baseline of quality and then speed up their CI/CD pipelines by testing only what has changed since the last iteration of the application. This process ensures an acceptable level of quality without introducing what are perceived to be unnecessary steps.

    However, those processes do not take into account newly discovered flaws in older APIs and libraries, meaning that those flaws could telegraph into deployed applications and leave DevOps teams completely unaware of the impact and possible disruption. Detecting flaws after an application is deployed is like putting the cart in front of the horse.

    Further complicating the situation is that many developers lack a well-defined and well-maintained software bill of materials (SBOM), which means developers may be unaware of the components and/or libraries being used in a deployed application.

    Without an SBOM, it becomes increasingly difficult to remediate an issue with an application—especially if developers are unaware of the issue to begin with.

    Where Should Responsibility Lie?

    More often than not, vulnerabilities introduced into applications become a game of finger-pointing, where developers blame library suppliers or infosec staffers, and infosec blames developers for not performing due diligence on the components they are selecting to incorporate into their applications.

    Of course, finger-pointing only adds to the angst and delays the process of remediation and does nothing to determine where the responsibility for detecting and remediating vulnerabilities lies. DevOps teams need to put these squabbles aside and establish a process that deals with unintentionally introduced vulnerabilities. What’s more, DevOps teams need to invest in DevSecOps to make sure that vulnerabilities are detected and remediated as quickly as possible.

    To avoid the potentially massive disruption that a vulnerability like Log4Shell can cause, those running DevOps should consider:

    • Creating and maintaining an SBOM 
    • Deploying automated tools that scan for discovered vulnerabilities
    • Automatically comparing the SBOM against vulnerabilities
    • Including DevSecOps processes within their CI/CD pipelines
    • Frequently testing applications for flaws
    • Coordinating with cybersecurity personnel when a flaw is discovered
    • Patching frequently

    With more and more vulnerabilities discovered on an almost daily basis, DevOps teams must be ever vigilant about what components are incorporated into their applications. What’s more, with the increased use of low-code/no-code tools and integrated development environments (IDEs), DevOps will also need to consider if those tools use other libraries that can become compromised.

     

     

  • Red Hat Adds Service to Customize RHEL Images

    Red Hat Adds Service to Customize RHEL Images

    Red Hat this week announced a public beta of an Image Builder service that streamlines the process of building customized instances of the Red Hat Enterprise Linux (RHEL) operating system.

    Scott Herold, director of product management for RHEL at Red Hat, said the Image Builder service enables an IT team to define a “gold” template for each target deployment environment that only contains the RHEL code required. Initially, Image Builder is configured to default to build a minimal, recommended RHEL server image for Amazon EC2, Microsoft Azure and Google Cloud Platform (GCP) cloud services.

    Future updates to the service will add support for VMware vSphere platforms as well as downloadable images and installation media for physical servers, customized filesystem layouts and versions 9.0, 8.5, 8.4 and 7.9 of RHEL. The service itself is accessed via the Red Hat Hybrid Cloud Console that is currently being made available in beta.

    Herold said building or installing operating systems is often a slow, tedious and error-prone process that needs to be manually performed in a slightly different way for each platform. Each physical server or edge computing platform requires different configurations and workflows. As a result, the chances mistakes will be made is high; Image Builder can greatly reduce the possibility of mistakes because there is no guesswork required, noted Herold.

    IT teams can also opt to include additional content from Red Hat in those builds to further streamline deployment processes, noted Herold. Those images can then be automatically registered with Red Hat to ensure future updates are received.

    The ability to customize those images also serves to reduce the overall size of the codebase that needs to be secured, added Herold. At a time when there is increased focus on software supply chains in the wake of a series of high-profile security breaches, many IT teams are revisiting their software deployment processes, he noted.

    Ultimately, Herold said Red Hat is moving toward providing IT teams with a more cloud-like experience regardless of where they deploy Red Hat’s curated open source software. The goal is to reduce as much friction as possible so that IT teams can more easily manage infrastructure at scale, he added.

    One way or another, the entiree software deployment process is becoming more automated. Much of the drudgery that has historically made it challenging to deploy and manage IT platforms at scale can be traced back to manual processes created when organizations ran a much smaller number of applications in on-premises IT environments. As IT teams increasingly deploy more applications in the cloud and at the network edge, it’s become more apparent that manual legacy processes won’t scale—especially when it’s already challenging to hire and retain IT personnel.

    It’s not clear how quickly IT teams will modernize software deployment processes using automated DevOps processes. Simple inertia is often a powerful force to overcome; however, most IT professionals can easily find something more compelling to do with their time that doesn’t involve using manually customizing operating system images.

  • Log4j: It’s All About the Supply Chain, Baby!

    Log4j: It’s All About the Supply Chain, Baby!

    In 2021, the security story in DevOps and DevSecOps has been the supply chain. So, it’s only fitting that we are currently experiencing the mother of all supply chain issues with the Log4j Log4Shell RCE vulnerability to close out the year.

    I won’t waste your time rehashing what Log4j is, why it’s so dangerous and warning you to test all of your own and third-party apps to make sure you’re insulated against this. I will say, though, that I don’t think Log4j is a bug—I think it is a defect in design. Worse, as we are seeing, there are multiple defects in design. All of you Java haters out there—back in the box. Java is no worse or better than any other language that we regularly use to build applications.

    So, what exactly is so special about supply chain security? Why do DevSecOps teams, developers and our security incident management teams have to interpret this as a warning and stay at the top of their game from now on? It’s simple. In the age of SaaSOps; in a world where we Frankenstein applications by cobbling and stitching together numerous third-party components (some open source, some not), we are increasingly reliant on code, even in our own applications, that we did not write. We probably didn’t test it, either. We rely more than ever on the “community” and on unknown third-party suppliers. It has always been a leap of faith with these apps and components, and this year some of those chickens have come home to roost.

    The White House executive order on improving the nation’s cybersecurity calls for a software bill of materials (SBOM) for applications; that is, as they say at the tables in Las Vegas, a fine start, but we need more. If we really had a BOM included with the apps we use (and that includes SaaS apps), how many of you are actually going to read it and act on that information? And besides including all these components in a BOM, what further obligations do suppliers have to ensure that they are replacing and updating vulnerable code?

    This issue becomes increasingly acute with open source. My partner and colleague Mitchell Ashley recently asked if too much open source was a bad thing. I don’t know if I would go that far; the more the better, as far as I am concerned. But we do need better monitoring and management of the open source code that’s out there. More testing before it goes GA, maybe? Yeah, the Cadillac open source projects might have the money and time to do that, maybe, but take Log4j as a perfect example: It’s two guys maintaining it in their free time and trying to do right by the community. How can they possibly keep up?

    I used to think that the answer was in the repos. My friends at Sonatype had a component firewall that would not let you download an old or vulnerable version of code without warning. That was fine if you were downloading it for the first time. But if it was already embedded—then what? Can we scan for older components in an app? My friends at JFrog can do something similar and more with Xray. And Sonatype’s Nexus platform also allows current and prospective users to scan older apps for vulnerabilities—including scanning for Log4j, absolutely free. But is that enough?

    All of these questions involve open source components. What about third-party SaaS apps? Often, we don’t even know what components or code are in those SaaS apps. Again, it is a leap of faith we are making when using them. 

    These supply chain security issues are a problem that we are just beginning to grasp. And while 2021 has been the year the stuff hit the fan and Log4j may be remembered as the biggest of the first, I, for one, am afraid to think about what else is coming. 

    I do know that the only way to tackle this and any future supply chain issues is if developers, DevSecOps, DevOps, SREs and our security incident response teams all work together. Just like most security issues, supply chain security is all of our responsibility.  I hope Log4j hasn’t bitten you too hard, and that you are on top of your exposure. But let’s start thinking about the next one and how we can better prepare for it.

  • Log4j: Is There Such a Thing as ‘Too Much’ Open Source?

    Log4j: Is There Such a Thing as ‘Too Much’ Open Source?

    The Log4j vulnerability got me thinking: Is there such a thing as too much open source?

    Before anyone immediately fires off a flaming email, rage tweet or scathing blog post, hear me out for a moment. If you know me, you know that I am an open source fanatic. I’ve been asked many times, “Should we use open source software and, if so, how much?” My response was (and still is) that open source software is a massive source of innovation and I can promise you, whether or not you use it, know that your competitors certainly are!

    I could go on and on extolling the virtues of open source and how it enabled the cloud, reshaped software architectures, created new product categories and gave us valuable tools like telemetry and increased visibility into cloud-native apps, infrastructure and beyond.

    When I first heard about the Log4j vulnerability on December 9, like many others, I quickly informed the tech leaders, engineers and SREs in my network about the serious potential risks such a zero-day vulnerability in the widely-used Log4j tool represented. In one way—probably the only way—the Log4j exploit shared a very important characteristic with the SolarWinds supply chain attack of late 2020: The widespread distribution and ubiquitous use of the software across applications, systems and organizations. Arguably, Log4j is used much more widely.

    This similarity raised questions I hadn’t fully considered in the context of such widely used software. Can we use too much open source software? Is Log4j too prevalent in its use and, therefore, a single point of failure? How bad would the Log4j situation be if a fix wasn’t immediately available or if one wasn’t forthcoming?

    The answer I kept coming back to is a resounding ‘no’. I still don’t think there’s such a thing as using too much (insert any open source project name) software. Log4j—and so many other open source software projects—are massively beneficial to us, and we virtually couldn’t function without these systems, tools and solutions in today’s age of ubiquitous software.

    What the current Log4j vulnerability, recent software supply chain attacks and other cybersecurity incidents with widespread impact highlight is our need to quickly respond to security incidents and take remediation steps anywhere they are found in the software stack. And that includes open source software. High-impact security vulnerabilities can occur anywhere across the numerous software stacks and services we use, though they have even greater impact when they occur in widely-used infrastructure, reusable components and microservices.

    In a world of infrastructure-as-code (IaC), it’s no longer a matter of patching a particular set of routers or servers. Unfortunately, much of the software stack is not containerized or built using smaller, easily replaceable containers and microservices—at least not yet. Our mindset and processes going forward from this incident must change, and we have to be prepared to respond to and remediate vulnerabilities both horizontally and vertically across the software stack. In certain circumstances, we have to do this on a large scale.

    As is common with active open source projects, Log4j contributors responded quickly with a readily available fix. The real answer to my original question above is that few are as prepared to respond to an important security vulnerability as those incredible, hard-working, active and engaged open source software developers.

  • The Graph Donates $48M to Advance GraphQL Platforms

    The Graph Donates $48M to Advance GraphQL Platforms

    The Graph, a provider of indexing and query tools based on the GraphQL language designed for blockchain platforms, today announced it has given $48 million to The Guild, a group of developers advancing an open source application programming interface (API) platform to further development of GraphQL-based technologies.

    Administered via The Graph Foundation, a nonprofit consortium that oversees the development of an open source graph database dubbed ONgDB, the agreement also calls for members of The Guild to contribute to the development of indexing and query tools being advanced by The Graph over the next four years.

    Eva Beylin, director of The Graph Foundation, said the goal is to fund the development of projects that leverage GraphQL, an open source data query and manipulation language for application programming interfaces (APIs) originally developed by Facebook, to create subgraphs used on blockchain platforms like Ethereum to index data in a way that is more accessible to developers.

    The open source community that is starting to evolve around blockchain platforms, also known as Web3, is now a major contributor to a range of GraphQL projects, noted Beylin. Those blockchain platforms are now being used to drive decentralized finance (DeFi) applications that do not rely on any single platform to process transactions. Those platforms are being employed by both startup and incumbent providers for financial services around the globe.

    It remains to be seen, however, how the GraphQL community will address security concerns. GraphQL APIs can easily suffer from broken object-level authentication issues just like any other API. The issue is a single GraphQL query could exfiltrate much more information than a typical API request. If the GraphQL provider does not have granular authorization checks for each particular method and resource, a GraphQL endpoint could become a major vulnerability.

    Cybercriminals also tend to launch brute-force attacks against APIs. With REST APIs, that malicious behavior is easier to catch with standard web application firewalls (WAFs) because it’s easier to notice hundreds of peculiar malformed requests. However, with GraphQL, a brute-force attack is harder to detect. It’s easier to limit this type of nefarious traffic with REST APIs by assigning access parameters to each resource and handling authorization checks one at a time.

    It’s not likely GraphQL APIs will replace REST APIs overnight. However, as the backend platforms for managing GraphQL APIs become more robust, the number of these APIs being used in a production environment by DevOps teams will dramatically increase—assuming there are no objections from cybersecurity teams.

    In the meantime, the open source community is clearly starting to rally around GraphQL, which should result in innovations applicable across a wide range of use cases. The challenge now is not only determining which type of API to employ and when, but also deciding whether any existing legacy APIs need to be replaced. One way or the other, however, the managing and securing of APIs—like it or not—is only going to become that much more challenging in 2022.

  • Log4j Puts Effective IT Operations at Center Stage

    Log4j Puts Effective IT Operations at Center Stage

    News of the Apache Log4j vulnerability exploit is striking fear into the hearts of both software makers and users. Log4j is the most popular Java logging service used today, with over 400,000 GitHub downloads and has been embedded in most internet services and products from companies all over the world, including Apple, Amazon, Cloudflare, Steam, Tesla, Twitter and Baidu, just to name a few.

    Exploiting this vulnerability is simple and allows threat actors to control Java-based web servers and launch remote code execution (RCE) attacks. New variations of the original exploit are being introduced rapidly—over 60 in less than 24 hours. The attack surface grows by the minute. Good credential management and advanced authentication will not help as this exploit is pre-authentication, which means an attacker doesn’t need to authenticate to your web applications to attack. Your web server is vulnerable.

    The good news is your organization likely has a team of seasoned IT operations pros that can help protect against the worst of Log4j’s devastation.

    How IT Operations Can Protect Against Log4j Vulnerabilities

    Step One: Find Log4j Wherever it Exists

    You need to investigate every internet-facing application, website and system that you own or use. This includes self-hosted installations of vendor products and cloud-based services. Focus on systems that are internet-facing that contain sensitive data. Once you’ve completed assessing your hosted apps and vendor systems, move on to endpoint applications. Java-based apps like WebEx, Minecraft, Citrix and many more.

    Step Two: Patch, Patch, Patch

    Patch in the same order as above: Internet-facing applications, websites and systems. If no patch is yet available for your apps and systems, uninstall.

    The pandemic has forced many endpoints into workers’ home environments, which creates extra challenges for patching those endpoints and keeping them compliant. Operations teams can’t rely on work-from-home folks to patch their own systems, even if they’re given clear instructions. Many of them will just ignore those directions, so you need to adopt systems that enforce continuous compliance.

    Step Three: IT Operations is Essential

    Communicate with your senior leaders. They should be in the know about this one! Senior leadership needs to position their IT operations teams among their most vital employees. Don’t think this is going to go away any time soon—we’re just starting to get a glimpse of what is being tried out there in the wild. Buckle up. 

     

  • U.S. Govt. CX EO | Mozilla Revenue | Log4j Latest

    U.S. Govt. CX EO | Mozilla Revenue | Log4j Latest

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: Improving U.S. government CX, how much money Mozilla makes, and the latest on the Log4j/Log4Shell débâcle.

    EO: “Transforming Federal Customer Experience”

    First up this week: The Biden–Harris administration makes an executive order directing its Federal agencies to make 36 specific customer experience (CX) improvement commitments. Much of this involves using technology to improve what DevOps teams would call UX—but government doesn’t like the language of “users.”

    Analysis: Customer-centricity FTW

    Remarkably, a lot of interaction with government still involves paper forms. But this EO shouldn’t merely be about putting existing processes online. Work also needs to be done to solve the underlying complexity and fragmentation of services: Focusing on CX is a helpful frame.

    Brett Samuels: Biden to sign order to streamline government services

    [The] executive order [is] intended to cut back on the bureaucracy around government services for the public such as renewing passports, applying for loans or changing names. [It] affects 36 “customer experience improvement commitments” across 17 federal agencies, [targeting] travel, retirement, business, health and updating personal information.
    …
    For example … to renew their passports online rather than dealing with print forms. [It] will create a single portal for … student loan debt, and small-business owners will have a more streamlined process for … loans, grants and certifications.
    …
    Survivors of natural disasters will also no longer be required to complete forms with several agencies when applying for assistance … and they will be able to file smartphone photos and use virtual inspections when filing claims. [That] could be critical as natural disasters become increasingly devastating, as evidenced by the tornadoes that ravaged Kentucky over the weekend, killing dozens of people and destroying communities.


    Is this a technology problem? Yes, in part—but it’s also a government problem, argues Jennifer Pahlka:

    Surely laws like the 21st Century IDEA Act … which required agencies to … “eliminate or consolidate websites or web pages that are duplicative or overlapping … and ensure that each website or digital service is designed around user needs,” made life easier for the customers of those agencies? What’s that you say? You’re not sure you’ve noticed that? Turns out you’re not alone.
    …
    An EO is one of the many mechanisms our government has at its disposal for what I like to call “magic words.” [But there’s a] gap between magic words and the changes they are supposed to compel. [Failures] happen at such a staggering scale they have become entirely expected.
    …
    We tend to assign the blame … to government technology [but] at some point, you can only improve the customer experience by addressing [underlying] complexity. … Most of the Kafka-esque nightmares people get stuck in happen because each service sees only the burden they impose, and not [what] the customer experiences.


    If, like me, you’re inherently cynical about government efficiency, you’d expect this effort to be a huge failure. But let’s not be too cynical: Warning of “the hazards of selection bias,” here’s ShanghaiBill:

    In the olden days, I took half a day off work to go to DMV and renew my DL. Now I can do it online in 5 minutes. … I pay my property tax online. I renew my business licenses online. I incorporated online.

    $500 Million per Year—But for How Long?

    The Mozilla Foundation has unveiled its revenue forecasts for this year. As in previous years, Mozilla makes the lion’s share of its cash from Google—by making it the default search engine. But there are new revenue sources.

    Analysis: And not a moment too soon

    The future of the browser market should be a strategic focus for DevOps teams. Mozilla’s Googley cash cow might not last. It’s often assumed the maker of Chrome keeps Mozilla alive so it can point to Firefox as “competition.” But with Microsoft’s browser Edging up the usage charts, that might no longer be true.

    Frederic Lardinois: Mozilla expects to generate more than $500M in revenue

    For 2021, the organization forecasts revenue of more than $500 million. What’s maybe most important, though, is that Mozilla’s new products like its Mozilla VPN service, Firefox Relay Premium, Pocket and other commercial initiatives are slowly but surely starting to pay off.
    …
    But for all intents and purposes, Mozilla remains fully dependent on Google for the time being. Diversifying its revenue sources is really the only way for Mozilla to decrease its reliance on a search deal with a company that is both a competitor, thanks to its dominant Chrome browser, and is increasingly out of alignment with Mozilla’s overall philosophy.
    …
    There is a window here for a non-Chrome browser, with users increasingly skeptical about Google’s motivations and Microsoft’s Edge team making a few missteps in recent months. Yet at the same time, Mozilla’s efforts to bring sponsored suggestions and ads to Firefox haven’t necessarily endeared the organization to its own users either.


    Why is Google funding its competitor? u/SerAaron:

    Google is Mozilla’s biggest donor. … It’s basically to avoid any monopoly allegations—so they can point to Firefox and say “look, we have competition”.

    Mozilla takes their money, sure, but … they don’t sell data, and they aren’t beholden to Google.


    But don’t count on that cash continuing to flow, says Maelwryth:

    That money is probably under threat now with Edge going for the market. Why give Mozilla hundreds of millions to show competition when you have actual competition from Microsoft?

    Log4j/Log4Shell Seems Worse Every Time We Look

    The list of popular DevOps tools vulnerable to Log4Shell grows by the hour. Merely finding your exploitable services can be an impossible task—let alone patching them.

    Analysis: Defense in depth is key

    Searching for and patching your vulnerable code is not enough. You’re also going to need to shore up your network defenses, by auditing internet-visible services and by tightening your application firewall rules. And—ironically—keep a close eye on your logs for signs of attempted exploits and IoCs.

    Steven J. Vaughan-Nichols: We Are in So Much Trouble

    Log4j is used a lot. And, when I say a lot, I mean it’s used in hundreds of applications [including] Druid, Dubbo, Flink, Flume, Hadoop, Kafka, Solr, Spark, Struts, Apple iCloud, … LogStash, GrayLog2, … Steam, Twitter, and … VMware.
    …
    Log4Shell is both easy to exploit and can be used to grab complete control of vulnerable servers. … Even as you’re reading this, odds are decent you’re being attacked. [But] even if you patch it in your systems, it’s all too likely to be hiding in other applications.
    …
    Log4Shell may be the worst security problem in a generation. … This is going to be a long, hard ride.


    Hyperbole? Even December 25 isn’t close enough for government work. Catalin Cimpanu:

    The US Cybersecurity and Infrastructure Security Agency has told federal civilian agencies to patch systems affected by the Log4Shell vulnerability by Christmas Eve. … Federal agencies have ten days … to test which of their internal apps and servers utilize the Log4j Java library, check if systems are vulnerable to the Log4Shell exploit, and patch affected servers.
    …
    Despite being a recent vulnerability, it is already considered one of the worst security flaws ever discovered, primarily due to its near-ubiquitous use among enterprise software makers, its simple and easy-to-use exploit, and the ability to hijack systems remotely. … Days after being disclosed, the vulnerability has been massively abused. … The first attacks were seen on December 1.


    There’s always another shoe. Here’s Teri Robinson:

    Maybe Log4j vulnerabilities are like rats—for every one that’s visible, multiple others scurry beneath the surface. … Apache said the fix addressing “CVE-2021-44228 in Apache Log4j 2.15.0 was incomplete in certain non-default configurations,” according to … CVE 2021-45046.

    The Moral of the Story: In the end, the love you take is equal to the love you make.


    You have been reading The Long View by Richi Jennings. You can contact him at @RiCHi or tlv@richi.uk.

    Image: Bernd Dittrich (via Unsplash)

  • Extending GitOps to the Enterprise

    Extending GitOps to the Enterprise

    GitOps is a relatively new term, but it continues to gain momentum as more organizations embrace the paradigm. In DevOps, GitOps lands on the software engineering side of the development and operations continuum. Even though GitOps is more likely to be adopted by smaller organizations, larger enterprises can successfully implement GitOps with just a few tweaks to the overarching framework.

    Let’s start at the beginning. Wouldn’t it be great if we could combine DevOps and GitOps approaches and use the Git distributed version control system as the ultimate source of truth? Then, when there’s a dispute over the correct state, people know where to go for the correct version.


     

    In GitOps, the system’s desired configuration is stored in a revision control system such as Git. Any difference between the desired state stored in Git and the system’s actual state indicates to DevOps teams that not all changes have been deployed. These changes can be reviewed and approved through standard revision control processes such as pull requests (PRs). When a PR is approved and merged to the main branch, an operator software process is responsible for changing the system’s current state to the desired state based on the configuration in Git.

    GitOps doesn’t require a particular set of tools, but the tools must:

    1. Operate according to the desired system state stored in Git.
    2. Detect differences between the desired and actual states.
    3. Perform required operations on the infrastructure to synchronize the actual and desired states. 

    In an ideal implementation of GitOps, manual changes to the system are not permitted and all changes to configuration must be made to files stored in Git. The infrastructure and operations engineers’ roles in a GitOps model shift from performing infrastructure changes and application deployments to developing and maintaining GitOps automation and helping teams review and approve changes through Git. 

    GitOps is great for non-critical environments, but enterprises are less likely to adopt GitOps due to a number of challenges facing GitOps users.

    Challenges with GitOps

    GitOps is applicable only to a subset of the software development life cycle (SDLC). This is important because GitOps tools are sometimes marketed as a one-size-fits-all solution that will solve all release problems—this is simply not true. First, GitOps requires that your deployment artifacts are already there. This means that tasks such as

    • Compiling code
    • Running unit/integration tests
    • Security scanning
    • Static analysis

    are not a concern of GitOps tools and are assumed to already be in place. GitOps also doesn’t address promotion of releases between environments.

    Auditability

    A default GitOps process is not fully auditable because of force push. Force push essentially allows users to remove any unwanted blocks of commits from Git history on the central repository.

    Business Approvals

    PRs are a typical approval model used in GitOps. Developers make changes, create PRs and then an approver may accept that PR. This would be recorded and a change would be deployed. A PR-based approval model assumes peers are already using Git and are familiar with PRs. 

    Generally this works great until we need to include business approvals. Business personnel may not be familiar with Git or PRs. If a team needs business approvals, they have to build a process on top of GitOps. This drastically increases lead time. 

    Governance 

    GitOps is dependent on a pure PR-approval based model, one in which the PR approver is responsible for reviews. After a PR is approved, there is no way to enforce enterprise policy rules. Enterprises need to control what end users can do on specific clusters to ensure those clusters are in compliance with policies to meet governance, legal requirements or enforce best practices.

    An Enterprise Approach to GitOps

    However, there are ways to solve this problem at the enterprise level.

    Alerting on Drift Detection 

    Enterprise customers get alerted whenever there is a drift in their repository and, based on that, customers can choose to approve or reject the drift. If rejected, the merged PR should be reverted to the previous state to ensure that Git is in sync with the desired state of the cluster.

    Use a Pipeline for Promotion of Releases Between Environments

    To promote an artifact between environments, users would have the ability to use a pipeline with deployment stages, performance monitoring and log analysis/approvals. 


     

    Let’s say the developer commits the source code and the GitOps deployment happens in the QA environment. The PR can be automatically approved upon meeting the policy standards set by the company. Then it can be pushed to a production environment.

    Governance 

    There are many reasons why developers and platform engineers integrate policy-as-code into their current GitOps processes. For one, this strategy helps to accelerate application development and deployment because it helps solve many of the change management hurdles that slow development pipelines.

    On the development side, policy-as-code helps app developers understand and abide by the company’s configuration, security and compliance policies. For example, a developer may not remember—or may have no reason to know—when deploying a load balancer onto Kubernetes in AWS is or is not sanctioned. Policy-as-code solves this problem automatically.

    The ability to run policies during GitOps deployments and adhere to those policies as a part of the deployment and upon any violations—alert, audit, notify and automatically roll back the changes from Git to the previous successful compliant state so that Git remains in sync with the cluster.

    GitOps + AIOps

    There is a notion that once GitOps is in place, everything will work seamlessly with no problems. Not so much; it is important to continually monitor deployment state to ensure everything stays as it should. That is where AIOps comes into picture.

    GitOps enables DevOps and development teams to see any manifest change, infrastructure change or artifact change. Post-deployment, the GitOps operator can instantly understand the changes in the service health, see what caused those changes and automate the process of investigating them using AIOps. If any severe anomalies are found, the operator can determine the current state of the system and, should anything differ, revert to the previous compliant Git commit. 

  • Red Hat Brings Ansible Automation to Microsoft Azure Cloud

    Red Hat Brings Ansible Automation to Microsoft Azure Cloud

    Red Hat this week announced that a private preview of the Red Hat Ansible Automation Platform is now available on the Microsoft Azure Cloud.

    Tom Anderson, vice president of Red Hat Ansible automation, said making the Red Hat instance of the open source automation framework available on Azure is the first step toward making it possible for IT teams to use the IT automation framework across multiple clouds.

    The goal is to make it simpler for IT teams to bridge multiple IT environments using a common framework to automatically configure everything from operating systems to security platforms across a hybrid cloud computing environment.

    Ansible is emerging as a de facto standard for automating IT environments largely because it provides an IT team with a declarative approach to automating IT processes. An IT team can write a playbook that describes their desired state for a platform without having to know anything about its current state.

    Interest in IT automation has soared as the number of workloads being deployed in the cloud and on-premises IT environments have steadily increased. The simple truth is that many organizations can’t keep pace with the rate at which applications are built and deployed because they are unable to hire and retain enough IT staff. The challenge is that writing a playbook to automate any IT process requires a fair amount of expertise that many organizations lack. There are, however, now large numbers of existing templates that organizations can reuse.

    Anderson also noted that security concerns will also force the IT automation issue. Many of the application workloads running in cloud computing environments are misconfigured because they were provisioned by developers that have little to no security expertise. A playbook greatly reduces the chances an application workload will be misconfigured in the cloud without having to turn every developer into a security expert, Anderson explained.

    Ultimately, IT automation frameworks reduce the level of friction that often exists between developers and the rest of the IT operations team in a way that actually serves to increase developer productivity, added Anderson.

    Regardless of how IT organizations automate IT processes, it’s now a question of the rate at which that goal will be achieved rather than if. DevOps teams, of course, have always been at the forefront of automation, but now it’s clear that as IT environments become more complex, IT organizations are embracing automation more aggressively. Most of those initial efforts are focused on repetitive, tedious and manual IT tasks, but the potential to automate IT processes at levels of unprecedented scale has never been greater.

    In fact, many organizations may soon discover it’s difficult to retain IT professionals without investing in automation. After all, the most talented IT professionals will not want to work for an organization that requires them to always manually configure IT resources. Many IT teams might also discover that automation frameworks like Ansible are the gateway to embracing additional DevOps best practices. The challenge, of course, is finding the time to get started automating processes when there are so many immediate issues that IT teams can’t ignore.