devopsy.kinsta.cloud

Tag: continuous delivery

  • Modernizing Continuous Integration

    Modernizing Continuous Integration

    In the duality of continuous integration/continuous delivery (CI/CD), CI focuses on build automation. Leveraging CI practices for software delivery is normal for most organizations and can be seen as a solved problem. However, the demand for increasingly distributed applications has risen with the expansion of microservices, and development teams have the expectation that every commit should be a build.

    Continuous integration has, surprisingly, become an unexpected bottleneck with these expectations. To begin modernizing CI, it’s important to first understand some of the challenges. Let’s dig in.

    Challenges of Continuous Integration

    Given that build and release candidates follow advancements in development technology closely — such as new languages, packaging and paradigms in testing the artifact — expanding the capabilities in continuous integration implementations can be challenging. With the introduction of containerized technology, the firepower and velocity required to build greatly increased.

    Scaling Continuous Integration Platforms

    The infrastructure required to run a distributed CI platform can be as complex as the applications they are building because of the heavy compute requirements. Consider how much a local machine’s resources are tied up during a local build and test cycle. Now, multiply that by the number of folks on a team or in an organization and you start to understand just how much compute power is being used. Distributed build runners are just one area that can be complex, and managing when new build nodes are spun up and spun down can depend on the platform/end user.

    Keeping Up With the Technology Velocity

    The adage of “the only constant in technology is change” is true. New languages, platforms and paradigms are to be expected as technology pushes forward. The ease of including new technologies in a heterogeneous build or accepting new testing paradigms, however, can be difficult for more rigid CI platforms. These legacy platforms were designed for only a small subset of technologies. With legacy or rigid approaches, the dependency management required to maintain technical velocity becomes a significant burden.

    What it Takes to Modernize Continuous Integration

    The simplest way to change continuous integration practices and platforms is to take what I call the four-pillar approach: platform infrastructure modernization, engineering efficiency strategy, test optimization and rapid pipeline development. Making strides in any of these pillars will put you on the path toward modernizing your continuous integration platforms and practices. Making strides in all of the pillars will set up your DevOps teams for significant success.

    Platform Infrastructure Modernization

    The location where the actual builds and packaging take place (e.g. build nodes/runners) do most of the heavy lifting and have a fairly elastic workload nature. Builds (e.g. a JAVA JAR build) and packaging (e.g. a Docker Image Compose) are compute-heavy tasks. Once the build and packaging are complete, the runners can sit idle. This shows the importance of having an ephemeral build node. The build node spins up during the task, then spins down or is destroyed after the build task is done, so as to not drain resources.

    Engineering Efficiency Strategy

    A core tenet of engineering efficiency is meeting your internal customers where they are. For software engineers, this means being as close to their tools and projects as possible. Like many modern pieces of application infrastructure, shifting left to the developer means being included in the project structure in source code management (SCM).

    A common disconnect in CI platforms is dependency management. Over the past decade, for software engineers, this problem has been solved with package or build tools such as Maven, Gradle and NPM. Engineers need to simply define implicitly or explicitly what they need, and the dependencies will be resolved. Continuous integration tools can suffer from a disconnect, since we are leveraging several tools that potentially don’t have a common syntax. So, again, it’s important developers have close oversight on all the tools they need to complete their projects to reduce this friction.

    Test Optimization

    While initial confidence in getting tests into the CI pipeline expands, more tests — and sometimes inappropriate test coverage — is introduced by the ease of integration with the pipeline. An even harder problem to identify and rectify is flaky tests. A flaky test is one that both passes and fails periodically without any code changes. This becomes a twofold problem: increasing execution time and lack of confidence with flakiness. The most logical solution to avoid these issues is adding test optimization. A modern continuous integration solution should be able to visualize pipeline order, timings and overall execution to help identity (and eventually rectify) excessive coverage and flakiness.

    Rapid Pipeline Development

    Software is an exercise in iteration. The lower the barriers to entry for iteration to occur, the more gains in engineering efficiency and agility are achieved. Local builds happen dozens of times before reaching a committable stage, moving them forward to a dev-integration environment. It’s here where having a local environment is key. Oddly, most continuous integration pipelines are designed to run externally from a local machine. So, the ability to move to a local CI pipeline helps build confidence that the externally run pipeline will be successful.

    Continue on Your Continuous Integration Journey

    Continuous integration might seem like a solved problem for many organizations, but as with any technology, there is always room for improvement and modernization. Like any engineering efficiency paradigm, CI is a journey and the destination ever-evolving.

  • How to Move Fast Without Breaking Security

    How to Move Fast Without Breaking Security

    “Move fast and break things,” is a familiar motto. The phrase, attributed to Facebook CEO Mark Zuckerberg, helps to explain the company’s stellar growth over the past decade, driven by its product innovations. However, while it’s a useful philosophy for software development, moving faster than you’d planned is a risky approach in other areas, as organizations globally realized during the COVID-19 pandemic. While 2020 saw digital transformation program timelines advance by up to seven years, enterprises’ quick moves to the cloud also meant that some things got damaged along the way – including security.

    Our recent survey conducted with the Cloud Security Alliance showed that more than half of organizations are now running over 41% of their workloads in public clouds, compared to just one quarter in 2019, and this will increase further by the end of 2021. Enterprises are moving fast to the cloud, but they are also finding that things are getting broken during this process.

    The research showed 11% of organizations reported a cloud security incident in the past year, with the three most common causes being cloud provider issues (26%), security misconfigurations (22%) and attacks such as denial of service exploits (20%). In terms of the business impact of these disruptive cloud outages, 24% said it took up to three hours to restore operations, and for 26% it took over half a day.

    As a result, it’s no surprise that organizations have significant concerns about enforcing and managing security in the cloud. Their leading concerns were maintaining overall network security, a lack of cloud expertise, problems when migrating workloads to the cloud and insufficient staff to manage their expanded cloud environments. So, what are the root causes of these cloud security concerns and challenges, and how should enterprises address them?

    Confusion Over Cloud Control

    When asked about which aspects of security worried them most when running applications in public clouds, respondents overwhelmingly cited getting clear visibility of topologies and policies for the entire hybrid network estate, followed by the ability to detect risks and misconfigurations.

    A key reason for these concerns is that organizations are using a range of different controls to manage cloud security as part of their application orchestration; 52% use cloud-native tools and 50% reported using orchestration and configuration management tools such as Ansible, Chef and Puppet. However, nearly a third (29%) said they use manual processes to manage cloud security.

    In addition, there’s competition for overall control over cloud security: 35% of respondents said their security operations team managed cloud security, followed by the cloud team (18%) and IT operations (16%). Other teams, such as network operations, DevOps and application owners, all figured in, too. Having different teams using multiple different controls for security limits overall visibility across the hybrid cloud environment, and also adds significant complexity and management overhead to security processes. Any time you need to make a change, you need to duplicate the work across each of these different controls and teams. This results in security holes and the types of misconfiguration-based incidents and outages we mentioned earlier.

    How to Move Fast and Not Break Things

    So how can organizations address these security and management issues, and get consistent control over their cloud and on-premises environments, so they can take full advantage of cloud agility and scalability without compromising on security? Here are the four key steps:

    Get holistic visibility across your networks: a lack of visibility across cloud estates was the main security challenge for survey respondents. Organizations need to get holistic visibility across all of their different cloud accounts, including the security controls that are being used both in the cloud and to manage traffic to and from the cloud environments. This enables IT and security teams to quickly identify any potential security issues or risks that could lead to outages.

    One management console to rule them all: When using a mix of the cloud providers’ security controls as well as other controls, both in the cloud and on-premises, it’s a huge challenge for organizations to manage policies consistently. If all these diverse security controls are managed from a single console using a single set of commands and syntax, security policies can be applied consistently. This avoids duplication of effort and the error-prone manual processes that lead to misconfigurations and outages. It also provides a full audit trail of every change.

    Automate security and compliance processes for speed: Automation is essential to managing cloud environments efficiently while orchestrating changes across a complex array of security controls. It brings speed and accuracy to managing security changes across cloud environments. It also accelerates audit preparation and helps ensure continuous compliance. Automation also helps organizations overcome skills gaps and staffing limitations, which were cited as a challenge by over three-quarters of our survey respondents.

    Accelerate incident response, anywhere: Our survey also showed organizations are concerned about overall network security in the cloud. Security management solutions that integrate with SIEM or SOAR tools can help to address these concerns by accelerating incident response. As soon as a threat is detected by the SIEM or SOAR product, the management solution can identify all the applications and servers that the threat could impact. The solution can then mitigate the threat’s risk by automatically isolating any affected servers or devices from the network – whether in the cloud or on-premises – preventing lateral movement.

    With a network security automation solution handling these steps, organizations can get holistic, single-console security management across all of their public cloud accounts, as well as their private cloud and on-premises deployments. This helps them to solve the cloud complexity challenge and ensures faster, safer and more compliant cloud management – making it possible for organizations to move fast in response to changing business needs without breaking things.

  • The Surprising Secret to DevOps ROI

    The Surprising Secret to DevOps ROI

    As a result of my many years of DevOps consulting, I have discovered a simple, but not well-understood, secret ingredient that drives ROI for all the Three Ways of DevOps.

    Measuring and optimizing return on investment (ROI) for DevOps is not well-understood. It is a proven fact that DevOps, when engineered well, delivers tremendous value to organizations by way of continuous delivery practices. Reduced lead time, more frequent releases, lower failure rates and faster time to restore services are well-accepted indicators of DevOps value. It is not a secret that the combination of engineering DevOps practices for people, processes, and technologies around the Three Ways of DevOps – continuous flow, continuous feedback and continuous improvement – gets results. As valuable as these things are, they are not a measure of ROI, and do not tell you what to do to improve the return on your DevOps investments.

    How Do You Calculate and Optimize DevOps ROI?

    First, it is important to understand that ROI for DevOps is best measured using a simple ratio of return (R) divided by investment (I). (OK – for the accounting purists out there; I know the real ROI formula is actually (((R-I)/I)*100%) but that is a technicality that we don’t need for this argument.)

    The “return” of DevOps value is the important factor in the numerator of the ROI ratio. While there are many values that can be attributed to DevOps, the most direct value that can be measured and justified for legitimate comparison purposes is the cost savings associated with accelerating software delivery value streams. Application developers, and others, can get more applications released faster because of DevOps, which has a dramatic effect on valuable cost reductions. The “investment” in DevOps projects is the denominator of the ROI ratio. To optimize ROI, simply increase return by getting more application releases out faster, while reducing the investment required for a DevOps project. Sounds easy enough. But wait! How do you do that?

    The primary factor in the return is derived from reducing bottlenecks. I have found, in more than half of the thousands of DevOps value streams I’ve studied, that the biggest bottleneck is testing! It is a myth that continuous testing is just the execution of automated tests. I am talking about the whole range of ‘testing things,’ including test strategy, test planning, test resource scheduling, test design, test implementation, test environment management, test execution, test results reporting, test result analysis, troubleshooting, fix, retest and test management. That is a lot! In fact, testing things are a source of bottlenecks for every stage of a DevOps pipeline, for all three ways of DevOps. The point is that ROI can be greatly improved by improving testing.

    As it turns out, the primary factor in reducing investment costs also has to do with testing! Putting ‘testing things’ in place is expensive in terms of labor and capital costs for both people and systems. The primary factor to reduce total investment cost is to standardize, share and use ‘testing things’ with as many application pipelines and stages as possible.

    What is the secret ingredient that I have discovered? After all, the importance of testing itself is not much of a secret. I have isolated the key ingredient that is the heart of all ‘testing things.’ The purpose of a test is to determine a verdict. A test verdict is the codified representation of a pass, fail or inconclusive result state of a test, assessed relative to the purpose of the test. Test verdicts are essential, no matter what kind of test it is; no matter the stage in the pipeline a test is used. Without a verdict, all the ‘testing things’ in the world are not valuable.

    So – the secret ingredient for optimizing DevOps ROI is to optimize test verdicts. Those that know how to optimize test verdicts hold the key to optimizing DevOps ROI.

    As I indicated in this eBook I authored for the DevOps Institute, verdicts from properly engineered testing at each stage in the pipeline are essential for assessing the artifacts produced in each stage.

    How Can You Optimize Test Verdicts?

    As I have pointed out in prior articles, well-engineered DevOps requires balancing the nine pillars of DevOps.
    Surprisingly, test verdicts play a crucial role in optimizing all nine pillars of DevOps, and are vital to accomplishing the best return for the least cost investment in DevOps. Here are some examples of how this works for each pillar:

    Leadership – Leaders tend to underestimate the importance that ‘testing things’ have on DevOps ROI, and that the central ingredient of test effectiveness is the optimization of test verdicts. By sponsoring DevOps projects and training programs that drive improved test verdicts, leaders will have a direct and lasting impact on ROI.

    Collaborative Culture – How often have you seen people wasting time arguing over the meaning and importance of a test? Collaboration between application developers and QA and Ops teams is more effective when test verdicts are clear and based on a common understanding of the purpose and importance of each test.

    Design – The creation of excellent tests, designed to efficiently resolve to clear verdicts during the design phase, makes for reusable test assets in all subsequent stages and releases, and can be reused with other applications.

    Continuous Integration – The validation of builds is faster when all integration tests are arranged to get clear verdicts quickly during the build process.

    Continuous Testing – Test verdicts associated with each test are the key information ingredient that test management systems can key off to optimize test selection and test execution for each stage in the continuous delivery pipeline.

    Elastic Infrastructure – Test environment management (TEM) systems can optimize the orchestration of test resources better when test verdicts are clearly understood. Knowing a history of verdicts for a group of tests indicate which resources and how much of each resource are needed to run each test. This knowledge is essential to drive test environment management systems that have scaling algorithms for optimizing the use of ephemeral test environments, and to ensure test resources are used optimally.

    Continuous Monitoring – Reporting and analysis of test results can be a major bottleneck in the pipeline, but can be dramatically accelerated when the test verdicts are clearly defined.

    Continuous Security – Security tests are not much use without a clear verdict. While there is no such thing as 100% security assurance, the aggregate set of security test verdicts helps to profile security for applications and releases.

    Continuous Delivery – Algorithms that govern the ‘doneness’ of an application release candidate, and decisions to promote a release to production, depend on analysis of service level objectives (SLOs) derived from service level indicators (SLIs), including test verdicts. The fidelity and quality of those verdicts is essential for operations and site reliability engineering (SRE) teams to be confident enough to not bottleneck an application release.

    What This Means

    Measuring and optimizing DevOps ROI requires strategies that deliver value by improving returns through reducing bottlenecks and reducing investment costs. ‘Testing things’ are a dominant factor contributing dramatically to both the return and investment components of the ROI equation. A comprehensive strategy for optimizing test verdicts is the secret ingredient for optimizing and leveraging investment in ‘testing things,’ which, in turn, has a dramatic effect on improving DevOps ROI.

  • On-Premises Software Delivery Tool: Build or Buy?

    On-Premises Software Delivery Tool: Build or Buy?

    The on-premises software market is four times larger than the SaaS market, but harder to reach. Every software vendor who wants to pursue it has to decide how they are going to package and deliver applications inside customers’ firewalls, automating away the complexity of keeping applications deployed and running. Should they build or buy a deployment tool (and how should they think about the features they’ll need beyond just deployment)? Both have their pros and cons, but as demand to reach the larger enterprise market grows, more software vendors are choosing to buy.

    Deploying apps on-premises used to be very different. Fifteen years ago, sysadmins and database administrators (DBAs) would manually configure servers and the applications that ran on them. When something broke in production, admins would often troubleshoot and fix by SSHing in to tweak a config file or setting. But now, particularly within agile development organizations, everything is done via automation (i.e., the evolution of DevOps and SRE). As a result, customers have lost patience with 150-page installation manuals and bespoke, hand-written scripts.

    Today, any ISV looking to modernize an existing on-premises app (or SaaS-only company considering creating their first on-premises offering) is, most likely, going to default to using Kubernetes because it abstracts away the technical overhead of keeping production-grade applications operating.

    With that in mind, software vendors should consider seven areas when deciding whether to build or buy a delivery and management solution. These include:

    Maintenance

    There are free or low cost resources if a software vendor wants to go it alone. Kubernetes and cloud-native technologies let capable engineers stand up a solution in months. But software vendors often overlook the long-term cost of keeping the tool up and running. There is a lot of tooling “around Kubernetes” the vendor has to build to successfully operationalize this. Sure, if you’re going to deploy to up to five customers, you don’t need formal tooling and processes that make it easy to engage whole teams of people (favoring a small build and manual effort), but when you want to go from five to 50, or 50 to 500 customers, you need advanced tooling (favoring buy).

    Cost

    Purchasing a ready-made solution comes with up-front costs, but building and maintaining a bespoke solution can be much more expensive. The obvious costs of building a deployment tool are design, engineering, support and operational costs. There are larger, long-term costs to consider, most notably pulling away one of your greatest assets, software engineers, to spend months (or years) on ongoing support needs. Redeploying your engineers results in opportunity and productivity costs, such as neglecting your core application(s) and stalling your business growth.

    Internal Resources

    You also need to evaluate your internal resources. Building a bespoke deployment tool is a commitment because the tool becomes a product in and of itself. Do you have a team with the available time and the technical skills? If your team isn’t large enough, you may incur costs for expanded headcount or contractors. If you don’t have added FTE costs, you may have a skills gap. When developing a custom deployment tool, most teams focus on the features that enable software distribution and installation, but they don’t consider management and troubleshooting features. In these situations, the FTE and skillset costs may be higher than paying for a comprehensive solution.

    Business Continuity

    Most homegrown solutions are created by an engineer or two who become experts in the systems they’re developing. This makes tons of sense for core business logic, apps that are constantly going to be improved and onto which new members will be onboarded. However, for ancillary systems that support the core business, these applications end up becoming a silo of knowledge. The question isn’t, “How much time will this take me to maintain after I build it?” but rather, “Who will maintain this after me, and who after them?” followed by, “How will they learn the intricacies of this system?” If the answer(s) to this question isn’t clear, this approach should be reconsidered.

    Beyond The First Team

    If you expect that your team will be the first, or one of the first, teams within your company to face the problem you are solving for (but you are certain others are sure to follow), then this decision is even more complex. A common customer delivery and management platform can increase cross-sell by simplifying the marginal effort for your customers to deploy a second, third, fourth, etc. application from your company. Buying will likely require a long-term relationship with a vendor dedicated to solving the problem. Similarly, building a solution will require extra investments. Beyond the initial technical/product requirements of abstracting a home grown solution to meet the needs of additional teams/architectures/products, there are also the other implications. The winning solution will win because it will evolve to meet the needs of multiple teams and have strong internal advocacy. Depending on the size and complexity of the company, this internal advocacy can be quite time consuming and require more than just engineering talents (generally, this is what customer success and account managers do 100% of the time).

    Target Environments

    Your application may need to be deployed in varied on-premises environments and may not have the time to adequately address all of them. If your team has not previously built an installer, they may not have the skillset to build out richer features in AWS, Azure, VMware and others, to solidify the customer experience and day 2 operations, beyond just bringing the application up. An AWS Cloud Formation template might get you out the door, but when the next customer says, “I need GCP,” you’d have to start over.

    It’s rinse-and-repeat for each of the target environments, which slows your time-to-market, potentially losing high-value sales. Conversely, if you look at ready-made solutions, you can select the best out-of-the-box experience for your customers.

    Control and Compatibility

    There are situations where your team must build the on-premises deployment tool themselves. For example, when the app requires so much last-mile configuration for each customer deployment that there just isn’t a way to standardize it. Or if you don’t have enough customers to justify paying for a commercial product, it might make more sense to do a quick sprint and hack something out.

    An off-the-shelf solution can significantly accelerate your time to market, but you have to order off the menu. Also, if you have access to your customer’s infrastructure, then building your own tool may make more sense. On the other hand, requirements such as faster time-to-market, an off-the-shelf comprehensive solution and lower costs over time may outweigh customization. If speed, lower costs and quality are important, a solution from a focused vendor who has thought through all of the technical and operational challenges, with no surprises or last-minute fixes, is the way to go.

    Competitive Advantage

    Your software engineering team may be your company’s greatest resource. But it’s far better to keep your engineers focused on innovation, adding value to your application and winning more customers through improved software. Coding a deployment tool is undifferentiated overhead. It does not make your business more successful. Installers and support tooling are table stakes to deliver a good customer experience, but the reality is that no customer is going to pay you more money for that capability.

    The Bottom Line

    Simplified access to the on-premises software market is one of the greatest opportunities for software vendors today. Kubernetes and cloud-native technologies make it easier to develop tools to package, deploy, manage and troubleshoot apps. But the job isn’t for everyone. The costs, both up-front and long-term, favor ready-made commercial applications for most software vendors. Consider your resources and customer needs to decide if it makes more sense to build or buy your solution.

  • Survey: The State of Database DevOps

    Survey: The State of Database DevOps

    Enterprises are undergoing rapid changes. Throughout the past year, companies have been pressed to learn a diversified set of technologies and methods to remain agile in a changing economy. In this digital-first world, data plays a heightened role; in tandem, so does the delivery of data and the persistence of the database layer itself.

    Applying DevOps principles and automation throughout the software life cycle is critical to maintaining a high level of efficiency. As with application development, databases are not immune to the DevOps takeover. Database DevOps is a discipline that considers the processes used to automate database changes and deployments — a field growing in mindshare as databases diversify.

    Seventy percent of companies now use more than one database in their stack. Many organizations now use a blend of on-premises and cloud databases. Styles are diverging, too, as non-relational databases like NoSQL gain favor, and new open-source databases emerge to fit niche circumstances. All of this means database operators may require an increased emphasis on database DevOps to handle additional complexity.

    The 2021 State of Database DevOps report, conducted by Redgate, surveyed 3,200 database professionals on the general state of their DevOps adoption. The survey also inquired about database change automation and deployment and underlying database technologies. Now, five years in, the study continues to reveal shifts in practices across industries.

    Below, we’ll review the report’s main takeaways to see how companies implement database DevOps in practice.

    DevOps Adoption Increases Across Projects

    To meet pressing digital innovation needs, DevOps has surged in importance across all industries — 74% of organizations have adopted some form of DevOps, and 26% of respondents reported applying DevOps practices across all projects — a significant uptick from previous years. The report acknowledges a slight increase in DevOps adoption among larger company sizes.

    So, which DevOps practices are companies employing the most? Well, in terms of DevOps for application development, the majority of organizations (50% or above) have adopted version control, automated builds, monitoring, automated unit tests, static code analysis and semi-automated deployments in production. Only 41% of organizations surveyed have fully automated deployments to production. These methods can be viewed as a basic benchmark for modern DevOps.

    The study demonstrated that in most organizations, DevOps is a communal undertaking. More respondents reported adopting a community of practice, rather than relying on individual decision makers, to disseminate DevOps strategies and direct planning. According to the report, a community of practice is a group of people that “makes decisions, provides recommendations, or is consulted.”

    Lack of Skills Tops DevOps Obstacles

    The pressure is on engineers to produce to meet digital transformation demands. However, organizations experiencing a developer drought may not possess the talent to introduce DevOps practices. It’s no surprise that the common obstacle hindering DevOps adoption was a “lack of appropriate skills in the team.”

    Other cultural barriers to DevOps adoption include disruption of existing workflows and a lack of alignment between teams. Interestingly, a lack of executive support was cited as the lowest hurdle to DevOps adoption, indicating universal consensus on DevOps goals. If we consider what technical barriers prevent improved software delivery and operations, 41% noted insufficient automation in release processes and 39% expressed hindrances due to legacy code.

    Continuous Delivery For Databases Increases

    Database DevOps appears to be growing in production use too. For the purpose of this study, Database DevOps is defined as “automating the delivery of database changes.” According to the report, 51% of respondents are already automating some parts of their database deployment process.

    Though growing year after year, database DevOps is nowhere near ubiquitous. Compared to DevOps in application development, database DevOps has lower overall use. For example, only 36% are implementing automated builds. And, only 22% adopt continuous delivery (CD) for all databases. Nevertheless, this could quickly change, though, as 80% of respondents are working to implement CD for database changes or have already implemented it.

    Setting up a CD process for database changes could increase overall software delivery speed. The top two drivers for adopting database automation are increasing the speed of database changes and freeing up time for other work. The third top reason, minimizing application downtime, rose in rank this year — which makes sense, as the desire to create more reliable systems is an ever-present concern.

    However, to reap these rewards, organizations must overcome some significant challenges, like synchronizing application and database changes and maintaining high speeds for database automation, among other concerns.

    Organizations Adopt Hybridized Database Approaches

    Companies increasingly use a mix of different database technologies and styles. We also see an increase in cloud and hybrid database estates.

    In terms of database styles, relational databases still make up the majority of market share — 98% of respondents have at least one relational database. However, non-relational databases, like NoSQL, are gaining popularity. Forty-seven percent of database professionals have a NoSQL database in their enterprise. Interestingly, only 2% use non-relational databases exclusively, indicating it’s a niche use case.

    Enterprises are now using more databases than ever before. Seventy percent of database professionals reported having more than one database in their stack. This figure rose nearly 10% since last year. Database types are multiplying too — 45% have more than one type of database. In terms of database management systems, Microsoft SQL Server holds the most use, at an 82% usage rate. Others like MySQL, Oracle, PostgreSQL, and MongoDB trail behind.

    Enterprises are also increasing their use of cloud and hybrid database deployment models. Eighty percent of respondents host a cloud database, and 65% report a mix of cloud and on-premises estate.

    One possible explanation for the diversified database effect is an increase in public cloud provider usage, combined with the rise of multicloud strategies. Database options are increasing as well — many free, open-source databases can cater to unique project needs.

    This “polygot persistence” could also simply be a natural occurrence within large enterprises supporting many application nuances. As Martin Fowler describes, “… any decent sized enterprise will have a variety of different data storage technologies for different kinds of data.”

    Quality Database Ops = Quality Software Performance

    In summary, non-relational databases are gaining steam, cloud and hybrid database models are becoming more common, and large organizations are increasingly using multiple database types. The landscape is forecasted to diversify further, meaning database DevOps is stepping up to manage this complexity.

    The report indicates a correlation between quality database Ops and overall software quality, noting that organizations that release more frequently typically deploy database changes in much less time. To achieve these goals, “… the next area for database DevOps is in the data engineering and analytics space,” writes Pramod Sadalage, director, data & DevOps, ThoughtWorks.

    While it’s important to take a vendor-endorsed study with a grain of salt, it’s safe to say we will likely see more interest in database DevOps, especially as DevOps themes continue to permeate through the entire spectrum of software development and delivery.

    For more information, you can download The State of Database DevOps report behind an email gate here.

  • Armory Offers Framework to Extend Spinnaker

    Armory Offers Framework to Extend Spinnaker

    Armory, provider of a curated instance of open source Spinnaker continuous delivery (CD) software, is making generally available an open source plugin framework for any instance of Spinnaker.

    Announced this week at its online OPEN CORE SUMMIT, the framework has already been employed to create 15 plugins for Spinnaker, including an observability tool from Armory and a provisioning tool from Pulumi. The plugin framework creates extension points that provide standardized interfaces for creating plugins that invoke various microservice within the Spinnaker CD platform.

    Armory CTO Isaac Mosquera said the goal is to make a Spinnaker platform now being developed under the auspices of the Continuous Delivery (CD) Foundation more easily extensible. The relationship between the frameworks and Spinnaker is naturally symbiotic; more extensions are needed to increase the appeal of the platform. Organizations are unlikely to build those extensions unless Spinnaker achieves a critical mass of adoption.

    Originally developed by Netflix, Spinnaker is being advanced by CD Foundation, along with the Jenkins and Jenkins X continuous integration/continuous delivery (CI/CD) platforms. Organizations will employ different platforms under different circumstances, but the Spinnaker platform does enjoy the capability of being integrated with multiple CI platforms that DevOps teams already have in place, noted Mosquera.

    That ability to mix and match DevOps platforms is becoming more critical as IT environments become more hybrid, he said. The expectation is support for Tekton pipelines, which were originally developed by Google and now are part of the CD Foundation portfolio, will be added to various CI/CD platforms to make it easier to integrate DevOps pipelines.

    In the meantime, a debate surrounding how to implement best GitOps practices is starting to swirl. Thanks to the rise of Kubernetes, it is becoming easier to employ GitOps processes to automate the pulling of code from a Git repository onto any platform. In some cases, that approach might reduce or eliminate the need to rely on a dedicated CI/CD platform to manage that process. Conversely, Mosquera noted CD platforms such as Spinnaker also provide a range of capabilities that span everything from automating audit process to rolling back application code in event of a major deployment issue.

    In the meantime, the amount of code moving through DevOps pipelines continues to increase. Simultaneously, the code is arriving in form of microservices that, although are comparatively easier to build, are often more challenging to deploy, update and manage.

    It’s not clear how many organizations might be looking to replace existing DevOps platforms to manage those processes more effectively. However, more organizations are embracing DevOps processes more widely to deal with an ever-increasing volume of code. As such, the number of organizations adopting open source DevOps platforms in one form or another will continue to rise. Most of those organizations, however, are unlikely to want to download on their own the raw bits that make up those projects.

  • CD Foundation Embraces Ortelius to Manage Microservices Deployments

    CD Foundation Embraces Ortelius to Manage Microservices Deployments

    The Continuous Delivery (CD) Foundation has added the open source Ortelius platform for managing the deployment of microservices as an incubation level project.

    Originally developed by DeployHub, Ortelius was created to provide a means to automate configuration at the application level. It also provides a central catalog of services that makes it easier to consume and deploy services across clusters that also tracks application versions and dependencies as updates to services are made.

    DeployHub CEO Tracy Ragan said Ortelius is now being donated to the CD Foundation, an arm of the Linux Foundation, which is working to create de facto continuous delivery standards. As such, Ortelius will benefit from contributions from other members of the consortium that includes Google, CloudBees, CircleCI, JFrog, IBM, GitLab, Netflix and DeployHub.

    The core CF Foundation projects are the open source Jenkins continuous integration/continuous delivery (CI/CD) platform; Jenkins X, an open source CI/CD solution on Kubernetes; Spinnaker, an open source CD platform; and the Tekton specification for building CI/CD components.

    Ortelius already integrates with CD orchestration tools and deployment tools such as Helm, Ansible and Spinnaker. The opportunity now is to address the entire application development life cycle of cloud-native applications using a GitOps methodology, said Ragan.

    While in theory the rise of Kubernetes should make it easier to automate CD processes across multiple platforms, Ragan noted that in practice there are still substantial differences between instances of Kubernetes that will require tools to automate the rollout of microservices across multiple clusters. Ortelius allows IT teams to logically manage groups for microservices as a single entity in much the same way IT teams manage monolithic applications as a single entity, she said.

    Ragan also noted that not all microservices are of equal value, so different approaches to managing them are required based on the number of dependencies they have and the value of the microservices to the business. As it becomes apparent IT teams will soon be managing perhaps thousands of microservices, the need for a standard approach that integrates with multiple CD platforms becomes more acute. In fact, she said, IT teams are approaching a new era when chaos engineering principles will need to be applied to the management of microservices.

    Longer-term, Ragan said Ortelius will also make it feasible to employ machine learning algorithms to provide predictive analytics that can be used to optimize the microservices environment. That capability eventually will provide a critical capability for managing microservices at scale. It may also serve to further the adoption of best CD practices that thus far have proven elusive to achieve in IT environments where each platform has unique attributes that make automating application delivery challenging regardless of what type of application is involved.

    In the meantime, IT organizations are being invited to contribute to the Ortelius project, now that it’s part of the CD Foundation. The more contributions that are made, the sooner the Ortelius platform graduates, which can’t come soon enough for many organizations that have already invested in microservices.

  • Continuous Delivery Without Exception

    Continuous Delivery Without Exception

    Continuous delivery (CD) and all the other continuous things—continuous integration (CI), continuous testing, continuous deployment—have become the new mantra of DevOps. CI/CD adoption typically starts with agile development teams working on new or well-funded systems. But what about the rest of the world and all of those “other” applications, under-funded IT teams are responsible for supporting? Can or should continuous delivery be a reality for all applications?

    The answer is a resounding yes! No application is an island. Every application is subject to change, whether it be code changes, patches to the OS, changes to the supporting databases or changes to the infrastructure it is running on. Without a regulated pipeline to deliver changes quickly across all systems, organizations expose themselves to unforeseen risks and escalating maintenance costs.

    No matter how new or well-funded an application is today, it will soon become one of many other applications that have to be secured, patched, updated and maintained as part of an organization’s IT estate. For example, consider all those “hot-new” Windows IIS sites built in the early 2000s now sitting on Windows Server 2008/R2, which reached end of life Jan. 14, 2020. Unsupported operating systems are not only costly but they also provide prime targets for cyberattacks, (the Bluekeep Vulnerability being just one of many examples).

    The term “technical debt” was introduced in 1992 by Ward Cunningham to describe what happens when we fail to adequately maintain what he calls “immature” code:

    “Although immature code may work fine and be completely acceptable to the customer, excess quantities will make a program un-masterable, leading to extreme specialization of programmers and finally an inflexible product. Shipping first-time code is like going into debt. A little debt speeds development so long as it is paid back promptly with a rewrite … The danger occurs when the debt is not repaid. Every minute spent on not-quite-right code counts as interest on that debt. Entire engineering organizations can be brought to a standstill under the debt load of anunconsolidated implementation.”

    Organizations Must Navigate Complexity

    The “J-Curve of Transformation” included in the “The State of DevOps, Dora Report” visualizes the challenges large, complex organizations face when automating delivery pipelines. Teams first target high-profile projects and quick wins and then see a drop-off as they’re faced with mounting complexity and technical debt.

    Nowhere is complexity and technical debt more relevant than when it comes to mission-critical commercial-off-the-shelf (COTS) applications Microsoft SQL. Manual controls, layers of process and siloed support teams are all commonly associated with these types of apps. The term “bi-modal” IT is commonly used to describe the variance in delivery processes used for these types of apps versus custom apps being released as part of continuous delivery pipelines.

    Not only are COTS apps handled in this “bi-modal” fashion, but they are also the “forgotten” and the “hidden” applications—the forgotten being legacy applications still in use but no longer being actively developed, and the hidden being all the applications needed by IT to run IT operations. To succeed at continuous delivery, organizations need to apply standardization across delivery pipelines for all teams and environments including complex, forgotten and hidden applications.

    Complex: Mission-Critical COTS Applications

    Large, mission-critical applications such as Microsoft SQL present unique challenges to teams focused on enterprise DevOps, but successful CD pipelines are possible. Teams must first recognize that packaged applications are fundamentally different than custom-built applications. Similar to older applications (which many packaged applications also are), the fundamental challenge is creating an automated way to build packages that can then be consumed by the pipeline. Success here also requires additional intelligence about the underpinning proprietary services and technologies supporting the packaged application.

    Forgotten: Legacy Applications

    There are myriad reasons why older applications present major challenges to organizations trying to achieve continuous delivery: technical debt, lack of resources, required refactoring, operating platform dependencies and a fear of business disruption, to name a few. Updating these apps many times requires individual acts of heroism, involving an abundance of manual efforts in which runbooks are individually updated, shadow sessions are invoked and the impact on dependent systems is typically not known until the update hits production.

    Getting started with continuous delivery requires creating an automated build package that can then be deployed via the pipeline to the preferred destination—cloud, on-premises or hybrid. If the application and all of its dependencies have been successfully abstracted from the underlying operating system and infrastructure and packaged accordingly, it can be consumed into the pipeline and deployed anywhere without having to refactor or rewrite the application.

    Hidden: IT ‘Owned’ Applications

    The Digital Age has led to an exponential growth of applications used by the business (websites, distribution, POS, etc.) as well as third-party applications/tools used by IT teams to build, test, deploy, manage, secure and monitor those applications. In many cases, these applications have become just as mission-critical as the business applications they’re supporting, but they’re not typically treated as “applications” by the business.

    In many cases, they remain completely hidden from the business, with little understanding of the value they provide. Often there is no clear owner for the application, different teams use the application in different ways, updates and patches are handled ad hoc, and expertise is limited to a few individuals within the organization (who take the knowledge about the application with them when they leave).

    Ultimately, third-party applications used by IT need to be managed like any other application. If not, delays in release and production failures caused by changes not properly tested in pre-prod will happen.

  • Maintaining Progressive Delivery Quality With Feature Flags

    Maintaining Progressive Delivery Quality With Feature Flags

    How does Netflix know exactly what you want to watch? How do social networks ensure major software updates are smoothly deployed across millions of users? The answer, in part, lies in progressive delivery. Progressive delivery, the incremental introduction of new features, has become key to rolling out new features. By testing unproven features on a small subset of users, platforms can advance quickly while keeping rapport with the main userbase. But, where does QA fit into the progressive delivery cycle?

    As software development teams face the need to iterate rapidly, the role of QA engineers is evolving as well. QA teams are increasingly assigned to support new customer experience initiatives and meet developer consumer needs. They must do so while understanding which customer metrics to monitor as developers introduce new features.

    I recently chatted with Jeff Sing, lead software QA engineer at Optimizely, recently acquired by Episerver, to explore how feature flags could be used to maintain quality and respond to user feedback throughout a progressive delivery life cycle.

    Defining Progressive Delivery and Feature Flags

    Progressive delivery is is a software release style that emphasizes very early code pushes and rapid feedback. By testing features among small subsets of a greater user base, developers can quickly gauge what works and what doesn’t.

    Feature flags are a necessary component to make progressive delivery work. Also called feature toggles, feature flags are switches that enable developers to turn a feature off or on, helping engineering teams test unfinished features or alpha features while avoiding maintaining multiple branches. “Feature Flags are a powerful technique, allowing teams to modify system behavior without changing code,” wrote Pete Hodgson in a post on MartinFowler.com.

    Feature Flags Enable Experimentation

    Adopting feature flags for progressive delivery allows for a lot of experimentation, said Sing. For example, companies could test a new user interface adjustment on a small subset of users. If it works well, the team could further and optimize it for specific user tastes.

    “Why would you want to build something that no one wants to use? How do you know you’re building the right thing?” asked Sing.

    Iterative release strategies are nothing new—and they go by other names: canary release, A/B testing, targeted rollouts and feature gating. However, what seems unique about feature flags is the ability to optimize granularly.

    For example, if you compare a friend’s Amazon home page with your own, you will likely find the dashboard, product recommendations and overall UI vastly different. These companies are continually “running active experiments on a per-user basis,” said Sing. This enables them to personalize experiences to niche user tastes, in real-time.

    Many other companies are utilizing progressive delivery tactics. Blue Apron, for example, adopts experimentation to find and refine business value. Experimentation is core to how HelloFresh distills its customer experiences. Compass also uses progressive delivery tooling to enable feature flagging within a suite of apps serving various platforms.

    For QA teams, this means shifting right—from business testing to nano-testing at the pace of constant delivery. Thus, QA teams must have an active role—“continually test customer feedback and continually reiterate on that feature,” said Sing.

    Implementing Feature Flags: QA Shifts Right

    By adding customer feedback into the continuous delivery loop, feature flags aid incremental deployment strategies and become essential for DevOps. So how do we implement feature flags in our application development processes?

    If we’re testing a single function within a small product, an obvious idea would be to comment out or uncomment code. But, that doesn’t scale well. Developers could, instead, program a conditional variable. However, there is still more dynamic feature-toggling options. More advanced solutions may involve a toggle router and/or a UI-driven alternative.

    Sing described a system using switches written as YAML files and adopting a visual dashboard to toggle on/off features. This setup can enable a process whereby the developer checks in code and passes it to site reliability engineers (SREs) or QA managers. They can then initiate testing in production, view data from beta users in real-time and respond accordingly.

    Integrating with a CI/CD pipeline would be the next logical step. Each build could carry many potential features within each build. If a flag in the code is on, the system would execute all code and components can easily be turned off or on remotely. This way, engineers “can control how things are represented in real-time without a code deploy,” said Sing.

    Melding feature flags and progressive delivery returns extremely valuable data, Sing said. If teams can see metrics such as system behavior, bounce rate, customer NPS score and business conversions, they can better increase the quality of future deployments. “You can’t argue with hard data,” he noted.

    Benefits of Feature Flags

    Utilizing feature flags within continuous development appears to afford many benefits. In the process, QA plays a more present role in how engineering teams roll out (or roll back) new features. “QA must understand the business impact and customer story,” said Sing. “It’s not just about uptime; it’s the experience, too.”

    As you can see from the use cases above, progressive delivery with feature flags brings many benefits:

    • Quick validation: Bring new product features and proof of concepts faster to market faster by improving testing.
    • Experiment and iterate: Release incremental software versions. Experiment with new or unfinished features that consider fringe cases.
    • Development: Avoid constant merging of branches.
    • Reflect customer journey: Gather more heuristics on the customer journey to test and measure customer data.
    • Optimize per-user: Test and optimize for user tastes. “Why would you want to build something that no one wants to use?” For Sing, targeted rollouts can answer that user imperative immediately.
    • Limit blast radius: If new features introduce outages, easy togglable capabilities mean they can quickly be turned off and put back into development, thus increasing service reliability.

    Drawbacks of Feature Flags

    Feature flags introduce many efficient methods for testing new features, but they don’t come without side effects in other areas. One potential drawback of feature flags is supporting a vast number of intricately defined user scenarios.

    With each segmentation comes new optimization requirements and support must adapt to a quickly growing archive of various feature sets. Companies need advanced tooling to support the easy addition or removal of features. “You need the power to understand what’s going out and how you manage it,” Sing noted.

    A fractured ecosystem with thousands of microservices can become difficult to support continuously. Engineering teams must draw the line somewhere between endless optimization and retaining a cohesive product. Ideally, developers can reuse base components but represent them in different ways.

    Another negative side effect is public opinion. With the release of Social Dilemma and related tech industry critique, many user-facing tech networks are under harsh criticism due to the unethical use of user data to develop hyper-targeted ad campaigns. Optimization tactics that disrespect users should be frowned upon.

    This is not to mention regulations around user data and privacy. If users do not grant consent to share user data to “improve the user experience,” engineering teams won’t have the necessary metrics and tracking data required to decide which features to implement.

    While feature flags appear to help optimize business value, there are many potential “gotchas” to carefully consider before implementation, from technical logistics to service fragmentation, user ethics and legal ramifications.

    Final Thoughts

    “As with any new technology, if you don’t use it correctly, it’s almost better if you don’t use it,” said Sing. If progressive delivery isn’t managed correctly, with a QA mindset, it could come back to bite them.

    As we continually build new tech to ship faster and smarter, it’s also important to consider customer needs. “Automation is great, but a computer can’t understand what a customer really wants,” Sing noted. To avoid the technical hurdles and social dangers of hyper-personalization, QA should be customer champions that consider customer happiness.

  • CDF Officially Graduates Jenkins CI/CD Platform

    CDF Officially Graduates Jenkins CI/CD Platform

    The Continuous Delivery Foundation (CDF) announced today that the open source Jenkins continuous integration/continuous delivery (CI/CD) has officially graduated.

    Jenkins was donated last year to the CDF, an arm of The Linux Foundation, along with Jenkins X, a CI/CD platform for Kubernetes; Spinnaker, a CD platform; and Tekton, a framework for building pipelines that will be employed as the foundation for next-generation CI/CD platforms.

    Tracy Miranda, CDF Governing Board Chair and director of open source community at CloudBees, said that although Jenkins traces its lineage back to 2004, the CDF wanted to make sure the same requirements that are being applied to graduate other projects are applied to Jenkins despite its obvious maturity. The CDF requirements for graduating a project are to demonstrate growing adoption, have an open governance process, feature maturity and a strong commitment to community, sustainability and inclusivity.

    The CDF also today published a public road map for Jenkins. Key features include improving the user/administrator experience and tighter integration with cloud platforms. Work has also begun on Operator tools for simplifying the deployment of Jenkins on a Kubernetes cluster.

    Longer-term, the CDF is focused on expanding the adoption of Tekton-based Trigger pipelines across both Jenkins and Spinnaker. Tekton pipelines are easier to create and can be reused across multiple platforms. Jenkins X already employs Tekton-based pipelines, which were originally developed by Google.

    The founding members of the CDF are CloudBees, Google, CapitalOne, CircleCI. Fujitsu, Huawei, IBM, JFrog, Netflix and Salesforce.

    In the nearer term, Miranda said the CDF is also attempting to standardize the glossary of terms applied to DevOps processes. The goal is to provide something akin to a “Rosetta Stone” that will help make DevOps more accessible to IT professionals just starting their journey.

    In theory, at least, platforms such as Jenkins X accelerate the development of microservices-based applications running on Kubernetes clusters. In practice, however, many IT teams continue to use Jenkins and other CI/CD platforms to build both monolithic and microservices-based applications.

    As Kubernetes gains momentum, however, the way applications are deployed may also evolve. Today most DevOps teams push application code from a CI/CD platform on to a target system. However, if Kubernetes is deployed on multiple platforms it may become easier to automate the delivery of code by enabling Kubernetes clusters to pull code from CI/CD platforms on demand.

    As organizations transition to building and deploying microservices-based applications, it’s also apparent there will be a much greater need to rely on best DevOps practices to manage all the dependencies that will exist between each microservice. Legacy application development processes will not be able to keep pace with the rate at which microservices based on containers are ripped and replaced.

    Of course, it’s not clear to what degree organizations may prefer to stay with Jenkins, which is built on a Java virtual machine, versus adopting Jenkins X or a CD platform such as Spinnaker. However, as the interoperability between these platforms continues to improve, it may not be surprising to one day find DevOps teams employing a mix of all of them across different application development projects.