devopsy.kinsta.cloud

Tag: pipeline

  • Dynatrace Extends Data Reach and Scope of Observability Platform

    Dynatrace Extends Data Reach and Scope of Observability Platform

    At its Perform 2024 event today, Dynatrace unfurled a Dynatrace OpenPipeline that makes it possible to apply analytics to multiple types of data sources in real time.

    The company also unveiled a Data Observability offering that can be used to vet the quality and lineage of data being exposed to the Davis artificial intelligence (AI) engine at the core of the Dynatrace observability platform to reduce false positives that otherwise might generate an alert in addition to helping reduce the volume of data that might need to be stored.

    Finally, Dynatrace announced it is starting today extending the reach of its observability platform to the large language models (LLMs) that are used to create generative AI platforms. That capability will, for example, make it simpler to monitor the consumption of tokens that are used to provide access to these models.

    Steve Tack, senior vice president for product for Dynatrace, said Dynatrace OpenPipeline will enable organizations to streamline the collection of data in a way that will enable them to apply observability more broadly by applying stream processing algorithms to petabytes of data.

    Scheduled to be available in 90 days, the Dynatrace OpenPipeline capability enables IT teams to ingest and route observability, security, and business events data from any source and in any format at the point of ingestion, including unstructured data that is automatically converted into a usable format. That data can then be enriched to enable deeper analytics.

    It also provides IT teams with more control over which data they analyze, store or exclude from analytics, which in turn helps reduce the total cost of observability, noted Tack.

    Finally, it presents IT teams with the ability to apply more customizable security and privacy controls to how that data is employed, he added.

    Dynatrace is pursuing a multimodal approach to AI that spans predictive, causal and generative models. Collectively, that approach makes it simpler to identify the root cause of issues, identify anomalies that might disrupt services and streamline incident management by, for example, providing summarizations of events in a natural language format.

    It’s not clear how broadly observability will be applied beyond the data collected to manage DevOps workflows and other IT processes, but it’s clear there is a high correlation between IT events and business outcomes as organizations become more dependent on software. As AI becomes more pervasively employed, it should become simpler to apply analytics to a much wider range of types of data to surface the relationship between IT events and business processes. In effect, Dynatrace is making it simpler to apply best data engineering practices to collect, manage and analyze that data, noted Tack.

    In the meantime, DevOps teams should revisit how their pipelines are currently constructed to make it simpler to capture all relevant data. As critical as telemetry data collected from the DevOps platform is to ensure application development and delivery occurs as fast as possible, that data is only one factor in a larger equation. The challenge and the opportunity now is determining how best to apply AI to correlate it all.

  • Cycode Leverages eBPF to Secure CI/CD Pipelines

    Cycode Leverages eBPF to Secure CI/CD Pipelines

    Cycode today added a Cimon extension to its application security platform that uses extended Berkeley Packet Filtering (eBPF) to thwart cyberattacks against continuous integration/continuous delivery (CI/CD) pipelines.

    Alex Ilgayev, head of security research for Cycode, said Cimon uses eBPF to inspect network connections, running processes and file modifications within a pipeline to learn standard behaviors. Armed with those insights, Cimon will then surface any anomaly in real-time, he added.

    DevOps teams can then craft and embed security and compliance rules within their pipelines based on the severity of the anomalies discovered, said Ilgayev.

    eBPF makes it possible to run sandbox programs at the kernel level without adversely impacting performance. In effect, eBPF bridges the boundary between kernel and user space by enabling developers to combine and apply logic across multiple subsystems that, historically, were completely independent of one another. That approach enables, for example, a security tool to scale to the point where it can identify threats at much higher levels of throughput to improve overall scale at a time when the volume of cybersecurity attacks continues to increase.

    Cycode is taking advantage of eBPF to enable DevOps teams to not only defend against known attacks involving malicious package installation, typosquatting, repojacking, dependency confusion, dependency hijacking and other types of dependency attacks but also zero-day threats, noted Ilgayev.

    Cycode provides a platform for mapping metadata and events using graph technology that makes it simpler for application development and cybersecurity to consolidate alerts generated by their DevSevOps tools. Most recently, Cycode added a Cycode Application Security Orchestration and Correlation (ASOC) module that will automatically discover all the tools that make up a DevSecOps workflow without requiring any integration effort.

    The approach enables the Cycode platform to discover, for example, dependency issues that would otherwise be missed by software composition analysis (SCA) or static application security testing tools (SAST).

    While a lot of progress has been made in terms of adopting DevSecOps best practices, many organizations are still struggling with securing their software supply chains. Most of the members of a DevOps team have limited cybersecurity expertise, so they need the help of a cybersecurity professional to determine what specific actions are required to remediate a vulnerability. Cybersecurity professionals, conversely, don’t have a lot of application development expertise.

    As the number of regulations specifically focused on application security steadily increase, every organization that builds software will soon need to embrace DevSecOps to better secure software supply chains. Unfortunately, developers have too often viewed cybersecurity as an obstacle to deployment while cybersecurity teams have historically considered developers to be a primary cause of the breaches they are expected to clean up. The first step toward bridging that divide is, of course, to realize that application developers and cybersecurity teams need to be invested in each other’s success.

  • Keeping the DevOps Pipeline Flowing as Attack Surfaces Grow

    Keeping the DevOps Pipeline Flowing as Attack Surfaces Grow

    The attack surfaces that today’s businesses and public entities must manage have never been more complex and difficult to protect. The introduction of cloud and SaaS offerings over the past decade has caused seemingly insurmountable IT sprawl, with a third of successful attacks now occurring via unmanaged or virtually unknown assets. And while shadow IT is an urgent threat confronting many organizations, the migration of the software development life cycle to the cloud—and the rise of machine identities—is compounding the attack surface in profound ways. Emerging out of the Agile and lean strategies for software development in the 2010s, DevOps predates the mainstreaming of cloud computing.

    But once the value of cloud-based IT resources came to be widely appreciated, development teams began moving their developer environments to the cloud in a big way—to the point where many people today think of DevOps and cloud as inseparable. Recent Gartner research showed that much of enterprise IT spending is now directed toward cloud projects. In fact, for IT categories like application software, infrastructure software, business process services and system infrastructure, more than 51% of IT spending will shift away from traditional solutions to the public cloud by 2025. Nearly two-thirds of spending on application software alone will be directed toward the cloud by 2025.

    The explosion in cloud DevOps has brought with it a huge uptick in the number of privileges, credentials, identities and access rights under management. We’re all familiar with how digital identity and access management (IAM) works in conventional settings through combinations of usernames and passwords. In the DevOps world, where cloud containers can run as distinct microservice processes interacting with other processes (machine-to-machine communication), identities and privileges are usually governed through SSL/TLS certificates. These machine identities have emerged as a critical security priority across multiple industries, not just because they’ve proliferated exponentially, but because of the lack of effective tools and strategies available to IT security teams.

    Machine Identities—Scope of the Issue

    A recent global survey of 1,000 CIOs found that the average organization has 250,000 machine identities under management coming into 2022, with that number expected to double by 2024. The study also found that 83% of organizations suffered a machine identity-related outage in the past year, with 57% of organizations experiencing at least one data breach or security incident related to compromised machine identities over the same time period.

    Because the perimeter defense solutions and strategies that formed the basis of IT security best practices for many years are of little use in cloud environments, security admins have had to rely on existing IAM and privileged access management (PAM) tools to secure data, accounts, and development pipelines. Not surprisingly, identities and access privileges have become the leading attack vector in the cloud, providing attackers with an exponentially higher number of vulnerabilities to target within the enterprise than were available even a few years ago. We’ve seen these points of weakness come into play in recent high-profile breaches and attacks such as SolarWinds, Codecov and PHP.

    Embracing Zero-Trust Principles to Secure the DevOps Pipeline

    Fortunately for today’s DevOps teams, cloud-oriented privilege management technologies have evolved rapidly in recent years, especially in the categories of cloud infrastructure entitlement management (CIEM) and time-based role access management or just-in-time privileging. CIEM solutions target the security weak points and process vulnerabilities that are unique to the cloud environments. These include the lack of deep visibility into user, group, and role privileges, which together make it difficult to conduct oversight and control over the activities of users within cloud infrastructure and applications. Where conventional approaches such as IAM and PAM are designed primarily to secure data, apps, and users within the organization, CIEM is designed to manage permissions risks to prevent cloud breaches and data theft.

    Emerging CIEM solutions are bringing to bear advanced capabilities in discovery, monitoring and auditing to significantly broaden visibility into DevOps processes for IT security professionals. With a CIEM solution, security teams can scan cloud applications for existing users, groups, roles and granted privileges down to the resource level. Each user, group, role, and privilege can be categorized to identify and verify that privileged users have the appropriate levels of access. Access reports can be generated to concisely outline users and groups with the most privilege and the highest risk.

    On the audit front, CIEM solutions provide the capability to track and report on levels of privilege as well as all activity associated with those privileges. In this way, it becomes possible to generate privileged access reports for consumption by internal and external auditors to ensure compliance with regulatory and other requirements such as SOX, GDPR, HIPAA, GLBA, PCI-DSS, ISO 27001, SOC 2, FedRAMP and more. Similarly, with the ability to monitor privileged access across all cloud applications, security teams can apply behavior analysis to identify privilege abuse and abnormal activities from users. As with conventional SIEM tools, CIEM solutions will generate alerts to administrators to suspect activities or behaviors—for instance, if a user attempts to circumvent policies preventing direct privileged access to cloud services.

    Just-In-Time Privileging and Zero Standing Privileges

    Importantly, CIEM solutions are also effectively eliminating standing privileges to cloud resources based on least-privilege access and zero-trust principles. The automated granting and expiring of permissions—just-in-time (JIT) privilege grants—is highly effective at minimizing attack surfaces. Because these JIT/zero standing privilege (ZSP) solutions work on a zero-trust model, no one and nothing is trusted with standing access to cloud accounts and data.

    With JIT permissioning, elevated privileges can extend either for the duration of a session or task, or for a set amount of time. Once the task is complete, those elevated privileges are automatically revoked–all without sys-admin involvement. Where a user previously had standing access privileges potentially extending around the clock for months at a time, converting to JIT granting would compress that attack surface to several hours per month. Further, JIT permissioning largely frees organizations from having to maintain and pay for both privileged and non-privileged accounts. Dynamic secrets generation – where a dynamic secret is generated on demand and is unique to a client, instead of a static secret being defined and shared ahead of time – also provides a better model for securing temporarily deployed services and features.

    DevOps and DevSecOps are still new and fast-evolving concepts within the wider computer science and cybersecurity universe. No doubt, DevOps has been wildly successful in accelerating automation and speeding time to market for innovative applications and business services. To date, however, security solutions providers have struggled with cloud access management. In other words, they have struggled to accelerate privileged access solutions that could secure the devices, data, and resources used by DevOps teams, especially in cross-cloud environments. Dynamic permissioning platforms using JIT privilege grants and employing ZSP principles show great promise in solving these problems.

  • Meet the New(ish) Pipeline

    Meet the New(ish) Pipeline

    My company keeps a sample project running pretty much all of the time. Our current project (multi-target web/Android app using Java and Spring built with Gradle and Ant that we’ll call “Finder” here) is getting a bit long in the tooth, so we have been casting about for a new one over the last several months. From non-work sources, we came upon the idea of multi-targeted Godot with C# as a viable side project. Moving to a completely different set of tools is kind of the point; it makes us stretch, and that is the idea of our ongoing projects. The stated goals for these projects are:

    1. Stretch knowledge about different languages, DevOps tools and platforms.
    2. Keep doing what you all are doing. Too many pundits have “grown out of” hands-on work, and it shows.
    3. Try out new ideas/architectures.
    4. Avoid “kitchen sink.” We can’t have a current test app that includes all available technologies, so keep it streamlined. Approach this as a production app that we use for testing, not as a test app.

    That’s why we’re looking to replace our Java/Spring project. We’ve got that pipeline down, we’ve tweaked the app until we are happy with the core and it hasn’t changed in a good while. But change is a part of DevOps, so we’re off.

    The Obvious

    Let’s take care of the obvious elements in our toolchain right away:

    1. Git. We’ll use GitHub, but it was a given that some form of Git would be holding our code.
    2. Visual Studio Code. We’re doing C#, so this is going to be the most useful option for code editing. The built-in Godot editor is good and will be used, but VS-Code offers C# functionality add-ons.
    3. Antsle is the deployment tool of choice, though this has changed over the years. We’ll use it to set up install sets for clients.
    4. Jenkins or Hudson. We’re going to use this only for test management.

    The Less Obvious

    There are several constraints we want to (or have to) meet. First, we want to avoid using Jenkins or Ant to control the entire build process, mostly because we’ve done both in previous projects, and most DevOps teams are no longer using these methods for new projects. Second, this is a sample we keep, not an investment, so we want to use open source technology if at all possible, to keep expenses down. Third, we’re using a game engine to do app development. This has some impacts–like most game engines, it is in control at runtime and we need to bundle the engine in the final application.

    1. The Godot Engine. Godot is written as a game platform, but its graphics/cross-platform abilities make it useful for graphics-heavy apps like ours. But we have to include the engine and its config files in the pipeline.
    2. Code Scanning. Static analysis was a late addition to Finder, and it’s in Java. This project is in C# and Godot Script. We have not yet found a tool that supports Godot Script, but C# we can get; it just adds restrictions. So we will use a static analysis tool, but we’re still deciding which.
    3. Build Automation. We’ve used Jenkins, Gradle, Maven and Ant to handle this step at different times in the past, but something newer would be a good idea. We are still evaluating this step. We will likely end up back on one of those tools because they are still used today. Godot is a new element to us, so we are also evaluating what is required to hook into it and make functioning apps in a repeatable process.

    Fueled Innovation

    We’re having fun—first couple sprints have us running a working prototype, so we’re happy with that. There’s still a lot to implement in the app and in the pipeline, so we’ll no doubt have more to say here soon. We’re talking about putting a server component in—not because the original concept needs it, but because our outgoing app—Finder—had server and client sides, and our need for things like Postman go away without this architecture. And yet, API security, testing, etc. is an important part of the DevOps architecture. As you can see, we’re still iterating.

    As to Finder, we normally take down the pipeline for a previous playground project when the new one goes up, but Finder costs us little. There are five containers (three permanent for the app hosting; two for build/test) and some processing power, so we’ll probably keep it since it is using a different set of pipeline tools (and languages—it is mostly Java/JavaScript).

    Meanwhile, you all keep rocking it. We’re building an(other) app that no one but us will ever see—but you are keeping the enterprise humming. Kudos to you!

  • Survey Sees Application Testing Shifting Further Left

    Survey Sees Application Testing Shifting Further Left

    PractiTest, a test management platform provider, published a report today that suggests professional application testers have become more adept at working within DevOps workflows.

    The annual survey of more than 1,000 application testing professionals finds that 80% of all application testing within organizations is being done by test engineers, test leads, testers and automation testers. A full 92% of survey respondents also noted they are working within the context of an agile methodology, while 42% cited DevOps specifically. Half of respondents said they work within the framework of a continuous integration/continuous delivery (CI/CD) platform.

    Significantly, more than half of respondents (59%) said testing is shifting left within their organization, but only a third (33%) said programmers participate in the testing process.

    PractiTest CEO Yaniv Iny said that level of DevOps inclusion suggests that, rather than shifting testing completely left on to the shoulders of application developers, a more nuanced approach to application testing is taking hold as professional testers learn to work within the sprints that make up a DevOps workflow. A full 90% of respondents said testing has become more automated within their quality assurance (QA) processes.

    By now, most application testers have come to terms with automation and, increasingly, AI, said Iny. The assumption now is that large swaths of the testing process will be automated, Iny added. A full 97% of respondents said that functional testing automation was either important or very important for them to succeed, while 96% said test automation patterns, principles and practices are also critical.

    Usage of behavior-driven development (BDD) as a testing model has also increased, with 27% of organizations now employing it to go beyond simply testing functions.

    However, the survey also found that, instead of automation decreasing the size of testing teams, those teams are getting larger. The number of organizations with 16 or more testers increased by 10% to 34% compared to a year ago. That suggests organizations are launching either more complex application development projects or launching multiple projects at the same time as they embrace digital business transformation initiatives, Iny said.

    The survey found nearly 70% of respondents have at least five years of experience. Nearly three-quarters (71%) of respondents reported that their income was not affected by the economic downturn brought on by the COVID-19 pandemic. However, nearly half of respondents (54%) reported that they were concerned about job stability, an increase of more than 30% from a year ago.

    The pandemic did change the way testers acquire new skills. The number of testers reporting that they attended online conferences, meetups and seminars increased to just under half (49.5%), up from 40.5% in 2019. Participation in online communities and forums grew to 44%, while only a third of respondents receiving any kind of formal training.

    Testing professionals with five to 10 years of experience earned an average salary of $89,000 in North America, which compares to $76,000 in Australia/New Zealand, the survey also found.

    Application testing as a distinct discipline doesn’t appear to be fading away anytime soon. Rather, organizations appear to be investing even more in testing to deliver higher-quality applications.

  • Top Pressing Concerns for CI/CD in 2020

    Top Pressing Concerns for CI/CD in 2020

    The tech landscape is changing rapidly. With the need for increased development velocity, CI/CD has become ubiquitous among most organizations. Yet, with the growing number of microservices, CI/CD may be tough to wrangle without the right strategy.

    At the same time, it’s difficult to keep pace with the constant evolution around CI/CD. So, what is the state of the CI/CD space in 2020, especially in these turbulent times? What do stakeholders see as top pressing concerns companies should consider to advance their release fluidity?

    For this article, I interviewed two stakeholders on the latest trends and industry cooperation around CI/CD. Read on to discover what two CI/CD thinkers, Tracy Miranda, board chair, Continuous Delivery Foundation (CDF), and Lachlan Donald, CEO, Buildkite, see are pressing concerns for CI/CD in 2020.

    More Microservices, More Problems

    With the number of microservices rising to the hundreds (or thousands) for some enterprises, maintaining consistent release strategies across the board could easily become a nightmare. If the goal is to replicate a unique CI/CD pipeline per microservice, pipeline automation and configuration management become top concerns.

    “Automation will be key,” said Miranda. “We can no longer afford to be writing each CI/CD pipeline by hand; we will need new approaches.” She added that recent industry discussion shows “configuration management is a top concern for CI/CD practitioners.”

    Tools like Jenkin X could enable this sort of automation, she noted. Companies should consider evolving their CI/CD pipelines with better configuration management.

    Platforms Must Evolve

    The recent “State of the Cloud Report” from Flexera found 39% of IT leaders saw implementing CI/CD in the cloud as a top priority. According to Miranda, the global crisis seems only to accelerate the need for cloud and CI/CD automation.

    But, traditional CI/CD platforms aren’t always suited for modern development workflows with numerous pipelines. For Donald, an evolution in new tooling is absolutely necessary to help companies keep pace.

    “We’re at a point where organizations need to completely rethink their approach to CI/CD,” Donald said. “Traditional CI/CD platforms are not suited for modern development workflows.” He cites slow speed, unreliability, scalability limitations and security risks as common hangups among many CI/CD tools.

    Lack of Clarity in CI/CD Market

    The market has expanded significantly to address the gap in CI/CD tooling effectiveness. According to Miranda, this has led to a continuous deployment tooling ecosystem wrought with interoperability and competing styles.

    “My biggest concern is the proliferation of tools and platforms with a lack of interoperability and lack of clarity of how to best use them or whether they are fit for purpose,” Miranda said.

    These differences make it difficult for developers to find the right integrations that work for their environments, leading to wasted time and effort. “Developers will experience growing pains with their CI/CD platforms as they work to keep up with the constant demand for quality software,” acknowledged Donald.

    Lack of Standards

    Part of the lack of clarity resides in the fact that there are little to no standards emerging in the CI/CD space. Aside from Accelerate rising as the “bible” for software delivery, Miranda sees a lack of standards in the CI/CD space.

    “Not having standards or de facto standards is a problem,” she said. “For example, why is there no de facto standard for reporting pipeline test results?”

    Advancing Open Source CI/CD Tools

    To improve Interoperability throughout the CI/CD industry, bodies such as CDF are working on setting general guidelines. Miranda described how CDF is accomplishing this, by supporting a CDF Interoperability working group, sharing CI/CD practices and maintaining open source CI/CD projects. Two CDF projects she finds quite compelling are Tekton, a pipeline execution engine, and Jenkins X, a new CI/CD platform for pipeline automation.

    Cloud vs. On-Premises: A False Dichotomy?

    In analyzing the CI/CD tooling ecosystem, Donald recognizes a recurring dichotomy; that companies must choose between the security of on-premises and the speed of cloud-native. In reality, this dichotomy may be false.

    “It is possible to have the best of both worlds,” he said. “A hybrid strategy [is] the solution for testing and deploying software at scale without compromising security or performance.’

    As opposed to cloud tooling, self-hosted CI/CD control planes could grant developers more code control and help meet security standards. “I believe self-hosted agents are always a good idea,” Donald said.

    As we recently covered, a hybrid cloud approach is picking up steam. Similarly, Jenkins X is designed to run on any Kubernetes cluster, regardless of on-premises or cloud, demonstrating industry movement toward tooling neutrality and hybrid motivations.

    Moving Beyond PASS or FAIL

    There is a reductive tendency in CI/CD to simply check whether a build passed or not. But, this mindset misses an enormous amount of useful information and undervalues the test and release suites themselves.

    “Teams should get granular about what worked and what didn’t,” Donald said. DevOps teams must evolve beyond a “PASS or FAIL” mindset to analyze more minute aspects of the build process itself.

    For Donald, this also means shifting to treat test code performance “as a first-class citizen, applying modern development and scaling practices to it and allocating the necessary resources to it.”

    Future Evolution: Progressive Delivery and Hybrid Approach

    In closing, I asked Miranda and Donald their thoughts on the future CI/CD evolution. They shared their perspectives on where CI/CD needs to evolve.

    Miranda views progressive delivery as the next milestone. With progressive delivery, teams deploy services in a staged manner, she said. “Tools, technologies and processes that fall under the ‘progressive delivery’ umbrella will be the next milestones for improving deployment speed and efficiency.”  Building blocks such as services meshes could help advance this process, she noted.

    Donald stressed the need for a hybrid approach to CI/CD. A neutral control plane could grant better control and future-proof continual release strategies.

    Assessing the CI/CD Landscape

    In summary, our panel shared thoughts on the top concerns for CI/CD in 2020:

    • Platforms must evolve: Teams should rethink their approach and emphasize automation in pipelines.
    • Lack of clarity: There are few standards and a lack of clarity among tools. This confuses developer adoption.
    • Interoperability: A focus on interoperability is critical toward advancing the industry.
    • Unhelpful dichotomies: The cloud-native versus on-premises divide may not be helpful for a hybrid strategy; thus, it’s recommended to seek out platform-neutral tools.
    • Move beyond PASS or FAIL: Switch the mindset to “Test code = application code.”

    High performant organizations must scale their technology tremendously. With this desire comes a mounting strain on how companies adopt a continuous deployment strategy. Without a proper CI/CD plan, business agility is sacrificed.

    As an ending note, I absolutely love Donald’s statement: “A good litmus test for how well a company is run is their approach to CI/CD.” Perhaps what all can agree on is that companies must prioritize CI/CD, as it is now business-critical. Otherwise, deployments may be numbered for those organizations that don’t adapt.

  • Chef Commits to 100 Percent Open Source Code

    Chef Commits to 100 Percent Open Source Code

    Chef has decided to open source its entire portfolio of IT automation software as part of an effort to make it easier for organizations to construct a DevOps pipeline using the company’s software.

    A part of that effort, Chef also launched the Chef Enterprise Automation Stack—which combines Chef Infra for managing infrastructure, Chef InSpec for maintaining compliance, Chef Habitat for managing applications, Chef Automate for managing hybrid clouds and Chef Workstation, a starter kit for launching Chef—within a single distribution of Chef software. Chef Infra is the original Chef project around which the company was launched.

    Corey Scobie, senior vice president of product and engineering for Chef, said the company previously offered a mix of open source and proprietary software that made engaging with some customers overly complex from a licensing perspective. This approach aligns the entire Chef business model around providing support for open source software, he said.

    At the same time, Chef is now open to collaborating more closely with developers who want to contribute code to its various projects. That’s critical because code contributions are now the primary means through which many IT organizations help shape the direction of products as they continue to evolve, Scobie noted.

    Of course, many IT organizations today either mandate the use of open source software or have instituted an “open source first” policy that makes it challenging for any proprietary software to be adopted.

    Chef expects that by making its software easier to consume via a unified open source transition it will be able to further accelerate adoption of what is described as a “Coded Enterprise” to managing IT. Rather than manually configuring IT assets in isolation, that approach encourages organizations to adopt best DevOps practices that treat all aspects of IT, including infrastructure, as code, said Scobie, adding that capability is more important than ever as IT organizations are increasingly require to manage heterogenous cloud computing environments.

    Chef claims that approach has already resulted in more than half of the Fortune 500 having adopted at least one module of its software and, in the fourth quarter of 2018, the highest level of bookings in the company’s history.

    While Chef helped pioneer the IT automation category, competition across this sector has intensified. Chef and its primary rivals Puppet and Red Hat are the leading proponents of frameworks for automating IT operations. Although adoption of those frameworks has increased substantially in recent years, use within IT organizations often remains fragmented, often resulting in islands of automation. Chef is clearly hoping that IT organizations will increase adoption of its software stack now that it’s  based entirely on open source software. Many organizations face challenges teaching their IT operations teams enough programming skills to automate their processes. In absence of those skills, adoption of IT automation frameworks within the context of a set of well-defined DevOps best practices often remains uneven, at best.

    — Mike Vizard

  • Atlassian Adds Pre-Configured Pipelines to Bitbucket CI/CD

    Atlassian Adds Pre-Configured Pipelines to Bitbucket CI/CD

    One of the primary reasons DevOps processes have not been widely adopted across the enterprise is that building pipelines across all the tools required is a painstaking effort. As a result, most pipelines today are constructed using manual pipelines. However, Atlassian has announced it is now making available pre-configured Bitbucket Pipelines that DevOps teams can access via a simple drag-and-drop interface.

    Harpreet Singh, head of product for Bitbucket Cloud at Atlassian, said DevOps teams that rely on the Atlassian Bitbucket continuous integration/continuous deployment (CI/CD) platform will be able to employ a series of pre-configured Bitbucket Pipelines built by Atlassian or one of its third-party partners, rather than having to manually build every pipeline.

    The goal is to make it much easier for DevOps teams to incorporate, for example, a new cloud platform into an existing set of DevOps processes without waiting for someone on the DevOps team to write, test and validate a script, said Singh. IT vendors that have worked with Atlassian to create Bitbucket Pipelines include Google Cloud, Microsoft, Slack and Amazon Web Services (AWS).

    One of the ironies of DevOps is that while CI/CD platforms make it easier to build and deploy applications, constructing the CI/CD platform can be exceedingly complex. A recent survey conducted by Atlassian found DevOps teams rely on 4.3 tools to move code from development into production environments. A full 62 percent of respondents reported they resort to performing manual testing due to insufficient automated test coverage, additional manual processes or a lack of build/deployment pipeline automation.

    While some organizations might have the engineering resources required to build pipelines, he noted, the average IT organization can’t afford to hire operations specialists that have the requisite programming skills required to build pipelines manually.

    Best of all, noted Singh, the pre-configured Bitbucket Pipelines that Atlassian makes available will be maintained and updated by their original authors.

    It’s too early to say to what impact pre-configured pipelines is likely to have on DevOps adoption in the enterprise. But most enterprise IT organizations won’t significantly alter their process until it becomes relatively easy to make that transition. Enterprise IT organizations have relied on administrators to manage IT for decades, and most of those administrators wouldn’t know where to begin if asked to manually construct a pipeline. The few enterprise developers who might be able to build those pipelines are simply to busy writing application code. The result is a paralysis that prevents organizations from modernizing their application development and deployment processes.

    Of course, any meaningful change in the enterprise also involves a chicken-and-egg debate. Lacking skills and tools, enterprise IT organizations are unable to overcome internal inertia, which in turn results in conservative IT organizations preferring to rely on known processes no matter how inefficient they might be. The real challenge facing IT leaders trying to advance DevOps adoption is to remove as many excuses for resisting change whenever and wherever possible, starting with how pipelines are constructed and maintained.

    — Mike Vizard

  • DevOps Chat: The Modern Software Factory and CD Pipelines w/ Scott Willson, CA Automic

    DevOps Chat: The Modern Software Factory and CD Pipelines w/ Scott Willson, CA Automic

    This DevOps Chat features Scott Willson of CA Automic talking about CD pipelines and the modern software factory. The concept of a software factory may be new to some, while others may have heard the term but not really thought about what it means and still others may be very familiar with the term and practice it at their own job. But the idea of a software factory utilizing a pipeline for continuous delivery is at the heart of DevOps.

    This is a really great chat that I enjoyed participating in with Scott. I hope you find it as such too.

    As usual, the streaming audio of our conversation is below, followed by the transcript of our conversation.

    Audio

    Transcript

    Alan Shimel: Hey, everyone, it’s Alan Shimel, devopsy.kinsta.cloud, and you’re listening to another episode of DevOps Chat. Today’s chat features my friend, Scott Willson, of CA Automic. Scott has been a frequent guest on DevOps Chat and the reason is he actually has real, usually, good stuff to say and he’s an interesting fellow to speak with. Scott, welcome again to DevOps Chats.

    Scott Willson: Thank you, Alan. As always, glad to be here.

    Shimel: Pleasure to have you here, my friend. So, Scott, we are going to delve into something today that maybe a lot of people in our audience are really familiar with and will like the drill-down, but there may be people who hear the terms but they’re not quite sure what it means. And so what I wanna talk about today are pipelines and especially when in terms of continuous delivery pipelines and how this fits into a concept – really, CA has coined it and led with it, but it’s the modern software factory, right, for enterprises. It’s a factory methodology, almost like an assembly line, if you will, of developing and deploying software and all that that entails.

    So why don’t we start with basics, Scott, if it’s okay. What is – when we talk about the CI/CD pipeline, how would you explain that to someone?

    Willson: Excellent. So the CI/CD pipeline is basically just the concept or the metaphor of taking any change, right, and this is any change that a developer has coded – a configuration change, a change to the data, and, currently, any change to an environment configuration – needs to go back to a starting point and then kind of run the gauntlet, as it were, where it is vetted, it’s tested, warranted, and then made available in production environments. And there’s a whole sequence of things that need to occur, and that sequence is referred to as the “pipeline.” And pipelines are often broken up into stages, from a logical perspective, but in those stages are specific tasks that have to occur in sequence, to have your software, ultimately, be ready to be safely deployed and delivered to production environments.

    Shimel: Yep. It sounds great. Now, Scott, one of the things we’ve seen is, for instance, with Jenkins 2.0, the concept of pipelines and CD pipelines has really, I think, taken on a new gravitas, if you will, within the market. People are now getting serious about it. What about at Automic? And, beyond Automic, the rest of CA even? Where does this whole pipeline thing fit in?

    Willson: Excellent. So, you know, Jenkins really is a continuous integration tool. It’s meant to run a sequential set of all-or-nothing tasks – I mean, “all-or-nothing,” that if step three of five were to fail, then the idea is the whole build would be marked as having failed. And they realized that, “Well, you know, maybe we ought to extend this to a continuous delivery thing, ie. move ourselves out of just building software and trying to push it to other environments. Now, of course, there are some challenges with this, so lots of vendors, including CA Automic’s ONE automation platform – and there’re several other vendors – have jumped in the space to help orchestrate the continuous delivery pipeline, continuous delivery kinda picking up where CI ends.

    And that’s really where the CA Automic release automation comes in, where it is set to orchestrate and automate that pipeline from where CI stops, where the build basically ends and deposits thing in a secure repository, and then it helps move it all the way through to production. And the thing that CA, as well as other vendors who have entered this space and provide tooling for release automation, is that you’re trying to focus on having the same automation mechanics run in every environment, from dev all the way to production.

    And, indeed, that is one of the formal tenets of continuous delivery – to use the same automation mechanics. If you don’t do that, you’re still throwing things over the wall, right? And you’re missing an opportunity to test and QA and vet the automation mechanics themselves so that you can have high level of confidence that they will work in production because they themselves have been tested out dozens of times prior to that new release or that new change that’s being published.

    Shimel: Fair enough. Fair enough. Now, Scott, I guess this sort of begs the question, in order to understand, right, how powerful this concept is, can you give us kind of a little historical perspective? You know, what happens before pipelines? Before this concept of a software factory?

    Willson: Well, right. Things were kind of done manually. You would use Jenkins to run your builds, not just to compile your little change, but to recompile all the other pieces that it integrates with. You know, integration. The integration part of continuous integration. And make sure that changes that have been made aren’t going to break the aggregate app. And so, once that was done, it was really handed off to QA, and QA would do their thing. There were all these manual handoffs. Spreadsheets were being used to track the progress or charts. Microsoft Project and other PM tools were used to help migrate and manage, rather, the path to production, with a lot of manual instruction sets.

    I, previously, worked in a previous life, where production environments or staging environments would be handed a set of instructions that operations team had to follow. As you can imagine, there were always problems. There was always some little step, some little configuration thing, that was missed in those instructions. And so there would inevitably be errors. And so it was just very error-prone and failure-prone, actually, I would say, so CD is the evolution of this, of trying to move things in an automated, repeatable, predictable way, and a safe and secure way as well. And that’s kind of what brings us into this market.

    And I would say, you know, I was going through some of my old notes. A lot of people don’t know this, but, prior to Automic being acquired by CA, Automic was the one who really started this ARA market. And we had approached Gartner and Forrester, trying to explain to ’em that there was actually a need for this market, and I won’t reveal which analyst, but, you know, a conversation like, “Well, what is this thing you’re talking about? This need to automate?” And we said, “Well, we like to call it ‘application release automation,’ ” and the term stuck with the analysts. So what the analyst call “application release automation,” funny enough, actually came from us, as we were starting down this journey. And, in fact, when I started, I had – with Automic, and I was part of the group that helped build this whole thing out, is that – it was funny – you’d go to do meetings and I heard, again and again, all over the world, literally globally, Alan, I’d hear this comment – “Wow, we didn’t know anything like that existed.” So this is kinda where we started.

    Now, within two years, that changed. Within two years, there were several competitors, Nolio, UrbanCode, which both, like Automic, were acquired. And the world just kind of understands this whole concept nowadays, of continuous delivery automation. But, when we started out, this was new. And, you know, Automic actually was at the forefront and named this market, so I thought that was – that’s kind of cool to know.

    Shimel: Absolutely. Absolutely. You know, Scott, I was just sitting here, thinking, you know, we’re both around in this business a long time, and so, in the late ’90s, dot-com days, I helped start a company called “Interliant,” and we were one of the early ASPs. Right? And we did Lotus Notes hosting and PeopleSoft hosting, Oracle apps, some other defunct software that’s not around anymore, but this was before cloud. And we had to customize every installation, every – you know, ’cause the apps needed to be customized. And I’m just thinking back to how we were developing and deploying back then. And, you know, it seems like 100 years ago, not 20 years ago, first of all, but it’s amazing when you think about the changes we’ve seen in such a short time.

    Willson: Yeah, but you know what? It’s funny, Alan. So I remember those days as well, the pain of working weekends and doing massive deployments, but what’s funny now – we see all the things that have changed, but a lot of things are still remaining the same. In a lot of the large enterprises I visit, Alan, they’re doing this DevOps thing, the continuous delivery thing, with their new cloud-native-based apps or even they’re taking a big app, like their online banking app for a bank I won’t name, and they’re rewriting it in Node.js. Okay? Great. They do this since it’s in the cloud. They’re able to do all these new things – Agile, DevOps, continuous delivery – but they still have all this infrastructure in apps in the background, what I call “classically architected,” right? Your IIS, JBoss, WebSphere, WebLogic, so forth, client servers. You then also have your legacy – your mainframes and maybe large, packaged systems, like Sable and so forth.

    So, even though they’re doing a lot of things – we’re hearing a lot of messaging towards the new ways to do stuff – a lot of enterprises I visit still haven’t gone on that journey ’cause they haven’t quite moved so many things in the cloud and they’re still doing the way things, Alan, you and I saw done in the ’90s.

    Shimel: Mm-hmm.

    Willson: In some ways, a lot of it’s kind of this panic. This fear of like, “Well, you just never know, man. If this thing goes down, there’s so much money at risk and at stake,” that they’re slow to change. And I would tell you that, in the years since I first started my journey during all the stuff, some eight or so years ago, that it’s – even just last year, it was kind of – I don’t wanna say refreshing, I don’t wanna say surprising, but it was certainly eye-opening to see how many large enterprises fundamentally still do things the same way they did in the late ’90s.

    Shimel: Yep.

    Willson: And they’re asking, “How do we go forward? How do we do this?”

    Shimel: Agreed.

    Willson: So it’s not a given, all this new stuff, Alan, is what, I guess, what I’m trying to say.

    Shimel: No, it’s not a given. It’s not a given. But you also have to ask yourselves, Scott, right, fundamentally, is it a survival threat or threat to the survival of these companies who don’t kinda get with it, right? And what you see with that, Scott, is, look, there may be some industries and some companies and some businesses who, frankly, it’s just not that important. Right? Important. And they could do what they want and if they don’t get with it, they don’t get with it – that’s okay. And you need to – you know, and they may never adopt these ways of doing things and it may not really be a huge effect, but I think, for the overwhelming majority of these organizations, I mean, they really start falling behind the eight ball here, if they don’t get with it, and I think that’s what we’re seeing play itself out in the market today.

    Willson: Absolutely. It’s a danger if you don’t adapt. If they don’t improve their customer experiences, they could absolutely find themselves going out of business. I remember reading a lot about Bill Gates in the mid-90s and how he was just really maniacally focused and concerned with the concept that he could be out of business within just a couple to three years. And, when I was much younger and reading this, I thought, “Wow, that’s a good lesson to learn, one, but, two, I think he’s a bit over-exaggerating things.” Well, he wasn’t. Turns out, big companies, as we’ve seen since the year 2000, have vanished, and it’s very important that organizations who haven’t begun this move begin to modernize their way of delivering software. They just have to.

    Shimel: Got it. Got it. And I think we’re seeing that play out, right? Where it’s kind of “DevOps or die” or whatever you wanna call it. Anyway, Scott, we’re just about out of time. Before we close up, anything news coming out of Automic you can share with us?

    Willson: Well, obviously, we’ve got new stuff coming out, but I really think the new thing – it’s not that new, but one of the things I’m very excited about in joining CA is just this whole concept of a modern software factory, where – and joining CA, the Automic team is an integral part of this strategy and what they’re trying to help these large enterprises do, right? ‘Cause the enterprises, as I said, it’s just not the cloud-native stuff. They have all the classical and legacy stuff that they want to onboard for continuous delivery too and they’re struggling.

    And this whole thing about this modern software factory, I think it’s a bit more of a focus on the tooling, right? It’s another layer to DevOps and to continuous delivery, is looking at assembling and organizing the tooling to get all of your platforms moving at continuous delivery speeds. And it’s definitely possible, but I’m very excited and passionate about this modern software factory and where we can help CA take it. And I think that’s really a good mindset to have, is, yeah, this whole idea of you just plug in your inputs in one end, where your destination’s on the other end, and have things just happen and get you to the promise land.

    Shimel: Yep. I hear ya. All right. Well, Scott Willson, thanks for being this episode’s guest on DevOps Chat. Hope to have you back soon. I know you’re working on some stuff, Scott, and, at the right time, we’ll have you on here to brief the world on it, but, until then, it’s always great catching up with you. Lots of success over at CA and Automic and we’ll talk to you soon.

    Willson: Sounds good, Alan, and, at this time of the year, I will just say, good luck on your bracket. [Laughs]

    Shimel: Yes to everyone out there. Okay. Hey, Scott Willson, CA Automic Software, your guest today on DevOps Chat. This is Alan Shimel and you’ve just listened to DevOps Chat. Have a great day, everyone. We’ll see you soon.

    — Alan Shimel

  • DevOps Gets More Exciting in 2018

    DevOps Gets More Exciting in 2018

    As 2017 winds down and the IT industry prepares for 2018, there are a number of DevOps trends that everyone should be keeping their eyes on. It’s not so much that something incredible and explosive is expected in the new year. Rather, 2018 will be a time when DevOps starts to mature alongside other trends important to developers.

    These trends don’t guarantee organizations will be immediately successful with DevOps. Most still will struggle with the management aspects of DevOps, especially the need to keep feuding silos within IT working together as one happy group. Assuming developers, QA, and Operations can all learn to get along and act as a team, the technology supporting DevOps will continue to improve and the ecosystem will grow.

    Here are some of the DevOps trends that Amalgam Insights is following for 2018:

    Everyone Loves Containers and the Ecosystem Grows

    Containers have become a common technology in many IT departments. Although the majority of applications remain housed on single servers and virtual machines, container deployments are growing. Containers are an important enabling technology for DevOps—by creating small, transportable bits of applications, containers make it easy for IT silos to work from the same page.

    In 2018, growth in container deployments is expected to accelerate as large companies outside the software industry, such as those in financial services and health care, adopt microservices infrastructure. Containers that are especially suited to microservices will continue to expand rapidly.

    Increased container growth is also due to an expanding ecosystem. More companies have entered the market with products necessary to large-scale production applications. Products now exist for securing, managing, deploying and enhancing container services that weren’t available a few years ago. What’s more, containers are now available in all the major cloud services including Oracle Cloud, Microsoft Azure, AWS, IBM Cloud and Google Cloud. Containers are becoming enterprise-ready and large enterprises are feeling more confident using them.

    Takeaway: Growth in the ecosystem contributes to growth in container deployments while expansion of containers entices companies to enter the ecosystem. Containers have reached mainstream normalization and we expect to see fast expansion throughout 2018.

    Kubernetes Eats the Container Orchestration World

    In the container orchestration space, Kubernetes has become the tool of choice. Orchestration is the deployment and management of container clusters and is necessary to deploy them at scale. Kubernetes, originally a Google open source project and now managed by the Cloud Native Computing Foundation, has become the most popular orchestration tool available. That became obvious when Docker decided to distribute and support Kubernetes, despite having invested in its own orchestration system called Swarm. CoreOS, Microsoft Azure, Red Hat OpenShift, Docker and others have their own enterprise distributions as well.

    Kubernetes brings to the container ecosystem a relatively easy way to manage containers at scale. In large container deployments, containers are constantly being spun up, shut down, moved and copied. Once container clusters are in place, they must be monitored for security and service failure.

    During development, there will be a number of clusters that must be managed for development, test and final release to production. Doing this manually is close to impossible when even a moderate number of containers are in the DevOps pipeline. Kubernetes provides the tools and workflow to simplify these tasks.

    Takeaway: As container production clusters grow and as they are integrated into the DevOps pipeline, the need for orchestration becomes acute. Kubernetes has won the game and is now the tool of choice for orchestration. 

    The Pipeline is Getting More Integrated

    DevOps is often described of as a “pipeline,” In this context, a pipeline is a linear workflow that moves from build to test to release. As DevOps becomes more common in organizations, the pipeline grows to include additional test phases, monitoring and continuous improvement to create more of a cycle than a linear flow.

    One of the great impediments to making DevOps a reality is a lack of tools that support the entire pipeline. There are plenty of tools that help build applications, such as IDEs and code repositories from companies such as IBM, Microsoft, GitHub and Oracle, as well as dozens of open source products. Testing is well-supported with test data management and test plan management tools from companies such Atlassian, Informatica and Oracle. Release management and deployment products are widely available from Atlassian, Microsoft, IBM and many other companies. The problem lies not with the support for the individual steps in the pipeline, but with the integration of these tools to provide a complete end-to-end workflow. This creates a disjointed workflow, prone to errors and supportive of silos instead of collaboration.

    This is changing, however, and announcements late in 2017 herald toolsets that can provide a continuous DevOps pipeline. A great example of this is the newest versions of Microsoft’s developer tools that were debuted at Microsoft Connect(); in November 2017. Tying together the Visual Studio debugger with Kubernetes allows developers to test and debug as close to the production environment as possible and then deploy cluster to the test environment. That, plus the upgrades to Team Server, shows the commitment Microsoft has to creating integrated DevOps systems.

    Takeaway: While DevOps tools are still more or less disjointed, we are seeing a movement to fully integrated toolsets that enable the end-to-end DevOps pipeline.

    2018 is the Year of DevOps—Unless IT Management Gets It Wrong

    This coming year could be the year that DevOps truly takes off and becomes mainstream. Those who believe it is already mainstream now are fooling themselves. Management may believe their organization is implementing DevOps, but too many barriers still exist—especially management itself. IT departments remain heavily siloed and independent of each other, as do development tools. DevOps at scale requires, above all else, a collaborative and cooperative approach. Sadly, that does not yet exist in many IT departments. As tools reach maturity and best practices become common, managers become the inhibitors of DevOps. New management structures, incentives and thought processes need to be in place or the money spent on tools is wasted.

    Takeaway: All the DevOps pieces are in place except for the breaking down of functional silos that exist in IT departments. 2018 may be the year of DevOps if IT managers can create a new environment of cooperation.

    About the Author / Tom Petrocelli

    Tom Petrocelli is a contributing analyst with Amalgam Insights. His area of interest is collaboration and new ways of work, developer tools, IT project efficiency, governance and methodologies and DevOps. Most recently, Tom worked for a large, global banking corporation. Previously, he was the research director for Enterprise Social, Mobile and Cloud Applications at Neuralytix. He is an experienced marketing, technology and business executive with 30+ years in the computer technology industry. Tom’s background spans software engineering, systems architecture, IT, product management and marketing and general management. Connect with him on LinkedIn and follow him on Twitter.