devopsy.kinsta.cloud

Tag: continuous integration/continuous delivery

  • Buildkite Adds Analytics Tools to Identify Flaky App Tests

    Buildkite Adds Analytics Tools to Identify Flaky App Tests

    Buildkite has added an analytics tool to its continuous integration/continuous delivery (CI/CD) platform that identifies flaky tests.

    Buildkite’s co-CEO Keith Pitt said the company’s Test Analytics tool enables continuous performance monitoring and real-time insights for test suites that, when not properly written, only serve to waste DevOps teams’ time. Rather than eliminating those tests, most DevOps teams just opt to run them again on the assumption that the code they are testing will eventually pass because the test is poorly constructed, noted Pitt.

    Buildkite reports there are more than 800 million seconds—or the equivalent of 25 years—of flaky tests manually re-run on its platform by developers each month.

    In addition to being integrated with the Buildkite CI/CD platform, the Test Analytics tool can also be applied to tests residing in CI platforms such as CircleCI, GitHub Actions and Jenkins.

    Test Analytics includes a dashboard that identifies the slowest and least reliable tests to help prioritize test remediation efforts. Historical views and trend graphs are also available to provide further insight into recurring issues.

    DevOps teams can also take advantage of user-defined thresholds to define whether a test is slow or unreliable, which generate custom alerts when discovered. Finally, there are integrations with commonly used testing frameworks to determine where tests are, for example, wasting time running database queries against application programming interfaces (APIs) that have timed out.

    Pitt noted that most tests today are created by developers and software engineers rather than dedicated testing teams. As such, the chances there will be an issue with a test that is returning flaky results are high.

    Organizations are, of course, trying to determine how far left to shift application testing. Developers can write their own tests, but how thorough and objective those tests are is debatable. There are also certain tests that should be applied to all applications that organizations can embed within their DevOps workflows to make sure they are always run.

    Obviously, the more testing that occurs the fewer issues DevOps teams will encounter after an application is deployed. However, when tests are flaky, it becomes easier for developers to justify not running them even though that test might still surface an issue they should address. In addition to increasing confidence in the quality of those tests, knowing which tests are flaky allows a DevOps team to determine which tests should be run first. That’s critical at a time when more organizations are trying to increase developer productivity; that productivity is often adversely impacted by waiting for tests to be completed before more code can be written.

    Eventually, the overall quality of tests should improve as machine learning algorithms and other forms of data science are applied. In the meantime, however, the onus for improving application testing is clearly on the engineers that create them. The challenge is many of those engineers may not even be aware how flaky a test really is until someone shows them some empirical data that can’t be disputed.

  • What is DevSecOps?

    What is DevSecOps?

    While traditional development practices have long separated security and compliance, DevSecOps as a series of best practices integrates security into every phase of the DevOps software development life cycle. DevSecOps introduces and automates security in the earlier phases of the software development life cycle rather than bolting it on at the end. The approach saves money, saves time on tedious manual tasks, helps organizations meet regulatory compliance requirements and significantly reduces the risk of critical security bugs being found after an application’s final build. DevSecOps promotes the use of, for example, active penetration testing, security audits and other security tools within an agile development process.

    The goal of DevSecOps is to create a collaborative environment between developers and security professionals that enables organizations to build secure code faster and more easily. By emphasizing security from the very beginning of the process, it becomes a priority in the app dev process rather than an afterthought. Over time, developers become more familiar with the common weaknesses in software that today result in more insecure applications being deployed than most anyone really wants to admit. The end goal is to create secure applications by making it easier for developers, security experts and operations professionals to collaborate throughout every stage of application development.

    Minimizing Application Vulnerabilities

    It may seem tedious and expensive to integrate security into the development process, but it’s become essential for organizations to embrace DevSecOps to minimize the number of application vulnerabilities that cybercriminals might exploit. When applied throughout the development process, scanning and reporting can take place within a matter of minutes rather than holding up deployment at the end of the process to review the entirety of an application.

    Leaving security to either the end of the application development process or, worse yet, after an application has been deployed only increases the total cost of ownership. It’s much more expensive to remediate a vulnerability after an application has been deployed than it is to address an issue during the application development process.

    Implementing DevSecOps requires organizations to set up workflows that are managed via a continuous integration/continuous delivery (CI/CD) platform. Most critically, developers must understand the role security plays in enabling organizations to identify vulnerabilities as early as possible in the application development life cycle.

    Inserting security audits and penetration testing into the development process, for example, helps ensure the security of an application. When changes are made in a project, scanning, architecture reviews and penetration testing can be triggered within the development flow to save time and resources.

    DevSecOps Adoption

    However, there are many technical and cultural challenges ranging from tool integration to a lack of trust between developers and security teams that can impede the adoption of DevSecOps. Security professionals are tasked with identifying and preventing vulnerabilities in applications. Acceptance test criteria, user designs and threat models should be created by security professionals. The development team then needs to define a code review system to ensure uniformity.

    Despite those concerns, the most important thing is to simply get started. Ideally, development and security teams should work together to create a safe application development and software development environment. There is no need to create separate teams or apply different policies.

    The biggest benefit of DevSecOps is that it eliminates silos between development and operations. As a result, software engineers integrate cybersecurity processes from the start of the development process. This includes ensuring that every component, configuration item, and installation process is securely patched and documented. This concept of shifting security to the left allows the security team to identify and remediate security threats early on.

    In the longer term, however, adopting DevSecOps best practices will ensure security remains a top priority in the wake of a series of high-profile cybersecurity breaches. In short, there is now more focus on DevSecOps with the goal of securing software supply chains.

  • 7 (More) Security Considerations for CI/CD

    7 (More) Security Considerations for CI/CD

    CI/CD orchestration has become increasingly popular in DevOps practices. It facilitates faster deployment of software and product delivery. Through the use of automation when building, testing and deploying applications, CI/CD bridges the gap between development, operation activities and teams. 

    However, there are many security considerations to keep in mind with frequent deployment. From quality automation to code history analysis, security must be baked into your CI/CD pipeline to ensure your product and processes remain safe from hackers.

    To gain insights on the top security considerations for CI/CD, I asked the speakers and sponsors for the upcoming SKILup Day as well as several DevOps Institute Ambassadors for their thoughts. In this two-part series, I will be sharing their insights. Did you miss part one? You can read it here. Here’s what they had to say in part two of this series:

    Speaker, Ravi Lachhman, Evangelist, Harness

    “CI/CD platforms are core to an organization’s engineering efficiency and typically have elevated privileges to sensitive environments such as production. Since CI/CD platforms typically will be deploying, this makes sense. Delving into the SolarWinds hack with binary artifact signing, etc., can separate security concerns into two parts. One concern would be the security around the actual CI/CD platform and one around the DevSecOps goals inside a CI/CD pipeline. With so many items shifting left, having proper role-based access control (RBAC) is crucial. With the separation of duties for regulated environments and the need for engineers to be efficient, controls that ensure only certain entities can deploy and certain entities can view pipelines is important. Around the DevSecOps movement, container/OSS scanning is still top of mind for many firms.”

    Sponsor, Basis Technologies, Eran Kriegshauser, Basis Technologies’ SAP DevOps Architect

    “The question you should be asking is ‘How do I maintain quality with the frequent deployment tempo demanded by business today?’ We need automation of quality and guaranteed rigor. For example, take promoting security configuration changes in systems, applications and products (SAP). Today, this is done by SAP transports where the change is created in an SAP development system, then imported into QA and, ultimately, into production. How do you ensure that a developer can’t give themselves extra security access through a role modification? Automated analysis and approval workflows must support segregation of duties requirements. Users can’t approve changes they create–the appropriate CI/CD process will not allow it.

    Another area is automatic peer review notification on changes to sensitive enterprise resource planning (ERP) objects. Once identified, the promotion of those object changes are halted until approved by a team lead or change and release manager.”

    Savinder Puri, DevOps Evangelist, Zensar Technologies

    “Infuse as many aspects of security as possible into your CI pipeline. CD is already too late for security—you’ve already deployed the application into an environment.

    Start from the basic static code analysis, security scanners, etc., and then keep maturing. Whatever tools you integrate into CI Server, ensure the corresponding integrated development environment (IDE) plugins are made available to developers. That way, devs can do the first-level validation within their IDE itself, and do it as they code, even before compiling. That’s the practical implementation of the often overused term ‘shift left.’”

    Vishnu Vasudevan, Head of Product Engineering and Development, Opsera

    “Now with cloud migration, emerging technologies and open source technology, companies will have to make sure they have the right tooling to measure and take action based on the security challenges born out of different development activities. 

    The focus will need to be on managing the risk that is tied to the business criticality. Each industry, domain, technology, etc., all have their own independent security goals, and you cannot have one security goal across the enterprise for different technologies and different domains. It is important to evaluate the technology being used and weigh the security requirements accordingly.”

    Stephen Walters, Sales Engineer, Everbridge

    “Inclusion! This is why the term DevSecOps came to be—because, erroneously, security was not included as part of a DevOps transformation, and the same is true for transformation of an SDLC to incorporate security.

    Security has still been treated as a non-functional requirement; an afterthought. If there is one thing we should learn, there is no such thing as a non-functional requirement. Security needs must be considered on an equal footing with all other requirements and built into the design, test, deployment and delivery of any feature.”

    Mark Peters, Technical Lead, Novetta

    “CI/CD pipelines should implement security at every step.  Tests can be automated in tooling to conduct tests of various functions at multiple levels for code accuracy, unit tests through fuzzing and integration or vulnerability testing. Most vulnerability testing in a pipeline is actually patch management, ensuring all commercial and proprietary holes are filled for release software. The first step should be using coding checks to make sure there are not any common problems with the code. While this may seem easy, if you are not managing that basic step, you cannot move ahead to the more complicated areas.”

    Pawel Piwosz, Lead Systems Engineer, EPAM Systems

    “At a high level, I would say that security is still outside the main scope of CI/CD. We should consider a few areas and different angles, here. Security should be not only implemented by CI/CD; CI/CD should be implemented in a secure way. Everything-as-code is a solution here: policy-as-code, security-as-code, drift-as-code, and if we talk about infrastructure, remediation-as-code.”

    If you’d like to learn more about the latest in CI/CD and similar topics, check out the upcoming SKILup Day schedule. Or, learn more about the benefits of becoming a DevOps Institute Member. 

  • A New DevOps Database Platform is Needed

    A New DevOps Database Platform is Needed

    The diverse number of database platforms, deployment environments and data variants needed to support the plethora of applications and services required for rapid DevOps continuous delivery lead times, coupled with the challenges of integrating database changes for application pipelines, is a growing challenge for many organizations as they strive to accelerate their DevOps transformations.

    When you stand back and look at the bigger picture, an enterprise with many databases and various data requirements for DevOps continuous delivery presents a problem. At best, it makes things chaotic; at worst, it is a gross impediment to overall organization agility—which undermines the very purpose that DevOps is meant to serve.

    In my book, Engineering DevOps, I explained that databases offer several challenges for CI/CD pipelines.

    In the simplest case, when a database is used solely by one application or microservice, the database pipeline can be integrated with the application pipeline to ensure its changes use the same version management system and change controls to ensure the database changes are kept in sync with the application. However, many database platforms are not as responsive to changes as the applications themselves and become a bottleneck for the pipeline.

    If a database is used by several applications, each with its own DevOps pipeline, then changes to the database must be coordinated with all applications that depend on it. Maintaining a separate database pipeline linked to the application pipelines through CI/CD toolchains, as a strategy, is overly complicated and can often introduce additional bottlenecks and synchronization issues.

    Another challenge with database pipelines is the management of data in the database during the CI and CD stages. Data in production needs to be persistent and is ever-changing, even while the underlying database code is changing. During the CI process, sample data from production needs to be used to ensure testing is realistic, and new test data must be introduced to verify new database capabilities are working.

    During the CD stages, rollbacks to prior versions of database code must avoid rolling back to versions that are no longer compatible with production data. This is a key reason why database changes must follow a disciplined approach that fully tests database changes with applications prior to full deployment. Rolling deployments, in which applications and database changes are deployed gradually in separate clusters, can minimize the blast radius in case of problems. The required coordination of separate systems adds complexity and bottlenecks.

    So, why not simply integrate database changes as artifacts within the same DevOps pipeline as the application?

    Gilad David Maayan’s September 2020 article DevOps Database Strategies and Challenges: Automation, Scalability, and More explained some of the specific challenges that arise with integration of traditional databases with DevOps.

    Database tools were not designed for integration with DevOps tools. These tools tend to be database- and environment-specific, which can make it difficult to incorporate into pipelines designed for flexibility.

    Database performance and ease of management often do not perform well in DevOps environments where multiple applications need to use the database in a short time, such as in the accelerated lead times for DevOps continuous delivery. This is especially true with larger-scale databases. If an application change must sync with a database schema, changing the pipeline can become complicated especially if there is a need to sync across multiple application pipelines to meet continuous delivery goals.

    While DevOps processes are proven for continuous delivery of applications, incorporating database changes into the DevOps pipeline is hindered by large data sets, limited database automation and security concerns. As a result, database development and testing continue to rely on outdated processes, with fixed and often shared instances refreshed from backups. A new DevOps Database platform is needed to provide highly dynamic and efficient cloud-native database solutions for modern enterprise digital transformations, and that platform must include DataOps use cases for DevOps, DevSecOps, continuous testing and governance. This new kind of data repository must be designed to integrate seamlessly with DevOps CI/CD pipelines through flexible APIs and cloud-native architectures.

    Given the points above, traditional database solutions and parallel data pipeline strategies are not sufficient to meet the challenges of enterprise DevOps transformations. What is needed is a new kind of DevOps database platform that allows the database, and database changes, to be as agile as any other artifact in the DevOps pipeline.

    Paul Stanton, VP of product management at Windocks, in his June 2021 YouTube video Windock DevOps Data Repo, said, “Imagine terabyte class database environments managed as DevOps artifacts with entire versioned databases available in seconds.” That’s what we need.

    The following are examples of the capabilities required for such a new DevOps database platform.

    • Automated database provisioning
    • Distributed repositories support
    • Simple declarative builds
    • Dynamically scalable
    • Virtual hard drive images for a wide variety of DBs – Linux, SQL, Oracle and others
    • Distributed versioned database environments working on standard file systems and storage systems delivery writeable data at the speed of DevOps – in other words, in seconds.
    • Deploys equally to all operating systems and cloud environments, Docker containers and Kubernetes.
    • Many clones per virtual machine (VM)
    • Security features including access management, data masking and encryption
    • Compatible with and easily integrates with CI/CD platforms
    • Compatible with multi-cloud (AWS, Azure, GCS)
    • Built-in data security and governance into DevOps data.

    I will explain these more completely in a subsequent article.

    What This Means

    Digital transformations are imperative for modern enterprises and organizations if they are to remain competitive. However, the transformations of many enterprises and organizations are being seriously impeded by having too many database platforms and tools that do not integrate well and that bottleneck continuous delivery goals. A new kind of DevOps database platform, in which the database is treated as any other artifact, is needed to resolve technical problems with database integration to DevOps application pipelines. A new DevOps database platform solution is needed to accelerate time-to-value and realize efficient, safe operation for digitally transforming enterprises and organizations.

  • 8 Security Considerations for CI/CD

    8 Security Considerations for CI/CD

    In the software development enterprise, CI/CD refers to the combined practices of continuous integration and either continuous delivery or continuous deployment. CI/CD enables organizations to bridge the gap between development, operation activities and teams by the use of automation when building, testing and deploying applications. 

    How do you maintain quality and security with frequent deployments? While CI/CD can improve the efficiency of an organization, there are many security considerations to take into account.

    To gain insights on the top security considerations for CI/CD, I asked the speakers and sponsors for the upcoming SKILup Day as well as several DevOps Institute Ambassadors for their thoughts. Here’s what they had to say in part one of this two-part series:

    Speaker, Grant Fritchey, Product Advocate, Redgate Software

    “There are a lot of security considerations when it comes to CI/CD. As a person focused on data and data management, I’m going to say that the top security consideration is ensuring that you have compliant data for your non-production environments. Just the sheer number of people who will have access to the information means that you have a huge number of possible vectors for a breach if you’re exposing protected information.”

    Speaker, Anders Wallgren, VP of Technology Strategy, CloudBees

    “Secure your software delivery pipelines themselves, not just the software that you deliver with those pipelines. Everyone should have only the level of access they require in order to perform their role.

    Maintain a Software Bill of Materials (SBOM) so you know what you have in production. When a new vulnerability is disclosed, you will have a much easier time deciding whether and how to mitigate it if you don’t have to chase down what you have in production. Your orchestration software should do the lion’s share of the work here.”

    Tracy Ragan, CEO and co-founder, DeployHub

    “CI/CD tools are driven mainly by one-off scripts. There is absolutely nothing secure about one-off scripts. From build through deploy, we need to start thinking of implementing automation and policy-driven tools and dump the one-off scripts.”

    Tiffany Jachja, Technical Evangelist, Harness

    “Supply chain management is a major security consideration for CI/CD. If an entity gains access to a CI/CD resource or component that they shouldn’t, this can lead to vulnerabilities, downtime or even malicious code. For example, in the SolarWinds hack of 2020, attackers legitimized their software malware by injecting it into the SolarWinds build process. This build pipeline produced trusted software artifacts and obtained digital certifications before being released to over 18,000 customers worldwide. After the hack, SolarWinds took action, further restricting access to its build environment and reviewing its build process, but not before it was too late. It’s not only about automating developer and operations workflows but also how we treat and handle CI/CD pipelines.”

    Jamal Walsh, Technical Product Owner, The Very Group

    “You must make sure you implement security tooling within your pipeline; it’s not enough to just use CI/CD to build and deploy applications. Steps in your pipeline should include security testing tools like SAST, DAST, dependency checks, etc.”

    Martin Chan, System Analyst, Global Executive Consultants Ltd. 

    “The most important thing is to define the access control list and rules to control the access in the CI/CD pipeline so that the audit log can easily trace issues. Also, the system should require authentication and define password policies for all users.”

    Bryan Finster, Software Engineer, DoD Platform One

    “Security must be automated as part of the pipeline. Just like any other quality gate, the pipeline must be the single source of truth for security. Any manual process after code is committed increases the effort to deliver. Increased effort increases the size of each change. Larger changes are harder to check for quality and fail more spectacularly. Manual security is just security theater and is easily bypassed or forgotten. That effort is better spent improving the ability of the pipeline to use security as-a-service.”

    Dheeraj Nayal, Global Community Ambassador, DevOps Institute

    “Follow the three security best practices for CI/CD. We usually divide security practices into three parts that you can address with different kinds of solutions. These are:

    1. Secure pipeline configuration
    2. Code and Git history analysis
    3. Security policy enforcement

    All three aspects are equally important.”

    If you’d like to learn more about the latest in CI/CD and similar topics, check out the upcoming SKILup Day schedule. Or, learn more about the benefits of becoming a DevOps Institute Member. 

     

  • GitLab Releases Massive Update to CI/CD Platform

    GitLab Releases Massive Update to CI/CD Platform

    GitLab has updated its continuous integration/continuous delivery (CI/CD) platform with a raft of capabilities spanning everything from value stream management to cybersecurity.

    In addition, GitLab announced it is making generally available Gitaly Clusters, which enable DevOps teams to create a warm replica of a Git repository.

    Brendan O’Leary, senior developer evangelist at GitLab, said with release 13 GitLab continues to build on a single code base that allows the platform to be managed and accessed as an integrated set of applications rather than disparate components that DevOps teams need to pull together and then manage.

    In terms of core DevOps capabilities, the latest release adds the ability to customize the Value Stream Analytics module to specific workflows. GitLab is also planning to make it possible to visualize stages of a workflow.

    GitLab is also improving traceability between requirements and has added the ability to visualize the workflow associated with a specific release. A future release will enable DevOps teams to employ feature flags to conduct both A/B testing based as well as create feature flags from merge requests and to filter feature flags by status, according to the company.

    Other capabilities added include more customizable dashboards, with security dashboards now being shareable outside of GitLab to foster collaboration with IT security teams. Future releases will add Kubernetes Clusters to the dashboard to show all clusters and pods in use.

    DevOps teams now can route existing monitoring alerts to GitLab for triage and incident resolution. At the same time, GitLab has moved design management to the core platform.

    GitLab is also now making AutoDevOps, which provides pre-defined CI/CD pipelines, available on Amazon Web Services (AWS).

    In terms of cybersecurity, GitLab has revamped the way the platform manages vulnerabilities to make it easier in a future update to prioritize remediation efforts.

    GitLab also has static application security testing (SAST) support for .NET frameworks and has expanded support for offline environments. There’s also now support for dynamic application security testing (DAST) for REST application programming interfaces (APIs) and access to full commit history scan for secrets. Fuzz testing support will be introduced later this year, according to the company, along with an ability to secure secrets across services. GitLab also plans to add integration with security tools that track container behavior analytics, as well as an out-of-the-box container network policy.

    Thanks to GitLab’s collaboration with Okta, there is now also tighter integration with Okta’s System for Cross-domain Identity Management, which can be provisioned for a GitLab group.

    In terms of compliance, GitLab also plans to enable DevOps teams to establish a compliance framework, adopt its regulatory controls and simplify audit reporting. The company is also working on an initial security policy user interface to simplify setting up security guardrails.

    Finally, GitLab has become a Common Vulnerabilities and Exposures (CVE) ID Numbering Authority (CNA). GitLab users can request a CVE from GitLab, either for GitLab itself or for any project hosted on GitLab.com. In the future, users can request CVE IDs directly from within GitLab.

    O’Leary said GitLab is working toward increasing developer confidence in their security skills. By making it easier to embrace best DevOps practices, developers no longer will have to tiptoe around security issues that they today largely have little to no familiarity with, he said.

    In general, O’Leary said GitLab’s biggest challenge is getting organizations to move beyond simply using it to manage source code. Once organizations do make their first foray beyond source code management, however, it’s not too long before they are employing a wide range of modules, he noted.

    Of course, encouraging organizations to dive deeper DevOps has as much to do with the culture of the organization as it does the capabilities of the CI/CD platform employed.

  • CloudBees Advances Continuous Delivery

    CloudBees Advances Continuous Delivery

    At the DevOps World | Jenkins World 2019 conference, CloudBees released an update to the continuous delivery (CD) platform it gained via its acquisition of Electric Cloud, signaling the company’s commitment to investing in a platform that’s employed widely in the enterprise.

    CloudBees CEO Sacha Labourey said the rechristened CloudBees Flow builds on the application release orchestration (ARO) platform to further automate the last mile of a DevOps process.

    Version 9.1 of CloudBees Flow provides a path to increased visibility into how a release is progressing, including the ability to view the inventory being deployed to a given environment and deeper insights into all the sub-pipelines that make up an overall release.

    The latest release also provides more granular control over release pipeline execution, customizable dashboards within the platform’s Release Command Center and data archiving capabilities that make it easier to comply with regulatory mandates.

    Labourey said that while companies that have developed commercial software-as-a-service (SaaS) applications tend to have very mature DevOps processes, most enterprise IT organizations are still defining the best DevOps practices they want to implement across their organization end to end. CloudBees Flow provides a continuous delivery framework for automating the deployment of applications at enterprise scale that complements the Jenkins continuous integration/continuous deployment (CI/CD) platform, he said.

    Labourey noted the goal is to eventually unify continuous delivery processes across multiple platforms, including the ARO platform provided by Electric Flow, the Jenkins CI/CD platform and the Jenkins X and Spinnaker platforms that are optimized for building and deploying microservices-based applications on top of Kubernetes. As part of that effort, earlier this year CloudBees became a founding member of the Continuous Delivery Foundation (CDF), an arm of The Linux Foundation dedicated to foster best continuous delivery practices.

    Most organizations that have adopted CI/CD platforms have seen varying degrees of success in taking advantage of continuous integration to accelerate the development of their applications. Continuous delivery, however, has proven to be a much bigger challenge. Organizations tend to struggle with everything from making sure they have the appropriate infrastructure resources available to support a new release of an application to debugging application code before and after it’s deployed in a production environment.

    It may be a while before most enterprise organizations truly master DevOps processes end to end. However, given how dependent on software most organizations are, increased reliance on DevOps processes has become a prerequisite for any form of digital business transformation. As the volume of software being developed increases, the rate at which updates to applications are being made also increases substantially. Most organizations are not going to be able to keep pace with that rate of change by relying on legacy software development and deployment processes.

    In many more cases now, however, the challenge organizations are wrestling with is not so much whether to transition to DevOps, but rather determining which DevOps processes to standardize on to satisfy the elevated expectations of business leaders anxious to provide customers with that next great application experience.

    — Mike Vizard