devopsy.kinsta.cloud

Tag: pipelines

  • DevOps Database Platforms Slash DX Bottlenecks

    DevOps Database Platforms Slash DX Bottlenecks

    An enterprise with many databases and various data requirements for DevOps continuous delivery presents a problem. For example, test data provisioning is a challenge today, particularly in pre-release testing of database updates. Standard practices involve shared database environments that need to be maintained from one test to the next, slowing test cycles and generally not fitting into an automated CI pipeline. As I explained in A New DevOps Database Platform is Needed, at best, it makes things chaotic; at worst, problems with database integrations with DevOps is a gross impediment to overall organization agility—which undermines the very purpose that DevOps is meant to serve.

    A recent whitepaper, A New Kind of DevOps Database Platform, indicates that there are multiple reasons why a new kind of DevOps database platform is needed, which are:

    • Compatibility requirements
    • Integration with current CI/CD pipelines
    • Integration with portfolios of value stream management
    • Infrastructure cost efficiency
    • Productivity for developers, testers and operations
    • Security and governance
    • DevOps performance

    The paper explained that a DevOps database platform requires the following capabilities to satisfy these requirements.

    • Automated database provisioning: the ability to deploy a database using automated procedures.
    • Database cloning: writable database environments are delivered on demand, based on a built versioned image.
    • Distributed repository: database images can reside on Linux and Windows servers, as well as storage systems.
    • Simple declarative builds: a simple, predefined method to make the creation of versioned data images simple, while enabling user/group access controls, data masking and other data engineering functions.
    • Scalable data cloning: database clones are served on demand, with many simultaneous clones served from a single image.
    • Comprehensive database support: the repo supports all common relational, NoSQL, machine learning and file-based data stores.
    • Versioned databases: each image becomes a versioned, immutable DevOps artifact.
    • Deploys equally to all cloud environments: DevOps databases can be deployed and operated on any on-premises infrastructure or cloud.
    • Cross-platform:  DevOps databases can be deployed and operated on common operating system environments such as Windows and Linux.
    • Security features including access management, data masking and encryption: database images easily incorporate access control, data masking and synthetic test data via SQL scripts, batch files or command line tools.
    • Compatible with CI/CD platforms:  DevOps databases can be deployed and operated equally well in multiple CI/CD platform tools such as Azure DevOps, Jenkins and CloudBees.
    • Compatible with containers and container orchestrators: DevOps databases can be deployed and operated equally well in multiple container environments such as Docker, Kubernetes and Red Hat OpenShift.
    • Deliver writeable data at the speed of DevOps: Deliver writable databases in seconds.
    • API for easy integration with DevOps tools: An API is available to enable integration of the DevOps database control plane with other tools.
    • Audit logging: The DevOps database includes a complete audit log of operations.

    Transforming to the New Database DevOps Platform

    A DevOps database platform in an enterprise requires more than simply buying and installing software and telling everyone to use it. With multiple entrenched database platforms already in place, there are practical and political considerations that must be overcome. Introducing any change to a large group of applications in an enterprise can be chaotic, and will often be more costly and take too much time if the changes are not compelling.

    To be sufficiently compelling, the database DevOps platform needs to be affordable, easy to understand, easy to set up and use with a wide variety of CI/CD pipelines. The following will be sufficiently compelling benefits to drive adoption:

    • Versioned, secure database environments or images managed as a DevOps artifact.
    • Images can be built and managed by the DevOps team, without requiring new infrastructure.
    • Multi-DB and multi-TB data environments in seconds.
    • Database environments can be served on demand for a CI/CD pipeline.
    • Lower-level database environment infrastructure is streamlined, with a single VM capable of supporting many simultaneous database environments which drives an immediate cost savings and positive ROI.
    • Data security and governance benefits with versioned database environments, with built-in and auditable security policies.

    What This Means

    Digital transformation is imperative for modern enterprises and organizations to remain competitive. However, the transformations of many enterprises and organizations are seriously impeded by the challenges of integrating databases into their DevOps application pipelines and value streams. A new DevOps database platform can allow the databases to be treated like any other DevOps artifacts that are orchestrated and automated within the DevOps toolchains.

    Suggested resources for automating database environment delivery include Windocks, Cohesity and other storage appliances that provide virtualized (cloned) database environments that can be adapted for DevOps purposes. I want to particularly acknowledge input from Windocks, who has introduced a DevOps data repo that features database virtualization on standard Linux and Windows servers, for versioned data environments and CI pipelines and cloud-native support for data delivery for containers, Kubernetes and conventional instances.

  • JFrog Extends Reach and Scope of CI/CD Platform

    JFrog Extends Reach and Scope of CI/CD Platform

    At its online JFrog swampUP Conference today, JFrog announced it has extended the reach of its ability to distribute software to include options based on peer-to-peer and content delivery network (CDN) mechanisms.

    In addition, JFrog expanded the reach of the JFrog Pipelines continuous integration/continuous delivery (CI/CD) platform, enabling DevOps teams to build their own library of reusable pipeline steps via an Extensions Development Kit (EDK). DevOps teams can encapsulate frequent and common actions in their pipelines that can run sequentially or in parallel. That approach also serves to reduce the need to rely on plugin modules to extend a JFrog delivery pipeline.

    Finally, JFrog has added another free public artifact center that provides a means to manage Helm charts, which are widely employed to deploy applications on Kubernetes clusters. ChartCenter is intended to make it easier to deploy microservices based on containers.

    JFrog CEO Shlomi Ben Haim said all three additions to the JFrog portfolio extends the ability for users to manage the entire DevOps workflow from integration to delivery regardless of what type of software is being deployed or where it is installed.

    As DevOps teams look to deploy software on edge computing platforms within an internet of things (IoT) environment, Ben Haim said they will need to be able to employ either a peer-to-peer or CDN-based approach to manage that process efficiently. Existing mechanisms are not designed to handle large volumes of artifacts that need to be delivered in a specific order. The new mechanism added to the JFrog CI/CD platform enables DevOps teams to achieve that goal without having to acquire a separate platform, he said.

    In fact, Ben Haim noted, efficiently delivering software to edge computing platforms represents the next great leap forward for DevOps. Achieving that goal, however, will only be practical if the entire CI/CD workflow process keeps track of all the dependencies and metadata end-to-end, he said, noting that approach will enable endpoints to pull only the delta code required to update the software after an application is initially deployed.

    There is no doubt extending best DevOps practices to edge computing platforms will be a major challenge. Most DevOps teams already struggle with CD because each platform software is being deployed on is unique. Because of a general lack of standardization in terms of potential endpoints that might be deployed at the edge, that “snowflake” issue is likely to become even more exacerbated as IT environments become more distributed than ever.

    In general, Ben Haim said far too much of the DevOps focus continues to be on source code management when in reality the biggest issue IT organizations face is how to efficiently manage binaries. At a time when organizations are making massive investments in digital business transformation, Ben Haim noted the rate at which software is deployed and updated is about to accelerate dramatically.

    As such, Ben Haim said it’s now only a matter of time before DevOps workflow processes will need to be able to scale well beyond their current limits.

  • CircleCI Extends Core CI/CD Platform

    CircleCI Extends Core CI/CD Platform

    CircleCI today in partnership with Plandek launched an analytics plugin dubbed Buildtrakr for its namesake continuous integration and continuous delivery (CI/CD) platform. Buildtrakr integrates with CircleCI and Git to synthesize commit and build performance data, with critical metrics being surfaced via a dashboard.

    In addition, CircleCI has revamped its user interface and reconfigured a new fleet of Docker images, also known as CircleCI convenience images, to optimize them for a continuous integration environment.

    CircleCI CEO Jim Rose said Docker images made available by CircleCI are faster, easier to download and provide a more efficient way to build custom images. CircleCI also released a new category of convenience images available for languages such as Python, Ruby, Rust, Go and several others, that come with improved documentation, more stability and more efficient infrastructure. Those convenience images are individual open source projects and can be used in IT environments other than the CI/CD platform from CircleCI.

    Finally, the company also announced that its insights endpoints now offer start and end date parameters for workflows and jobs. These parameters restrict the range of executions returned based on their start times in addition to providing a new filter to use for testing analysis.

    Rose said the new user interface is significant because it makes it possible to view the status of entire pipelines more easily while making it easier for more advanced DevOps teams to leverage workflows.

    The redesigned interface also adds a filterable view with nested workflow and job information, making it easier to scan the status of recent builds. Pages were also overhauled to deliver more digestible job details, full-screen step outputs for easier debugging and a cleaner configuration-as-code view. There’s also been a 65% reduction in page load times, noted Rose.

    In general, Rose said activity on the CircleCI platforms in cloud environments is on the rise in the wake of the COVID-19 pandemic. Organizations that are working remotely due to stay-at-home directives and can’t access a local data center have come to realize that having a build box located under someone’s desk is no longer a viable option.

    Of course, it remains to be seen how much of that initial burst of activity will result in new applications being deployed. However, with more organizations starting to build and deploy digital business applications to engage customers, it may be only a question of when IT operations teams will be able to catch up to a building backlog of applications. In fact, many of those applications are now likely to be deployed on the same cloud where they were constructed.

    Whatever the path forward, there most certainly will be a greater reliance on metrics. Naturally, the hope is that analytics will be used to coach DevOps teams to become more efficient versus simply riding roughshod over developers. After all, the goal isn’t to write more code faster but to deliver better code at the right time.

  • JFrog Makes Free DevOps Resources Available for COVID-19 Projects

    JFrog Makes Free DevOps Resources Available for COVID-19 Projects

    JFrog is providing free access to cloud-based DevOps tools to teams or organizations that have launched projects to actively research and fight the COVID-19 pandemic.

    Ido Green, vice president of technology for JFrog, said this initiative will be made available via its FrogCare program mainly to nonprofit organizations. The goal is to make it easier for teams launching these projects to embrace DevOps practices to more easily sustain these applications for the long term, he said.

    Thus far, JFrog is supporting a mix of short- and long-term projects that address various aspects of the pandemic, he said, adding JFrog is not so much interested in the quantity of the projects it supports as much as the merits of the research project.

    Specifically, JFrog will provide access to cloud-based tools such as Artifactory, Xray and Pipelines with no time limits. Organizations participating in the program will get 5,000 build minutes and 250GB of storage space for free each month along with 500GB of free data transfer. Those teams will also receive free support. If they exceed quotas in a given month, JFrog promises to not shut off access to its services.

    Green said JFrog is in a better position to help these projects because many of them are already using various application development tools. The modular architecture of the JFrog platform makes it easier for those teams to extend those tools using a DevOps platform that is designed to meet each team where they are in terms of application development and deployment maturity, he said.

    JFrog expects most of these applications will be accessing massive amounts of big data in the cloud, Green noted. As such, most of these applications will be built using modern cloud-native stacks of software that make it easier to scale workloads up and down as needed. Of course, many of the organizations building these applications may not have the greatest appreciation yet for best DevOps practices. However, free access to platforms such as JFrog should also make it easier for DevOps practitioners to donate their time and expertise to help build and deploy these applications. In many cases, application development projects in vertical industry segments impacted by the pandemic have been suspended or terminated altogether. In the short term at least, there should be more DevOps practitioners with some extra time on their hands.

    Longer-term, it’s also clear that health care and life sciences will emerge as hotbeds of application development and deployment for years to come. DevOps specialists who want to move into that industry sector might benefit from volunteering to work on a research project. JFrog also envisions that over time it will serve as a conduit through which multiple teams working on these projects around the globe could share insights and best practices.

    It’s not clear to what degree the COVID-19 pandemic will transform IT in general and DevOps in particular. What is certain is that the amount of focus on preventing future pandemics both inside and out of IT is creating a unique opportunity to make a unique DevOps contribution.