devopsy.kinsta.cloud

Tag: gitlab

  • GitLab Open Sources Monitoring Tools

    GitLab Open Sources Monitoring Tools

    GitLab has decided to make its monitoring and observability tools available as part of its core open source platform to help reduce the total cost of DevOps. Previously, the tools were available as part of the commercial tools in GitLab’s DevOps platform.

    Kenny Johnston, director of product/Ops products for GitLab, said that as responsibility for observability shifts left, it is clear developers are now exercising more influence over which monitoring tools are employed. As developers begin to employ monitoring tools early in the application development process, most of them prefer open source tools because it’s unclear whether the application being developed will ever make it into a production environment, he said.

    However, if the application does make it into a production environment, it can be challenging for developers to make a case for standardizing on whatever monitoring tools they’re using in place of an existing commercial offering employed by an IT operations team. In addition to reducing costs, the GitLab approach could serve to eliminate any potential friction in the DevOps environment, as applications move from development into production environments.

    GitLab’s move to make its monitoring tools available as open source software comes at a time when more telemetry applications being employed to collect data are also based on open source software. In addition, developers of modern cloud-native applications based on containers are taking advantage of open source Prometheus monitoring software developed under the auspices of the Cloud Native Computing Foundation (CNCF) to monitor their applications. GitLab already provides integration with Prometheus.

    GitLab is also making it clear it expects its monitoring tools to benefit from the contributions of a larger pool of developers. Specifically, GitLab is calling for help in the areas of ongoing integrations with Prometheus along with improved logging, tracing and alerting capabilities.

    It’s not at all clear just yet to what degree DevOps teams will prefer to rely on open source versus commercial monitoring tools. While open source tools have some obvious advantages in terms of cost, many organizations have already standardized on commercial monitoring tools for a wide variety of existing applications. Open source tools often are more difficult to deploy and don’t tend to provide as much observability depth for IT operations teams. In addition, most of the providers of commercial monitoring tools are integrating their offerings with both open source telemetry software and Prometheus to bridge any divide that might occur between when an application is developed and deployed in a production environment.

    The one thing that is clear is a lot more individuals within the IT environments will be accessing these tools now that developers increasingly are being held accountable for the performance of their applications in production environments. In addition, modern applications require monitoring tools to make sense of all the dependencies that existing between microservices. Previously, IT operations teams tended to limit usage of monitoring tools based on commercial software to their most critical applications because of the costs involved. With the rise of open source monitoring tools, the opportunity to apply monitoring more broadly becomes more economically feasible.

    Regardless of the path forward chosen, the more visibility into IT environments provided, the easier it becomes to embrace best DevOps practices.

    — Mike Vizard

  • There’s No Such Thing as an Open Source Business Model

    There’s No Such Thing as an Open Source Business Model

    Open source technology works best when it’s adopted as a series of strategies and tactics that support a core business model—not the other way around.

    With all the talk about open source business models, the title of this article might seem a bit odd—especially when is written by someone whose entire job is to support an ecosystem surrounding an open source software project. Bear with me.

    We all know what comprises a business model. But when we focus a company’s products around open source technologies, we suddenly start acting like there’s something fundamentally different about the business itself. 

    While I agree that open source, as both concept and movement, has made a huge impact on our economy across nearly every industry, it’s my belief that many enterprise technology companies (especially startups) are losing sight of the difference between their business and the open source projects to which they are tied.

    Pressures on an Open Source Oriented Company

    Venture capital-backed startups in search of an open source business model quickly find themselves in a tough situation. Investors want returns, but the open source community expects free software and a healthy community. Other pressures, such as the natural erosion of differentiating features in a commercial version of the software through the community’s own development efforts, add to the stress. 

    Furthermore, the more the open source version of the software is used in the enterprise, the more likely that competition will emerge. The most challenging competition for software vendors is that hyperscale cloud providers can offer fully managed versions of the open source software. These cloud providers pride themselves on being quick to recognize the opportunity to support customer adoption of open source. This is intimidating to founders and investors alike, but attempting to create new “almost” open source licenses isn’t the answer.

    When It’s Done Right

    Instead of thinking of open source as a core component of your business model, define a business model that can stand on its own as a software or services company. Open source works best for a technology company when it’s adopted as a series of strategies and tactics that support an otherwise sound business model—not the other way around.

    I’m not the only person in this space that thinks this way. Earlier this year at the Linux Foundation’s Open Source Leadership Summit, I had the pleasure of seeing GitLab CEO Sid Sijbrandij share details about how the company he co-founded iterated and eventually landed on a business model that takes advantage of open source technology without being completely reliant on it.

    Open source is a shared approach to research and development among a community of individuals or like-minded companies working together to create an asset for anyone’s use. The entire notion of open source is predicated on the creation of a community for the purpose of developing software that can help organizations of many types and interests achieve their desired outcomes. This community—which includes your customers, end users of the open source software and even competitors—is where open source projects are able to create massive amounts of value. The trick for any company building, or helping to build, open source projects that are key to their commercial efforts is to focus on dual tracks:

    1. Support the growth of the community.
    2. Ensure you monetize a reasonable percent of the overall value that is being created.

    GitLab has been very thoughtful about its approach. While the core of GitLab’s product is open sourced, the DevOps lifecycle tool was built with multiple components that create a more valuable experience than any open source technology can deliver on its own.

    The critical challenge for GitLab—and any business, frankly—is revenue generation. Instead of selling itself as an open source vendor, GitLab has wrapped multiple proprietary features around its open core. That, in turn, allows GitLab to package proprietary features and support for that open source core as a more impactful and valuable proposition for enterprises.

    The goal is to create something that is unique and unlikely to be replicated or commoditized by other vendors. GitLab focuses on selling predominantly to executives and, to a lesser extent, it sells a lower cost version of its product to departmental managers. It also offers a free version to individual developers or small developer teams to drive adoption, interest and shared development of its open source core.

    GitLab made the bold decision to make a full-featured version of its Git-repository manager available for free to small teams. As with any software, once it’s adopted more broadly within an organization, new challenges will arise. That’s where a company can come in and offer enterprise-scale licenses, complete with features that solve problems that are unique to that wide adoption.

    That’s a very big difference in the way GitLab thinks about its business model, and it has led to a series of clear decision points about what it should make proprietary and what should be included in open source software. This approach allows them to build a community around open source, affords them ample goodwill, a groundswell of adoption and the benefit of tremendous developer feedback. It’s a perfect example of a company that uses open source for a set of strategic goals that supports a standalone business model. 

    Your Next Move

    It’s critical for some aspect of your business to be proprietary or well-differentiated from open source projects that are readily available. Adding capabilities, integrations, different features and enterprise class support will extend what an open source project can accomplish on its own.

    For example, if your product requires enterprises to put in a lot of work to adopt and implement it, it makes perfect sense to develop a strong services division as part of the hook to make your organization critical to the success of its customers. We should all be striving to solve difficult problems that are fairly unique to larger enterprises. 

    Remember: there’s no such thing as an open source business model. The business needs to make sense on its own. Work from the outside in, rather than the inside out.

    — Chip Childers

  • Dynatrace Extends APM Reach Into Business Analytics

    Dynatrace Extends APM Reach Into Business Analytics

    A growing number of providers of tools widely employed by DevOps teams that are extending their reach into business analytics has been joined by Dynatrace. The provider of a namesake application performance management (APM) platform unveiled a set of business analytics applications aimed at application owners who need to understand how changes to the  application experience their organization is providing ultimately impacts the business.

    Ajay Gandhi, vice president of product marketing for Dynatrace, said Digital Business Analytics applications from Dynatrace extends the single-agent architecture Dynatrace created for its Software Intelligence Platform to make it easy to employ an APM service delivered via the cloud into the realm of business analytics.

    The difference between the applications being made available by Dynatrace and traditional business intelligence (BI) applications is the agent software Dynatrace developed enables application owners to track all events in real-time, said Gandhi.

    As organizations launch digital transformation initiatives, Gandhi said many discover they lack visibility into the actual end user experience. The suite of Digital Business Analytics applications provides access to a set of applications that enables application owners to track transactions as well as metrics such as conversions and revenue performance by product, customer segment and geography.

    Gandhi noted organizations don’t necessarily need to have embraced best DevOps practices to derive value from the company’s analytics applications. However, those that have embraced DevOps are likely to appreciate the value of these applications because they view themselves as a software company that happens to build a specific product or digital service.

    It’s not clear to what degree application owners may be willing to give up relying on spreadsheets and legacy BI applications they have relied on for decades to manage the business. However, a case can be made for aligning the business to IT more closely by tracking the events in real-time as they occur across application environments and the IT infrastructure on which they depend. In fact, the one thing that has always differentiated Dynatrace is its ability to leverage an artificial intelligence (AI) engine, dubbed Davis, to surface events and deliver actionable intelligence by identifying the root cause of an issue, said Gandhi.

    Other DevOps vendors that are venturing into the realm of business analytics include CloudBees and GitLab. As these vendors increasingly realize that business processes now frequently revolve around DevOps processes, they all see an opportunity to extend their franchises by providing analytics applications that are aimed well beyond their existing base of IT users. Less clear right now is which set of tools within a DevOps toolchain will serve as the primary source of data for those analytics applications. However, Gandhi did note that unlike continuous integration/continuous deployment (CI/CD) platforms, APM services can provide real-time visibility into the actual digital experiences that change as various events impact production applications.

    Regardless of the ultimate source of data, however, the one thing that is certain is the level of insight business users can have into application development and deployment processes trends is, for better or worse, about to increase substantially.

    — Mike Vizard

  • DevOps Deeper Dive: DevOps Arms Race Heats Up

    DevOps Deeper Dive: DevOps Arms Race Heats Up

    A massive amount of capital is pouring into DevOps companies as organizations finally begin to realize the extent to which they depend on software. While Marc Andreesen may have penned his famous “Software Is Eating the World” opinion piece way back in 2011, it’s only now that many organizations are starting to realize they are software companies that also manufacture something or specialize in the delivery of some type of digital service.

    The most recent example of a massive injection of capital into the DevOps space came in the form of a $268 million investment in GitLab earlier this month. That move followed the acquisition of SignalFx, provider of an application performance management (APM) platform, by Splunk in a deal valued at just over $1 billion.

    Arguably, DevOps vendors have been gearing up for this fight for the better part of the last decade. Way back in 2014, New Relic picked up $115 million after going public. Three years later, Cisco Systems acquired AppDynamics for $3.7 billion.

    Now all these companies are racing to build the platform through which organizations of all sizes will transform themselves into true software companies by embracing best DevOps practices infused with large amounts of artificial intelligence (AI).

    DevOps as a concept has been around now for the better part of a decade. However, it’s only now that organizations are embracing DevOps across the entire application development life cycle. Previously, organizations embraced DevOps within isolated areas of the IT organization. Today, there is a much more concerted effort underway to apply DevOps practices to build and deploy software on an industrial scale.

    Achieving that goal, however, will require a lot more than simply adding higher levels of automation to a continuous integration/continuous deployment (CI/CD) platform. As business leaders realize how dependent the organization is on new applications to drive additional revenue, they are demanding more visibility into the software delivery process. Every delay to a software project now has revenue implications for the business.

    As more organizations come to that realization, it becomes apparent that monitoring software continuously as it is being built and after its deployed is now crucial to the business, said New Relic CEO Lew Cirne.

    “If mission-critical software not working, then the business is not working,” he said. “You must watch it all; you can’t pick and choose.”

    To become the de facto standard within an organization for monitoring those processes, DevOps vendors are racing to build DevOps platforms capable of operating at unprecedented levels of both scale and speed, said SignalFx CTO Arijit Mukherji.

    “The scale of the data has increased,” Mukherji noted. “Everything needs to happen in real-time.”

    DevOps platforms need to be extended to include support for everything from distributed tracing to machine learning algorithms, which organizations will rely on to identify the patterns employed to build AI models, he added.

    That level of investment requires a lot of additional capital that DevOps vendors either need to raise or, in the case of SignalFx and AppDynamics, tap into as a subsidiary of a larger company.

    The rise of microservices is further exacerbating those challenges as the number of dependencies within and between smaller segments of code increases. That fundamental shift in application architecture will require organizations to re-engineer DevOps processes across both the application and infrastructure levels of the entire IT environment, said Matt Cleve, director and global lead for Business iQ at AppDynamics.

    “Real-time visibility into the entire application stack is going to be a lot more important,” Cleve said.

    AppDynamics and parent company Cisco together will be in a unique position to deliver on that promise, he added.

    A recent survey conducted by the Harvard Business Review (HBR) on behalf of AppDynamics finds only 22% of IT organizations have access to any kind of real-time IT metrics.

    Widespread adoption of best DevOps practices has been a long time in coming. What is clear now, however, is that the organizations that take full advantage of DevOps by the end of this decade will thrive the most through the next.

    — Mike Vizard

  • GitLab Outlines Scope of DevOps Ambitions

    GitLab Outlines Scope of DevOps Ambitions

    At the GitLab Commit event in New York today, GitLab launched an ambitious effort to extend the scope of its DevOps platform to provide a view into the software development life cycle for not just developers but also designers, product managers and business leaders.

    Fresh off raising an additional $268 million in funding, GitLab now plans to create different ways to view a common pool of metadata and metrics generated by the GitLab DevOps platform.

    It may be a while before GitLab can deliver on that promise, but the core concept builds on the dashboards the company already has built for IT operations and cybersecurity teams. Rather than requiring each team participating in the software development process to acquire a separate platform, GitLab should be viewed as a DevOps platform delivered as a single application that can now be applied to everyone, said Mark Pundsack, vice president of product strategy for GitLab, in speaking with conference attendees.

    As part of that commitment, Pundsack noted GitLab is expanding its DevSecOps capabilities to include auto remediation. The scanning capabilities built into GitLab will be extended not only to detect issues based on the cybersecurity policies of an organization, but also, when allowed, automatically remediate routine issues, Pundsack said, adding GitLab also will automatically roll back applications if necessary. Once in place, GitLab will have pushed application cybersecurity much further left than anyone might have expected when DevSecOps was first broached as a concept, he said.

    At the conference, GitLab demonstrated a pipeline graph capability that will provide a foundation for many of these promised offerings. GitLab already can be employed to accomplish tasks such as code quality scanning, container scanning, dependency scanning and licensing dependencies.

    All these features are an extension of GitLab’s existing strategy that aims to rationalize the number of tools organizations need to stitch together to drive a set of best DevOps processes. However, other DevOps platform providers have similar ambitions.

    Thanks to its latest round of funding, GitLab—which also announced its plans to go public next year—is now valued at $2.75 billion. The company has not released any revenue results but does claim to support more than 100,000 organizations using its platform, including Ask Media Group, Charter Communications, Delta Air Lines, Goldman Sachs, Ticketmaster and Nvidia. GitLab also claims to have more than 4,800 active contributors to its open core platform. The company now has more than 800 employees in more than 55 countries across the globe, and there currently are 222 open GitLab jobs posted.

    It remains to be seen how much $268 million will enable GitLab to fend off rivals such as GitHub, which was acquired by Microsoft, and CloudBees, which has built a large DevOps community around the open source Jenkins continuous integration/continuous deployment (CI/CD) platform. Whatever the outcome of this arms race, the massive amount of funding pouring into the platforms soon should take DevOps platforms to places many might not have ever imagined possible.

    — Mike Vizard

  • An Inside Look at GitOps

    An Inside Look at GitOps

    At a time when variations on DevOps abound, get ready for the latest: GitOps. This new term promises to bring some order to what can be operational chaos and empower operational teams to leverage best practices learned from their developer counterparts. It’s early days yet for GitOps, but here’s what we think you need to understand about this new facet of DevOps.

    What is GitOps?

    Everyone we spoke with has a slightly different perspective on GitOps, but this is our take: In its most basic form, GitOps is infrastructure and operational procedures defined as code, and living in a source control system, such as Git. Git becomes the single source of truth for what is running in production (or staging, or development).

    Declarative systems, such as Terraform, AWS CloudFormation and Kubernetes help enable this practice; but operations is much more than Infrastructure as Code. True operations requires a way to plan, review and approve moves, adds and changes to the infrastructure. It also requires well documented procedures, or runbooks, to be successful. This is where GitOps differentiates itself. We believe GitOps is the full operations lifecycle of infrastructure management, living as code in a git repository.

    GitOps may be used for managing both application deployments and base infrastructure made up of systems needed regardless of the applications deployed (i.e. networking, security, etc.). Both will follow the same procedures to plan, review, approve and release code into production. Done right, GitOps will let engineers constantly apply updates to infrastructure meet all the needs of the applications in a continuous delivery format. The version control system ensures everything is recorded and visible and an audit trail keeps teams compliant. GitOps will make it easy to revert problematic changes, becoming a single source of truth about what is happening in the system from both the software development and infrastructure perspective.

    What Technology is Required?

    In order to kick off GitOps, you’ll use a Git-based version control system. Then you’ll need the ability to create a declarative cloud infrastructure, using something such as Terraform and Kubernetes, with the help of Helm or Flux. Because both Terraform and Kubernetes are declarative systems, they make it simpler to create and deploy infrastructure as code in a manner that is both replicable and transparent. And you’ll also want to implement a solution such as GitLab that wraps operational procedures around these tools. GitLab, being an end-to-end DevOps solution can manage the backlog of requests, assign those requests to the appropriate subject matter expert to create the change in Git, facilitate the review and approval process, and kick off the execution of the changes using its flexible CI/CD system.

    While it is possible to achieve this sort of workflow by using different solutions to merge and ship code to Kubernetes, GitLab is certainly a straightforward path to GitOps. It allows developers and operations administrators to work from the same source of truth and with a common set of procedures and tools.

    What Are the Benefits?

    In the DevOps world, GitOps is perhaps unique in that it can potentially benefit both operations team members and developers. For starters, GitOps aims to address the sometimes cave-like nature of operations (a single person working largely in isolation) by taking all of the best practices from software development and letting them loose on the deployment side.

    Full oversight, architectural review, project management and all the other pieces of the process are brought together in GitOps in an effort to end ops versus everyone else and get all the stakeholders involved in the process. The basics don’t change–an ops engineer will still create an infrastructure–but that will just be one part of a much larger process that everyone can contribute to and collaborate on. No more worrying about an engineer creating a cloud infrastructure that can’t be replicated; with GitOps everything is tracked making repeatability easy. Suddenly it will also be simple to answer the “what changed?” question because no one will have to hunt for an answer. The bottom line is a successful GitOps program will benefit the actual bottom line because operational processes will be dramatically streamlined.

    Possible Challenges

    With any collaborative effort, change can be tricky and GitOps is no exception to that. It is usually easier to create one-offs and not follow a process, so the first thing to consider about GitOps is it will require discipline from all participants. It will be vital for teams to write everything down.

    At GitLab we live and die by this thanks to our culture of collaboration, but we know it’s not necessarily something that comes naturally to any individual or organization. It will be vital for everyone on the team to record what’s going on in merge requests and issues. The temptation to edit something directly in production or change something manually is going to be difficult to suppress, but the less “cowboy engineering” the better GitOps will work.

    If GitOps sounds like something your organization might benefit from, our best advice is to start small. Pick a tiny area to begin with, get a win and then build on that. At GitLab we call that iteration and it’s worked well for us.

     

    This article was co-authored by Priyanka Sharma, director of technical evangelism at GitLab.

    — Brad Downey

  • GitLab Research Reveals DevOps and Cybersecurity Disconnects

    GitLab Research Reveals DevOps and Cybersecurity Disconnects

    DevOps and cybersecurity teams have a complicated relationship, which impacts code quality, research shows

    Finger-pointing has long been the name of the game when it comes to cybersecurity failures. After all, who should be responsible when so-called bad code or bad practices are blamed for the latest incident? 

    A recent survey by DevOps lifecycle tool company GitLab reveals a significant disconnect between application developers and cybersecurity professionals—a disconnect that could only lead to more security woes and additional finger-pointing. GitLab surveyed more than 4,000 app development and cybersecurity professionals and brought some light to the complicated relationship between both. 

    For example, some 49% of the cybersecurity professionals surveyed claimed they struggle to get developers to make the remediation of vulnerabilities a priority. Sixty-eight percent of those same security professionals believe less than half of the developers they work with can identify security vulnerabilities before new code moves into the test cycle. 

    While cybersecurity professionals like to point the finger at the developer community, there are two sides to the survey. Almost 70% of the developers surveyed said that while they are expected to create secure code, they get little help or guidance from the cybersecurity experts with whom they work. Perhaps the fault lies with the lack of mechanisms to make cybersecurity a priority in many organizations. “We don’t have clear guidelines about security, so the different services present different levels of security,” noted one developer in the report.

    While developers and cybersecurity professionals may be at odds, the report also revealed there is plenty of additional blame to go around. Case in point is the revelation that many organizations don’t make cybersecurity a priority. Some 44% of those surveyed reported they are not judged on security vulnerabilities in their code. That creates an environment where developers don’t have to consider cybersecurity issues when writing code.

    With developers, cybersecurity professionals and even the companies they work for all at odds about the responsibilities of proper cybersecurity hygiene, one has to wonder how tomorrow’s vulnerabilities will be prevented.

    “Our research tells us that while most developers are aware of the dangers that vulnerabilities present and want to dramatically improve their security capabilities, they often still lack organizational support for prioritizing secure code creation, increasing secure coding skills and implementing automated scanning and testing tooling to make that happen sooner rather than later,” said Colin Fletcher, GitLab’s manager of market research and customer insights.

    Solving the blind spots around cybersecurity is going to take a different approach, one that further tears down the silos created around cybersecurity teams and development teams. There is a little bit of irony in that observation; after all, DevOps is all about tearing down silos and creating functional teams. 

    It’s also an irony not lost on those at GitLab. The company promotes solving the cybersecurity issues by adopting a good DevOps practice, where cybersecurity becomes an integral part of the DevOps process. According to GitLab, incorporating cybersecurity into DevOps will have the positive result of the combined team being three times more likely to discover security bugs before code is merged into operations.

    The report offers some significant conclusions that support GitLab’s claims. For example, some of the security top findings include:

    • Sec teams are three times more likely to discover bugs before code is merged with a good DevOps practice in place, and are 90% more likely to test between 91% and 100% of code than in an organization with early-stage DevOps.  
    • Nearly half of security pros surveyed (49%) said they struggle to get developers to make remediation of vulnerabilities a priority.
    • Half of security professionals said bugs were most often found by them after code is merged in a test environment.  
    • Mostly remote teams are 23% more likely to have mature security practices than primarily office-based teams.
    • Only 44% report security vulnerabilities are a performance metric for developers in their organizations.

    GitLab also revealed there is another consideration, one of ownership. According to the report: 

    No part of software development raises more issues about ownership than the subject of security. The idea that everyone is responsible for security might be the ideal but it can also be part of the problem as everyone can easily turn into no one. Security professionals often complain about being on the outside, while developers and operations teams can resent being told how to prioritize their work. Our survey responses indicate developers are taking more responsibility for security, but of course a lot of work remains. In fact, only 20% of those surveyed rated their organization’s security efforts as good, while 36% said they were fair and 24% said they were poor.

    — Frank Ohlhorst

  • GitLab Expands Scope of DevOps Ambitions

    GitLab Expands Scope of DevOps Ambitions

    GitLab, with the release of its 12.0 update, is extending the reach of its namesake continuous integration/continuous delivery (CI/CD) platform to include both cybersecurity professionals and traditional IT operations staff.

    John Jeremiah, senior product marketing manager at GitLab, said the goal is to provide organizations with a single CI/CD platform that unifies the management of everything from source code to best DevSecOps processes.

    To address that latter requirement, GitLab 12.0 adds a dashboard designed specifically for security personnel along with the ability for cybersecurity teams to approve merge requests and automatically apply patches for vulnerabilities detected. Those capabilities complement existing support for scanning and secrets detection.

    New capabilities embedded in the platform for IT Operations professionals include an operations dashboard and an embedded incident management capability that makes it easier to create a coordinated response to a specific event.

    GitLab 12.0 also adds support for feature flags to better manage the application delivery process, along with a platform-as-a-service (PaaS) environment to provision and maintain the infrastructure needed for developing and launching an application.

    Jeremiah said the new and existing capabilities in GitLab will make it possible for organizations to unify their DevOps tool chains in a way that also enables IT operations teams and cybersecurity professionals to be included in the process.
    At the same time, GitLab is making DevOps more accessible by eliminating many of the silos between various tools that have emerged over the years, including, for example, built-in support for feature flagging versus requiring organizations to acquire and maintain a separate tool, he noted.

    Because those capabilities are embedded in the platform, organizations will be able to make the cultural shift required to adopt DevOps more easily, he added. IT teams don’t always know what other tools they will need as their DevOps skills mature. With all the tools embedded in GitLab readily available, it becomes easier for organizations to expand their DevOps processes as the organization starts to implement more sophisticated processes, Jeremiah said.

    That approach is starting to resonate with the 100,000-plus developers actively using the GitLab platform and more than 2,000 contributors actively contributing code to the underlying open source platform, he added. GitLab in March saw its millionth merge request.

    It’s too early to tell how broadly any CI/CD platform might be able to consolidate the DevOps toolchain. Developers are notorious for adopting new tools as they see fit. Internal IT teams, of course, are not always excited to be asked to support yet another tool. The right level of balance between ultimate flexibility and the need for a standard set of processes that can be implemented across multiple application development teams must be determined by each organization.

    In the meantime, DevOps processes are starting to expand beyond traditional development team to now include IT operations staff and cybersecurity personnel. The challenge now that they’ve been invited to the proverbial DevOps party is making them feel welcome and appreciated.

    — Mike Vizard

  • Using Spark and Jenkins to Deploy Code into Hadoop Clusters

    Using Spark and Jenkins to Deploy Code into Hadoop Clusters

    This article illustrates how to deploy an application into a Hadoop cluster using GitLab/Jenkins/Spark application.

    In this article, we will cover the following:

    • Auto-triggering Jenkins build when code is committed in Git.
    • Pulling the latest code from the repository through Jenkins.
    • Building, compiling and packaging the “.jar” using SBT tool.
    • Uploading the .jar to the Nexus repository with defined version.
    • Pulling the versioned jar and dependent files from Nexus repository to server.
    • Deploying the versioned “.jar” file through Spark job into Hadoop cluster.

    Getting Started

    The following must be performed first:

    1. Download and install Java 1.8.
    2. Install Jenkins.
    3. Integrate GitLab and Jira.
    4. Install SBT.
    5. Create the Hadoop cluster: Hadoop and Spark.

    End to End Steps in Detail

    When a developer pushes code to GitLab, Jenkins will trigger a predefined job using webhooks.

    The Jenkins job will pull the code from version control using Git; it builds the code and makes the package as .jar file using the build tool SBT. This .jar file can be deployed into a Hadoop cluster with the help of a Spark command.

    Once the deployment is completed in the Hadoop cluster, the application will start running in the background.

    Initially, we can see the current running application in Hadoop (see Figure 1). It has an application ID, which is in a running state.

    Figure 1
    • If we click the tracking UI, which is Application Master, it will navigate into the Spark streaming page.
    Figure 2
    • In Figure 2, we can see batches are running in 40 seconds. For this scenario, we are changing from 40 seconds in the code.
    • When a developer makes changes to the code, he will commit in his local machine and test his code, then push to remote repository. I have updated steaming time from 40 to 50 seconds and committed.
    • If we can specify the Jira task number in the commit message, the same comment will be updated in the Jira task as below.
    Figure 3

    In the Jira application, we can see our commit message as comments.

    Figure 4

    GitLab and Jira integration process:

    • Install Git plugin to pull code from GitHub repository and SBT plugin to build and make a package as a .jar file in Jenkins.
    • Configure the Jenkins job under source code management and provide the Repository URL along with credentials and branch details.
    Figure 5
    • Optional: Set webhooks to automatically trigger the Jenkins job when the push happens in GitLab.
    Figure 6
    • In the “build” section, add the parameters clean compile package in the Actions field, which will build the code and create a .jar file.
    Figure 7
    • Once the job has been triggered in Jenkins, monitor the Jenkins console for the next step and errors, if any.
    Figure 8

    Optional: If we have configured SonarQube, it will analyze the code files and provide the result in the SonarQube dashboard. This allows us to know the code quality, code coverage vulnerabilities and bugs to improve efficiency of code.

    • Optional: To maintain the versioning, we can upload our *.jar into Nexus/Artifactory.
    • Once .jar has been created, we can keep it in a specific folder and trigger a Spark job. This can be performed by triggering a Spark command from Jenkins. Execute the below command to submit spark job:

    # spark-submit –name appName –master yarn –deploy-mode cluster –executor-memory 1g –driver-memory 1g –jars $(echo $dependencyJarDir/*.jar | tr ‘ ‘ ‘,’) –class com.coe.spotter.DQModule –conf spark.driver.extraJavaOptions=../src/main/resources/log4j-yarn.properties –conf spark.sql.warehouse.dir=./spark-warehouse –conf spark.yarn.submit.waitAppCompletion=false app.Jar

     
    • Once deployed, we can see the application ID number has been increased.
    Figure 9

    Timeout in the Spark streaming job has been increased to 50 seconds.

    Figure 10

    — Nodagala Nagarjuna

  • Digital Transformation Technology and Process: More Similar Than You Think

    Digital Transformation Technology and Process: More Similar Than You Think

    Right around this time of the year, the gym starts to get a little less crowded. Throughout January, it’s packed to the gills with folks freshly motivated by their New Year’s resolutions to get in shape. But by mid-February, the commitment seems to have all but dissipated.

    Why is it so hard to just commit?

    During my last workout, it occurred to me that the set of vacant treadmills next to mine was eerily reminiscent of the state of enterprise DevOps. According to Forrester, 50 percent of digital transformation efforts stalled in 2018. It seems commitment and follow-through are difficult in both personal and professional arenas. Admittedly, adopting and scaling DevOps is no easy feat. While reflecting on what does work, I found some parallels between the transformation process and DevOps itself.

    What Success Looks Like

    According to DORA’s “2018 State of DevOps report,” organizations with the highest level of DevOps execution, as measured by throughput and stability, also achieved the highest level of business performance in terms of profitability, market share and customer satisfaction. In a true case of going fast without breaking things, elite performers realized impressive performance gains:

    • 46 times more frequent deployments (throughput)
    • 2,555 times shorter lead times (throughput)
    • 7 times lower change failure rate (stability)
    • 2,604 times shorter time to recovery (stability)

    These metrics underlie a key tenet of DevOps transformation, which is breaking up large chunks of work into many small pieces.

    Shrinking the Change

    Smaller changes mean getting work out faster, with less risk. They take less time to plan, less effort to execute and are easier to troubleshoot. But, smaller code changes alone aren’t meaningful unless they are coupled with shorter release cycles. In essence, agile without CD doesn’t work.

    Stockpiling a few months worth of agile code iterations into a single monolithic release reintroduces an enormous amount of risk and negates much of the work done to adopt agile software development in the first place. Big queued-up releases often mean someone is stuck working the weekend to iron out all of the things that inevitably go wrong and no one should be stuck with weekend release duty. As Jez Humble put it, “The most important metric for continuous delivery is the number of person hours spent outside of business hours doing releases… and the goal for that is zero.”

    In practice, CD is what brings agile development to fruition; it’s why Gartner finds that 40 percent of IT departments have already invested in CI/CD with a full third planning to invest in 2019. CD is all about small pieces. In the same way agile enables smaller code changes, CD enables smaller releases.

    Not Just Smaller, but Iterative

    At GitLab we use the phrase, “just commit” as a shorthand for continuous delivery and deployment. The idea is that when you have a completely automated CI/CD pipeline, all a developer needs to do is commit code to their branch, and everything else needed to get that code running live in a production-like environment—from containerization to security testing—is automated. Similarly, getting code to production is as easy as merging the feature branch back into trunk with the entire production deployment automated. The message to the developer is, “Just commit your code and let your DevOps tooling do the rest.” This is CD in a nutshell.

    With this type of automation in place, the cost of making a change significantly decreases. This means the cost of getting started is very low, but so is the cost to keep going. Instead of spending a massive effort to map an entire path to your solution, you simply need to figure out your next step and keep iterating. According to GitLab’s “2018 Global Developer Report,” 67 percent of developers report the top challenge to getting work done is unclear direction. By shrinking the change, business leaders will find it easier to set direction for the next step without having to map the entire path. This brings us back to digital transformation.

    Commit to Digital Transformation

    Just like a monolith being decomposed into microservices, the practice of breaking large chunks into small pieces applies to driving digital transformation as well. Like Creighton Abrams said, “When eating an elephant, take one bite at a time.”

    How can enterprises shrink the change?

    For most organizations that achieve successful digital transformation, the initiative usually starts with a single team or innovation unit. This group proves the success for the rest of the organization. It develops that culture, tools and processes that other parts of the organization can then adopt. One example of this principle in practice is the transformation that Chris Hill, Head of Systems Engineering, Infotainment, led at Jaguar Land Rover. When Chris came on board, only three build engineers had the ability to release to production and it took four to six weeks to complete each release. Chris gradually scaled a CI/CD platform, step-by-step over two years, until every team across the organization was able to complete a release in less than 30 minutes. Today, the Jaguar Land Rover completes 50 to 70 deployments per day. This iterative pattern is seen over and over again with enterprise digital transformation. It’s encouraging because it means successful digital transformation can be driven by either executives top-down or engineers bottom-up.

    Keep the Fuel Burning

    According to Argyle, “91 percent of companies have at least started on digital transformation, but only half have created an organization-wide strategy.” This seems to align with Forrester’s metrics on half of the organizations that stall.

    So, how can organizations keep going and avoid the stall out?

    If I can make a confession, I am also one of those New Year’s resolution makers who just started working out in January. How it is that I’m still hitting the gym while others have dropped off? I have found that taking a “DevOps” approach is a key to my success. Instead of broad sweeping changes, I make small incremental improvements one day at a time.

    It turns out the strategy for getting started—“just commit”—is the same strategy to keep going.

    By starting small with a single project and gradually expanding scope in iterative steps, an organization can continue to improve over time at a rapid pace. Small, single-commit code changes coupled with CD increase throughput and stability. In the same way, small, incremental process improvements bring about the greatest success with digital transformation.

    — William Chia