devopsy.kinsta.cloud

Tag: open source

China’s Open Source Activity Surged in 2020

Activity among China-based open source project contributors and creators surged in 2020, as Alibaba, Huawei, Tencent and other leading Chinese tech companies are playing an increasingly important role in the open source development community worldwide.

According to the recently released Open Source Contributor Index (OSCI) ranking, as of March 2021, Alibaba, Huawei, and Tencent collectively ranked in the top 20 worldwide for the first time. Rankings are based on the total number of contributors and contributions on GitHub. Microsoft, Google and Red Hat ranked first, second and third, respectively.

Alibaba, as well as China-based Pingcap, came within striking distance of reaching the top 20 worldwide open source ranking for 2020 measured in activity; the total number of forks, commits, stars, pull requests, issue comments and other metrics, according to the GitHub 2020 Digital Insight Report released in April.

Authored by X-lab in conjunction with East China Normal University, Kaiyuanshe, Shanghai Open Source Information Technology Association and other unnamed “scientific research institutions and open source community institutes,” the X-lab report revealed Pingcap’s eponymous Pingcap (a distributed relational database) and Alibaba’s AntDesign (a Microsoft React framework) activity metrics of 210 and 193, respectively, in 2020. Meanwhile, jwasham (an open source study plan detailing how to become a software engineer) ranked 20th worldwide with an activity metric of 234.

Open Elastic’s Kibana, recently forked by Amazon Web Services (AWS) and Kubernetes ranked first and second worldwide with activity metrics of 698 and 602, respectively.

Torsten Volk, an analyst for Enterprise Management Associates (EMA), says the increased popularity of and activity in China-based open source projects are proof that the concept behind open source software actually works.

“This is the classic ‘one plus one adds up to more than only two’ theory, and that applies to most companies anywhere in the world. And if it was possibly to gain advantages by only using open source code but not contributing, companies anywhere else in the world would have done so right from the start,” says Volk. “Remember that the core idea behind open source software is not to give out your secret sauce for other companies to enjoy for free, but to spend the least possible amount of funding on building commodity code.”

What, if anything, China-backed open source projects means for the community worldwide — as well as China’s influence in the industry in general – remain to be seen. Tencent, which entered the top 20 in March ranked by contributors in the OSCI study, for example, has become a leading supplier of components for machine learning (ML) databases in, for example, Kubernetes and cloud-native environments.

“This is great for everyone, and there is no benefit for anyone to purposefully not contribute to a project they benefit from. The benefit comes from your own contributions, in terms of reputation and credibility gain, across your industry. This motivates others to also contribute, as there is little to no benefit in keeping your own code secret while self-serving from the GitHub buffet,” says Volk. “As in the case of Kubernetes, your own company’s differentiation needs to come from the manner your proprietary apps [integrate] on top of this platform, or from the way you are able to integrate Kubernetes clusters with machine learning databases and other elements higher up the stack. And that’s why open source works.”

May 5, 2021
5 Steps to an Open Source DevOps Strategy

Open source technology thrives during times of mass disruption, and this pandemic has been no different. Across the board, companies are accelerating their digital transformation and DevOps strategies, looking to open source technology as a way to streamline operations. While deploying open source software (OSS) in DevOps can help organizations lower costs and avoid vendor lock-in, the tools are often not backed by an ecosystem that makes them transferable across the enterprise, leading to interoperability issues across the board and creating a haphazard workflow.

Before embarking on an open source migration, companies need to plan out their strategy. Following these five steps will help companies create an organized plan and a seamless open source stack so that they can capitalize on the lower costs and tap into the benefits of the open source community.

Determine if you Can Self-Support Your Strategy

A recent Gartner study determined that at least 75% of open source policies don’t yield effective results. This stat is largely due to companies believing that they can self-support their open source stack. However, as technology becomes increasingly complicated, companies should take a step back and evaluate if they actually have the right resources and skill sets to get the job done. Enterprise architects with open source expertise are a niche group that’s in high demand, so don’t worry if you don’t have the resources – there are plenty of trusted vendors out there who have a proven track record of delivering results.

Consolidate

Fewer vendors equal fewer problems. When evaluating potential partners, keep in mind that working with a few key partners (versus several) will be beneficial in the long run by ensuring that applications will work together seamlessly, while also avoiding vendor finger-pointing if they don’t. Assess potential partners to see which packages, application stacks and infrastructure they support, and what level of freedom you have to switch out various components. Avoiding vendor lock-in is key, as switching vendors will be a costly undertaking if you ultimately decide to go in another direction.

Invest in Security

Security is a major concern in enterprise open source software, and it’s critical to make sure you have the ability to act quickly on potential issues and vulnerabilities. Consider partnering with a vendor who specializes in OSS security, and can keep clients informed of the latest security updates and bug fixes to ensure that your program isn’t vulnerable to security threats. Having a security specialist on standby is a must-have for organizations so that when security is breached, it can be remediated immediately.

Pick the Right Tools

Not all tools are alike. There are plenty of options out there that can help elevate your open source strategy, but consider how each will affect your overall program. At each phase of your pipeline, test the open source tools selected to ensure that they can integrate together within the broader toolchain and every layer of your software stack, including databases, platforms and monitoring solutions. If you have questions about their connectivity, either consult with an expert who understands how the software will truly behave in various environments or engage with free apps that help developers determine if their tools are compatible across the broader stack. Doing so will help enterprises choose the right open source technology early on, saving both time and money.

Adapt

Open source implementation in DevOps is more than a one and done approach; it requires adapting and tweaking in order to stay relevant, keep a competitive advantage and optimize new technology. Ensure that your open source strategy can be improved to include new offerings and services that the company will implement down the road and that your vendors—or internal team—can modify the framework fairly easily when necessary.

While open source technology continues to drive innovation in the DevOps industry, including it in a company’s DevOps strategy requires thoughtful planning. When done right, an open source policy will save a company time, money and people power, allowing them to collaborate seamlessly with the greatest minds in the industry.

May 4, 2021
Are Developers Responsible for Open Source Governance?

There are lots of factors in the open source software world converging to make it a big year for “shift left” in software development. Heightened security concerns, an increasing need for software supply chain visibility and the growth and complexity of open source ecosystems will continue to push the responsibility for ensuring code is secure earlier in the engineering process as far left as the request and design phases.

This means 2021 will be yet another year in which developers are challenged to take on more responsibility, and pushed further onto the front lines of open source governance. Developers will need access to the right training, resources and tools to do this job. But they also can’t be successful if they’re expected to do it alone. Shift left can’t mean shifting all responsibility to the developer as the sole protector of secure code and the only go-to open source expert. The reality is that the depth, breadth and complexity of open source governance in 2021 demands a cross-functional team and layered approach.

Equipping Developers With Governance Knowledge

Let’s start with how to better equip developers and other stakeholders in the governance process, including legal, security, IP compliance and product people. These teams need enough knowledge to be aware of the top compliance issues, and their corresponding remediation options, to be active contributors in this rapidly evolving space. Research from Forrester shows that of the top 40 computer science programs in the United States and the top five programs internationally, none include open source licensing and security coding practices in the curriculum.

Instead, it has fallen to the industry itself to impart this expertise on the job. Leading companies are packaging open source training programs into something that resembles standard HR-led and managed training, including check-ins and quizzes to test and solidify knowledge, and regularly updating content and scheduling trainings to make sure people stay abreast of changes and challenges. In the spirit of open source, many industries work collaboratively to share best practices on governance in this regard, with industry experts speaking on the topic at other companies. Others are leveraging existing management training courses, such as the ones offered by the Linux Foundation.

The growing importance of the Software Bill of Materials (SBoM) also points to the need for cross-functional collaboration. Software supply chain visibility will become even more crucial in 2021 so that companies can build and sell secure and safe products, as well as ensure compliance. More people across the organization will need to access and view the SBoM in ways relevant to their functional needs. As a result, its assembly is transforming from being assembly-line-esque (passed along and viewed in the context of siloed business functions) to becoming much more collaborative. The SBoM is becoming akin to a business intelligence tool, providing the ability for different functions and stakeholders to filter for relevant information.

Depending on your function, you may be interested in a top-down view of the open source and third-party components across your portfolio of products in order to understand risks in the organization. For instance, as an engineering leader, you may want to understand if you are impacted by a new security vulnerability, where in your portfolio of products you may be exposed and the current status of remediation work for that vulnerability. If you are in legal, you may be interested in a significant licensing change for a component in use at your organization. This allows you to be prepared with changes to legal policies for automated reviews and obligation fulfillment guidance for engineering when the next build occurs and a more recent version of a dependency is pulled in. If you are in the export control function, and are required to provide your application’s encryption capabilities as part of the certification process, understanding which open source components are part of the application, and which provide cryptographic capabilities is essential. Finally, don’t overlook your customers, who today are requiring open source disclosures as part of commercial software contracts to protect themselves from insecure software.

Collaboration has also been made simpler by the fact that new industry standards now enable an asset portfolio view of open source security. Open Chain obtained ISO certification in late 2020, ISO/IEC 5230:2020, providing a clear and effective process management standard for open source license compliance. It allows companies of all sizes and in all sectors to adopt the key requirements of a quality open source compliance program. Coupled with ISO/IEC 27001–the de facto standard for managing security in information security management systems (ISMS) – organizations can now take a more holistic view of all of their software assets in an effort to make the information they hold more secure.

Open Source Program Office

The best way to ensure developers aren’t left footing the bill for open source governance is to create a centralized team dedicated to it. Savvy organizations could form an Open Source Program Office (OSPO), which brings together key stakeholders in legal, engineering, product and security to develop, communicate and operationalize open source strategy for their entire organization. This team works to define a framework and process for consuming open source software as part of their development work (inbound use case) and/or contributing code to the community via open source projects (outbound use case). The mission and main charter of these teams is to help their organization create and manage an open source strategy in terms of the adoption, use, support, participation and development of open source software. However, they’ve also found their role has evolved to become the experts on open source internally and evangelists of the technology and its use.

The bottom line is that the developer’s job is to develop software – and every piece put in place for stronger open source governance means less time spent focusing on the problems that could be in the open source code and more time on the promise of using it to rapidly innovate for the benefit of their customers.

April 27, 2021
WhiteSource Acquires Diffend to Secure Open Source Supply Chains

WhiteSource this week announced it has acquired Diffend as part of an expended effort to discover malware that has been deliberately injected into open source software by a contributor acting in bad faith.

Company CEO Rami Sass said cybercriminals are increasingly trying to compromise software supply chains that today depend heavily on open source projects. The Diffend platform surfaces anomalous contributor behavior that would be indicative of someone trying to compromise the integrity of a project, said Sass.

Previously, WhiteSource provided a tool that would discover malware that had been inadvertently included in open source software. WhiteSource now plans to add the Diffend tools, now known as WhiteSource Diffend, for free to a portfolio of security tools that many organizations have already incorporated into DevSecOps workflows. Earlier this month, the company raised an additional $75 million in financing that, in part, was used to fund this acquisition.

As cybercriminals begin to target software supply chains, many of them are taking aim at open source software that is likely to be adopted by a broad spectrum of organizations, noted Sass. The challenge is that bad actors have become adept at embedding malware in software components in ways that are difficult to detect, he said. However, spikes in activity from, for example, a new contributor to the project might warrant additional investigation, Saas added.

These type of attacks, unfortunately, will require the maintainers of open source software to vet more closely who is allowed to contribute to the project, added SaaS. New contributors that might have once been greeted with open arms need to now be initially met with a modicum of paranoia, Saas said.

That level of suspicion will be considerably higher if that contributor is located in a country that has a reputation for trying to compromise the security of both adversaries and companies that have intellectual property they might want to surreptitiously acquire.

In the wake of some recent high-profile breaches involving software supply chain compromises, organizations are revisiting their application development processes. Cybersecurity teams are now asking developers to vet any external code that is incorporated within an application. Most applications regularly incorporate both open source and commercial components that developers assume have been vetted for malware when, in most cases, they have not. The Diffend platform, for example, aided in the detecting of 60 suspicious packages found in Rubygems, an open source platform that makes it easier to package software built using Ruby programming tools.

It’s unclear how increased inspection of software components may impact application developer productivity. However, given how dependent organizations are on software today, the discovery of malware downstream that compromises the security of the application is potentially catastrophic.

Sass said it’s doubtful these security concerns will lead to decreased reliance on open source software in general. There is no real alternative for efficiently driving software innovation, he said. The challenge now is finding a way to make sure all that innovation continues to occur as securely as possible.

April 23, 2021
Is Open Source More Secure Than Closed Source?

There’s no getting around the fact that security is one of the most essential factors in day-to-day operations for most software developers. From proprietary code to sensitive customer data and everything in between, there are hundreds – if not thousands – of crisis scenarios that can result from gaps in security.

Data security is an issue that every company faces, no matter how large or small. Developers look for security solutions that follow today’s data and application security best practices in order to reduce risk and provide the most secure experience possible to the end user.

A common debate among both users and developers is whether open source software is more or less secure than closed source alternatives. While there are a variety of different schools of thought as to what makes for a truly secure environment, there is good reason to believe that open source software is the superior option for those looking to prioritize security.

While open source code may not be inherently more secure than its closed source brethren, here are a few reasons why it almost certainly can be — so long as you take the right approach.

Open Source Offers True Transparency

Both well-established tech giants and up-and-coming startups work with some of the brightest minds in the development community to bring valuable proprietary software products and services to market. At the end of the day, however, users have no choice but to trust what a vendor says about how a particular piece of software actually “works.” Since you can’t see the code itself, there’s no way to really verify whether or not what you’re being told – about features, functionality and security – is accurate.

This can be problematic for a host of reasons—not least of which being the threat of potential security flaws. A company might boast about its commitment to user security and privacy through savvy public relations and product marketing efforts. But, by no means does this provide the user with anything more than lip service if the code doesn’t measure up; even if it’s rock-solid, the question of “How truly secure is this?” can never be answered with confidence.

Open source software offers greater transparency to the teams that use it; visibility into both the code itself and how it is maintained. Giving organizations access to the source code allows them the opportunity to evaluate the security of the code for themselves. Additionally, users have more visibility into how and what changes are made to the code base, including the pre-release review process, how often dependencies are updated and how developers and organizations respond to security vulnerabilities. As a result, open source software users have a more complete picture of the overall security of the software they’re using.

Open Source Encourages Community Input

Another major benefit is found in the communities which drive the growth and development of open source software. The vast majority of open source software is backed by communities of forward-thinking developers, many of whom use the same software they build and maintain as a primary means of communicating with team members. Open source developers and the communities around the software value users’ input to a significant degree, and many user suggestions end up getting incorporated into new versions. The community involvement and investment ultimately leads to many more responsible vulnerability disclosures than would likely occur if the software was closed source.

When vulnerability reports do occur, they tend to be more detailed because the code is open source; often, they even come with code suggestions for how to remedy a given issue. This is incredibly helpful and can dramatically speed up patching the vulnerability, as it allows developers to move faster without spinning wheels and getting stuck in the diagnostic process.

Faster Security Updates

Within the framework of closed source software, users are at the mercy of the companies behind those products or services when it comes to receiving software updates―open source tends to be a different story. Fully open source projects even allow users and developers to take things into their own hands and contribute feedback and fixes, should they choose to do so.

More often than not, updates and fixes for high-profile closed source applications involve a great deal of complex planning. If it’s not in the budget, for example, users may not see a new update for weeks, months, or even longer—security flaws or not.

In most circumstances, open source tends to move faster than closed source when it comes to iterating and releasing new versions. This is true for a handful of reasons, including the fact that open source software tends to have more eyes on the source code at any given time, as well as a community-driven interest in bettering the product whenever possible. If a significant security flaw is uncovered, users may even choose to fix the code themselves, resulting in a level of control that can never be achieved in a closed source environment.

Open Source Offers Flexibility, Collaboration and Enhanced Security

An open source methodology, in and of itself, is no guarantee of security, but it does offer teams greater insight and control over the software they rely on. And as open core and source-available software are more broadly adopted, they offer a wealth of opportunities to build incredibly secure environments without sacrificing usability for end users or support from official maintainers.

April 14, 2021
Gitpod Revamps Dev Automation Tool

Gitpod announced today it refreshed the dashboard for its namesake open source tool for automating the deployment of development environments, in addition to making Microsoft VS Code its default tool for editing code.

In addition, Gitpod has included support for sudo privileges and Docker images using an advanced user namespace to map and limit access to workloads using a built-in Supervisor capability.

Fresh off raising an additional $13 million in funding, Gitpod CEO Sven Efftinge said now that it’s clear developers will be working from anywhere for the foreseeable future, the need for a tool to automate setting up a developer environment on a local machine outside of an office has become more apparent.

Gitpod claims there are now more than 350,000 developers using its tool that creates a Kubernetes pod on each developer machine to deploy a complete application development environment, including the integrated development environment (IDE), plugins, compilers, build tools, code generators, databases and application servers. Gitpod makes use of the version control systems provided within, for example, a continuous integration/continuous delivery (CI/CD) platform as the single source of truth for configuring an application development environment.

The overall goal is to eliminate the time developers waste setting up application development environments so they can write code on their own machines. That approach also makes it simpler for developers to collaborate around the same application development environment regardless of where they happen to be physically located.

Efftinge said Gitpod provides DevOps teams with a tool to automate setting up development environments across any Git repository that is the equivalent of the Codespaces tool that GitHub recently made available. Rather than having that capability for only one repository, Efftinge noted that most developers work on projects that span multiple types of repositories.

Automating that process also makes it simpler to align development environments with production environments. One of the reasons why code doesn’t work in a production environment when it is first deployed is because the local development environment has been manually configured in a way that doesn’t accurately reflect the production environment. That issue can become even more problematic when developers are working on multiple projects involving, for example, separate microservices constructed using different repositories.

In the wake of the COVID-19 pandemic and the acceleration of multiple digital business transformation initiatives there’s naturally a lot more focus on automation and developer productivity. Most developers will be working outside the office for the foreseeable future. Of course, most developers have been writing code outside the office for years. The challenge now is enabling teams of developers to develop applications as efficiently as possible, no matter where they are in the world. In fact, now that development teams have become more distributed, organizations are becoming more comfortable hiring developers wherever they find them.

Eliminating inefficient manual tasks, such as maintaining development environments, is, of course, going to be a prerequisite for making it possible for those developers to, over time, navigate multiple application development projects without ever having met their colleagues in person.

April 8, 2021
AlmaLinux Foundation Delivers on CentOS Promise

A nonprofit organization launched this week to oversee future development of AlmaLinux, a distribution of Linux based on code originally contributed to the open source CentOS project.

At the same time, CloudLinux, which created AlmaLinux in response to a decision by Red Hat to pull financial support for CentOS, announced that a stable version of the operating system is now available. CentOS itself was based on the Red Hat Enterprise Linux (RHEL) distribution that is made available via a subscription.

Jack Aboutboul, community manager of AlmaLinux, said AlmaLinux is the result of the $1 million that CloudLinux committed toward creating a distribution of Linux based on a CentOS project used by hosting providers, such as CloudLinux, to run RHEL-compatible applications at a lower cost. AlmaLinux OS is a binary compatible fork of RHEL 8, which the foundation pledged to update to maintain that compatibility.

AlmaLinux is one of two distributions based on CentOS that have been created. The other, RockyLinux was created by Gregory Kurtzer, one of the original creators of a CentOS project. Red Hat, in the meantime, has moved to make RHEL subscriptions available for free to individual developers.

Aboutboul said AlmaLinux is more mature than RockyLinux, because it is based on a Linux distribution that has been battle-tested by IT service providers. AlmaLinux also added a SecureBoot capability as well as support for VMware and the Amazon Web Services (AWS) public cloud, noted Aboutboul.

The AlmaLinux Foundation also decided to publicly share errata about the ongoing development of the operating system, to make it easier for IT teams to troubleshoot issues that might arise in a production environment.

It’s not clear yet how much support either AlmaLinux or RockyLinux have. However, at this juncture, forks are becoming fairly common across the open source community. Those disruptions, however, don’t seem to be dampening enthusiasm for open source software. A global survey of 1,250 IT leaders conducted by Red Hat finds nearly 80% of respondents expect to increase their use of emerging open source software technologies over the next two years, with the top three use cases being infrastructure modernization (64%), application development (54%) and digital transformation (53%).

As Winston Churchill famously said, democracies are messy, and the open source community is no exception. Passions run high. At the same time, the various business models that spring up around open source projects often conflict; because of animosity, greed or differences of opinion, it is not uncommon for multiple forks of an open source project to emerge and evolve over time.

Despite that issue, IT organizations remain committed to open source for two primary reasons. First, no one needs permission to download open source software to a laptop. That enables developers to more easily experiment with new technologies. Second, cost. Many IT organizations prefer to use free open source software, rather than paying to license commercial software. They may one day pay for support, but the number of open source software users still dwarfs the number willing to pay for support; especially since knowledge and expertise is freely and equally shared across a growing open source community.

Given those benefits, any friction in the open source community itself is largely seen as an acceptable risk and just the potential cost of doing business.

April 2, 2021
Red Hat Survey Reveals More Confidence in Open Source

A recent Red Hat survey revealed how DevOps teams increasingly depend on open source to support their shift to cloud-native environments and digital transformations, and to maintain security in these highly distributed environments.

Based on a survey of 1,250 IT professionals, Red Hat maintains that the survey respondents — from organizations that were Red Hat customers — were unaware that Red Hat sponsored the study to help avoid “biased or influenced responses,” Red Hat said in a statement.

During the past two years, the number of organizations that deemed open source necessary for their digital transformations increased 11% to 53%, representing the majority of the respondents’ organizations.

The vast majority (87%) of the respondents considered open source to be “more secure” than or “as secure” as proprietary software, while 84% indicated that enterprise open source “is a key part of my organization’s security strategy.”

While the survey reflects open source’s importance for security for IT professionals in DevOps, its support of the overall business mission is key, as well, says Momodou Jaiteh, application security consultant at nVisium, an application security provider.

“Information security is about reducing and mitigating risk to enterprises, from technology, people and process perspectives. It helps enable the business to operate securely in the best interest of its shareholders and customers,” Jaiteh said. “Thus, security typically plays a supporting, [but] nonetheless critical role in the success of the business, especially in a digitally connected world.”

Open source software often helps businesses reduce overhead cost in delivering services to their customers through reduced time to market, Jaiteh said. One major additional advantage is “visibility and input from a lot of different people,” Jaiteh added.

“Greater visibility often translates to more observing eyes on all aspects of the open source software, which often translates to better security. This is even better for mission-critical software that deals with sensitive information, such as cryptographic algorithms, etc.,” Jaiteh said. “It is not a surprise to see open source used more in infrastructure modernization, because as we move to DevSecOps models of software delivery, infrastructure often is the bottleneck. Therefore, solving for infrastructure helps developers focus on aspects that are unique to their business.”

While survey respondents ranked improving collaboration between IT and business stakeholders to achieve a digital transformation as their highest priority, implementing cloud technologies or infrastructure ranked second. Infrastructure modernization is thus seen as a prerequisite for a full digital transformation, said Dirk Schrader, global vice president, security research at New Net Technologies (NNT), a cybersecurity and compliance software provider.

“Open source plays a vital, pivotal role in almost all digital trends driving any industry sector today,” Schrader said. “Cloud computing or containerization is rooted in open source technology, many development libraries are open source, and the market for open source solutions has been a model for servitization; the changes in one’s business model away from selling a product to selling a service, or an integration of service, and product that focus on additional value-add for the customer.”

For security, the main advantages consist of being able to review, inspect and audit “each and every piece of code,” Schrader said.

“The ability of open source to be a base for services makes it the natural companion for any digital initiative,” Schrader said.

April 1, 2021
Eclipse Foundation Creates Adoptium Working Group for OpenJDK Binaries

The Eclipse Foundation announced today a working group that will establish a governance framework for an Eclipse Adoptium project that enables developers to download compatible runtimes based on OpenJDK source code.

Members of the Adoptium working group include Alibaba Cloud, Huawei, IBM, iJUG, Karakun AG, Microsoft, New Relic and Red Hat.

OpenJDK is a free and open-source implementation of the Java Platform, Standard Edition that is provided by Oracle. AdoptOpenJDK was created in 2017 to provide open source, high-quality Java runtimes based on OpenJDK. Since then, AdoptOpenJDK has been downloaded more than 240 million times.

The Eclipse Foundation is now continuing that effort under an Eclipse Adoptium project that aims to become the primary source for OpenJDK-based binaries. The Eclipse Adoptium working group has negotiated with Oracle to create an OpenJDK Community TCK License Agreement (OCTLA) that will enable it to run compatibility tests to ensure conformance to the Java specification.

Mike Milinkovich, executive director of the Eclipse Foundation, said the Adoptium WG will provide the infrastructure, marketing, community building and developer education required to ensure timely releases and adoption of Eclipse Adoptium runtimes.

OpenJDK is a reference implementation used by multiple vendors to create their own binaries, which are only guaranteed to be compatible with the standard until the next update arrives; updates are released approximately every six months. The Eclipse Adoptium project will make it simpler for developer teams to identify which binaries are compatible with specific implementations of OpenJDK. Ultimately, the goal is to encourage developers to build applications using frameworks that make use of new OpenJDK features as they are added to the platform versus continuing to rely on older binaries.

At the same time, however, the Eclipse Foundation is also looking at making it simpler for enterprise IT teams to rely on long term support (LTS) licensing agreements that would be based on Eclipse Adoptium runtimes, Milinkovich noted.

Despite the plethora of programming languages developers use and even after 25 years, Java remains the dominant language for developers working in enterprise IT organizations. Over the last few years, OpenJDK has gained traction as organizations adopted an open source implementation of the specification that enabled providers of frameworks for building applications to reduce licensing costs.

It’s not clear at what rate enterprise IT organizations are replacing proprietary editions of Java with OpenJDK. However, in the wake of the economic downturn brought on by the COVID-19 pandemic, reliance on open source software has increased to reduce costs. At the same time, many organizations are looking to build microservices-based applications using Java frameworks that are based on OpenJDK runtimes.

Regardless of the underlying platform employed, as developers are asked to build and deploy applications faster, most of them will rely on the programming language they already know rather than take the time to master another.

March 23, 2021
Open Source Vs. Proprietary DataOps
Core DataOps concepts are making their way into data engineering teams and, from there, into the broader enterprise. Data engineers are retooling how they create data products, and much of this work revolves around creating data pipelines.

DataOps pipelines offer the kind of observability that traditional data integration and ETL processes don’t or can’t. They allow you to continuously integrate and test new data sources and deliver data in streaming or batch contexts with higher levels of quality and reliability than traditional, siloed approaches. They can also support machine learning efforts by preparing data for training, testing and deployment.

DataOps pipelines shorten the cycle time between a data consumer’s request and how quickly and reliably the data producer can fulfill it, all while observing service quality throughout its life cycle. The challenge teams face when building pipelines is how to construct them. The two approaches, as I see it, are do-it-yourself using open source, or through commercial product acquisition.

Open source certainly has its vocal adherents and some visible successes, but there are trade-offs to taking the open source route.

Building Or Buying Your Pipeline

One advantage of open source is in its flexibility and availability. Open source licenses, excluding the SSPL, gives users incredible freedom over what they can do with the software. If you have the skill, you can compose a DataOps pipeline that can take any data, enrich it and route it to the right place. That flexibility, though, is also a downside. While you can do anything you want, you also have to do it. Open source projects like Kafka, Pulsar, Spark, Airflow and Flink don’t know anything about the data they’re handling. That’s up to the developer.

This may not sound like a problem, but today’s data engineers are handling dozens of data types in hundreds – or even thousands – of different formats. If you add in operational data, you’re also looking at data flooding in from firewalls, containers, SNMP traps and HTTP sources. And that’s just what’s coming at you. You also need to fetch data from object stores, multiple activity hubs and other messaging sources. No open source project natively supports the variety and volumes of data required in a modern DataOps pipeline. Every new data source you add means starting from primordial components rather than higher-level abstractions, adding time and undifferentiated effort to your work.

You’ll also need essential features like per-source backpressure, support for a range of protocols and data stores, role and permission management and so on. That’s a lot for overworked data engineering teams to build and maintain over the life of a pipeline.

There’s no ‘easy button’ for proprietary products, either. Many commercial products only support a handful of sources and destinations and may have excessive licensing costs related to data ingest rates or per-connector licensing fees. Also, they can be challenging to scale, from an operations perspective. Seeing a cool drag-and-drop interface may look great during the demo, but it quickly becomes a cluttered mess when you’re looking at thousands of similar data processing pipelines.

The advantages to commercial products center around the features enterprises need, like governance, management and security. They may also integrate with other components more readily than open source. Finally, having someone to call when you need support is also essential when running in production, although open source core companies also offer support.

Take a Higher-Level View

As you’re evaluating DataOps pipeline options, keep two things in mind. First, regardless of which path you choose, ensure your chosen technology offers observability and monitoring features. DataOps is about reacting to change faster, and you can’t react if you don’t know what’s going on. Building observable DataOps pipelines ensures you are delivering a valuable data product to the range of data consumers in your enterprise.

Next, get past the technology. DataOps is more a people discipline than a technology practice. A key component of a successful DataOps implementation is aligning metrics and incentives across teams. If the staff running the DataOps pipeline are disconnected from the business outcomes their data products inform and influence, they’re not incentivized to run the pipeline as a mission-critical infrastructure component. It’s key to align all of your teams with shared metrics and incentives.

DataOps Determination

Before you decide on your DataOps pipeline implementation, there are some things you should do.
- Determine who your customers are and what their expectations are around data delivery, enrichment, user experience and observability requirements. Are you only serving data scientists, or are you also serving infrastructure and operations, business intelligence and marketing? Knowing your customer allows you to make better decisions.
- Test the software in a real-world environment. Everyone dislikes being told to perform a proof of concept before you buy/build, because POCs are time consuming. Remember – they’re less time-consuming than picking the wrong tool and looking foolish later.
Engineering teams sometimes choose technologies that are in fashion rather than selecting the right tool for the job. Taking a pragmatic view of performance, manageability and engineering effort, as well as involving a wide array of stakeholders, will give you a higher chance of success with DataOps.
March 19, 2021