Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.
This week: Network neutrality is back on the FCC’s agenda, and unused London office gets Zucked. (more…)
Category: DevOps and Open Technologies
Editorial Calendar for February 2016
-
Checkmarx Surfaces Threat to GitHub Repositories
Checkmarx, a provider of application security testing tools, today reported that malicious actors have been able to compromise the free automated dependency management tool for software projects, known as Dependabot, that GitHub provides.
It’s not clear how prevalent the compromise is, but the attack involves the tokens that GitHub provides to allow access to Dependabot. Once cybercriminals gain access to a repository, they then add additional emails to the account that, once authorized, are used to inject malicious code into a repository that exfiltrates defined secrets to an external server. Cybercriminals can then modify any existing JavaScript files using a password-stealer to steal the credentials of any end user submitting their password in a web form.
Guy Nachshon, a security researcher for Checkmarx, said Checkmarx was only able to discover the threat because one of the cyberattackers made a mistake. DevOps teams are advised to confirm the identity of each user that has access to both public and private GitHub repositories to ensure their software supply chain has not been compromised, he added.
In general, attacks against software supply chains are becoming more sophisticated, noted Nachshon. Beyond launching phishing attacks to steal the credentials of individual members of an application development team, it appears cybercriminals are employing tactics and techniques that enable them to steal the credentials of an entire team contributing to a repository.
These attacks, of course, have become a matter of national security. Governments around the world are crafting legislation that will hold organizations more accountable for the security of the applications they build and deploy. It may be a while before those proposals become law, but it’s clear the pressure on organizations to secure their software supply chains is building. More organizations than ever are, as a result, embracing DevSecOps best practices to better protect their software supply chains.
The challenge is that many of the components used within those supply chains are from open source projects that organizations don’t have the ability to patch should a vulnerability be discovered. More troubling still, even when a patch is available, many organizations are reluctant to upgrade for fear of breaking an application. In effect, they are betting cybercriminals are not going to discover a potential exploit that exists in their application portfolio or that the damage inflicted will be less than what might otherwise be experienced should an application break following the update of a critical component.
Playing the equivalent of Russian Roulette with application security is, to put it mildly, inadvisable. DevSecOps teams need to not only address vulnerabilities in applications as they are being built but also address the massive amount of technical debt relating to known vulnerabilities that have been allowed to accrue for more years than most anyone cares to admit.
Unfortunately, cybercriminals are now clearly aware of the soft underbelly that exists within application environments, so the race to address an issue that has been allowed to fester is now on. And the odds, alas, are already stacked in favor of the malicious actors that are ready and willing to wreak havoc.
-
Why Data Silos Are Killing Your Developer Productivity
Every company is at risk of fostering data silos. In fact, a study conducted by Forrester Consulting for Airtable across employees in the U.S. and the UK revealed that, on average, knowledge workers spend 30% of their week looking for the information they require to do their job and the difficulty in finding and accessing data is the top reason why they feel disengaged from work. What causes essential data to be inaccessible in the workplace? We will explore the root of modern data silos, why they drain developers’ productivity and how to break away from these silos with better observability in your organization.
What are Data Silos?
A data silo is a collection of data held by a group that is not easily or fully accessible by other groups in the same organization.
Data silos can be a byproduct of organizational structure. When different departments or business units in an organization operate independently, they develop their own way of collecting, storing and managing their data. The processes to access that data from other business units can be cumbersome or even nonexistent. Sometimes, the individuals responsible for the data will even be against sharing it for fear that it would be misused or misunderstood outside of their particular unit. The difficulty in accessing information then leads to independent data collection from other business units and ultimately creates other data silos and multiple sources of truth.
Data silos can also happen during times of growth for an organization. New tools are necessary to support the growing needs of the business and are added to the list of tools already in use. Over time, this list grows, and systems that were designed to operate in isolation can create data silos by limiting the ability to integrate with other systems. A recent survey of marketing organizations by Emarsys showed that the surveyed organizations are using an average of 91 marketing cloud services. These 91 tools undoubtedly create data silos unless they are fully integrated together or into a central system.
Why are Data Silos Killing Your Developer Productivity?
Data silos aren’t only slowing your developers down by limiting their access to the data needed for development. They also put a lot of extra work on their plates. And it goes without saying that this additional time spent on data management reduces the time available for improving your product.
Management of Redundant Data
Multiple teams in the organization storing their data separately means increased workloads for developers, who need to update and maintain separate copies of the same data.Inconsistent Data Quality
Data gaps or quality issues happen more frequently since different parts of the organization have their own standards for collecting, cleaning and validating data. This can lead to development delays or errors in the product.Time-Consuming Integrations
Developers have to code a lot of integrations to connect all the different tools your organization uses and keep the data flowing between them.Inefficient Collaboration
When different teams work in silos, the chances of misunderstanding are greater, leading to development delays and misaligned priorities.Difficulty in Data Governance
Maintaining high standards of data privacy, security and compliance is time-consuming when the data is spread out in different silos and can create costly regulatory risks.Steps Leaders Can Take to Break Down Data Silos
Fortunately, there are steps that you can take to break down the data silos in your organization, relieving your developers at the same time. Each step may have its own set of advantages and challenges, so the solutions you choose ultimately depend on your organization’s needs and objectives.
Create a Unified Data Strategy
Set clear goals and standards for data collection, storage, management and analysis at the company level. You should also implement data governance and security policies.Adjust the Culture
Foster a culture of collaboration and data sharing across the company by encouraging teams to share data, insights, and best practices across different departments. You can achieve this through regular communication, training and incentives for cross-functional collaboration.Establish Cross-Functional Teams
Bringing together members of different departments to collaborate on data-related projects can prove invaluable. These teams should have a shared understanding of the organization’s data strategy and should be empowered to make decisions about data collection, analysis and sharing.Invest in Data Integration and Interoperability
Invest in tools and technologies that promote data integration and interoperability across different systems and departments. Adopt standard data formats, implement APIs and data connectors or use data integration platforms to make observability collaborative. Make sure that the tools you choose to add to your company have various native integration capabilities to avoid custom integrations as much as possible.Congrats, you now know all about data silos and how to break them down! The next step for you is to identify the data silos in your company and put a plan in place to get rid of them. Your employees and coworkers will thank you!
-
Harness Launches Open Source Gitness Platform
At its unscripted conference today, Harness unveiled a streamlined open source git platform, dubbed Gitness, that is significantly lighter weight than the platforms DevOps teams traditionally rely on to build increasingly complex applications.
At the same time, the company is adding four additional modules to its continuous integration/continuous delivery (CI/CD) platform to improve developer productivity and make it simple to build and deploy secure applications.
Finally, Harness is launching an Engineering Excellence Collective, a consortium of 300 senior engineering leaders. The collective is developing a maturity model based on 11 pillars for achieving software engineering excellence; a FinOps Excellence community initiative was also announced to advance best practices for controlling cloud computing costs.
Harness CEO Jyoti Bansal said the need for a modern open source Git platform—as an alternative to legacy platforms developed a decade ago—has never been more apparent. Gitness provides all the collaborations, performance, analytics and security capabilities DevOps teams require in a streamlined platform that reduces overall bloat by seamlessly integrating, for example, code hosting and automated pipelines, he added.
Other core capabilities include automated status checks and mandatory reviews, branch protection rules, user access management tools and a semantic search engine that has been infused with generative artificial intelligence (AI) capabilities that invoke multiple large language models (LLMs).
Collectively, the approach reduces the amount of time spent when attempting to integrate and manage disparate tools on legacy platforms that inevitably lead to the creation of additional bottlenecks, said Bansal. Rather than bolting on additional tools, the time has come to simplify software engineering by migrating to a platform specifically designed to address modern software engineering requirements, he added.
Designed to be installed using Docker contains in a matter of minutes, DevOps teams can also take advantage of tools to automatically migrate their existing repositories and pipelines from platforms such as GitHub and GitLab.
Gitness is also at the core of one of the four additional modules that Harness is adding to its portfolio. The Harness Repository is a premium tool, available in beta, that provides governance and compliance-as-code capabilities embedded within the repository.
The three other modules being added are a Harness Internal Developer Portal based on open source Backstage software, Harness Infrastructure as Code Management that makes it simpler to implement GitOps workflows at scale and Harness Software Supply Chain Assurance to provide visibility into open source software components and manage the creation of software bills of materials (SBOMs).
It’s not clear whether DevOps teams are ready to rip and replace existing platforms, but the pressure to improve developer productivity has never been higher. Organizations increasingly realize that developers are spending the bulk of their time managing development environments tied to legacy platforms that create bottlenecks and are becoming more costly to manage and sustain.
Harness, in effect, is now calling for the establishment of an alternative approach based on open source software that promises to streamline software engineering workflows. As applications incorporate microservices, security controls and AI models, they are only going to become more challenging to build and deploy at scale.
-
Innersourcing Open Source Principles in the Enterprise
Software development has experienced a seismic shift in recent years, with open source becoming the heartbeat of innovation. A full 90% of enterprise IT leaders say their organization is using open source, according to a Red Hat report. Leading this transformation is GitHub, the ubiquitous platform that not only jumpstarted the open source movement but has also been quietly transforming how enterprises approach development at large.
But it’s not just tech companies trumpeting the open source creed—academics are on board, too. A paper recently published by the University of Georgia found a hard-to-dismiss correlation between open source innovation and overall business success. “We find a significantly positive association between GitHub activity and future realized net income, suggesting that open innovation is a strong predictor of future earnings,” according to the report.
I recently met with Chris Reddington, senior manager of developer advocacy at GitHub, to dive into how and why open source is becoming essential for enterprise innovation. According to Reddington, the once-dominant proprietary information mindset is shifting, with open source cultural principles now permeating most companies, a practice also known as innersource. Put simply, open source is now cool in the enterprise.
Bridging the Open Source-Enterprise Divide
GitHub’s close ties to the open source space have allowed them to illuminate the parallels between the open source and enterprise worlds. Reddington noted that, nowadays, open source and enterprise aren’t really separate communities but rather intertwined facets of the same software ecosystem. This movement is represented by the open source program office (OSPO) becoming more commonplace in large organizations.
For an example of innersource in practice, take Porsche, who recently announced the Porsche FOSS Movement to guide internal common values and development principles around open source. We’ve also seen similar large-scale commitments to open source values at Walmart, Spotify, Facebook and others.
Open source clearly isn’t just for startups—large companies are increasingly seeing an advantage in both using open source software and sharing their own open source components. The growing investment and maturation of OSS initiatives are positive strides, as it’s important to the stability of the free open source software (FOSS) ecosystem as a whole that those using software contribute back to its maintenance, especially to core packages the world depends upon.
The Power of Innersource
The study from the University of Georgia brings to light how enterprises increasingly recognize the need to work in sync with their developers and customers to remain competitive. This is a testament to the evolving nature of development—an industry that’s embracing a “team sport” mentality instead of a siloed approach, explained Reddington. And as enterprises align with open source principles, they are discovering the power of innersource, he said. “Innersource is a development methodology where engineers build proprietary software using best practices from large-scale open source projects, like Kubernetes or Microsoft’s Visual Studio Code,” GitHub explained.
Reddington elaborated on two key elements of this approach:
- Breaking down silos: The traditional organization-centric structure of projects is being replaced by a more transparent and collaborative approach. Innersource encourages transparency and asynchronous communication, tearing down barriers that previously inhibited efficient collaboration.
- Engaging responsibly: Reddington emphasized the significance of enterprises giving back to the open source community. Whether through sponsorships, allocating time for engineers to contribute, or open sourcing their projects, companies can foster symbiotic relationships where they benefit from and contribute to the open source realm.
Essentially, by adopting innersource principles, enterprises foster an environment that promotes collaboration and knowledge sharing, thus enabling innovation to flourish across teams and projects.
Balancing Open Innovation and Proprietary Software
One of the most intriguing aspects of this transformation is finding the equilibrium between open innovation and proprietary software. Reddington acknowledged the necessity for businesses to generate profits, which inherently stems from delivering value.
While reinventing the wheel may be tempting, Reddington highlighted the importance of recognizing existing solutions and incorporating them into ongoing projects. This pragmatic approach maximizes efficiency and enables teams to focus on the unique, value-generating aspects of their work.
Organizations should also consider what software of their own could be open sourced. From a business perspective, this most likely won’t be the “crown jewels,” said Reddington, but ancillary components that others could benefit from. For example, had it not been for Google open sourcing Kubernetes, Lyft open sourcing Envoy Proxy or Facebook sharing GraphQL, we would not have these standard components for all to use.
Open Source Movements: Past, Present, and Future
Over the last decade, we’ve seen significant, more institutional maturation around open source projects. Reddington pointed to the Cloud Native Computing Foundation (CNCF) as a standout. He specifically noted Kubernetes-based Event Driven Autoscaler (KEDA) as a cool project to help scale Kubernetes containers based on the number of events needing to be processed.
He also underscored the surge of interest in generative AI, evidenced by projects like GitHub Copilot. It will be curious to see how AI assistants incorporate open source at their core, such as open foundational models and community-supported datasets.
Potential Setbacks To The Open Source and Innersource Movement
In the dynamic realm of enterprise-scale software development, nurturing innovation can be a complex endeavor. Innersource principles are one way to change the attitude toward development, making it more efficient, collaborative and aligned to the current and future developer experience expectations.
Yet, this isn’t to say we haven’t had some setbacks recently that could threaten the progress the OSS community has built. For example, HashiCorp recently abandoned its open source license around Terraform in favor of a business source license (BSL). There is also an increasing number of supply chain-related attacks that threaten common open source libraries.
Gazing Into the Future
As Reddington aptly summarized, we’re still in the early days of this transformative journey. The developer community’s inclination for experimentation will undoubtedly drive the solutions to the most pressing challenges in open source. And with enterprise adoption on the rise, the future promises more cohesive collaboration, broader adoption of innersource principles and continued innovation driven by open source values.
Interestingly, we’re seeing more synergy between open source and enterprise domains, which used to feel like separate realms entirely. And it will be interesting to see how businesses continue to balance transparency and proprietary value into the future. No matter what, the lines between them are blurring, enabling companies to harness the power of innersource for enhanced collaboration and innovation.
-
Oracle Previews Latest Java 21 Innovations
Oracle today made generally available a Java 21 edition and provided early access to more than 15 proposed enhancements to the Java development kit (JDK).
Those enhancements were announced at an Oracle CloudWorld 2023 conference, including support for virtual threads to improve overall throughput and adding support for a vector application programming interface (API) that makes it simpler to build Java applications that involve artificial intelligence (AI) models.
Georges Saab, senior vice president of Oracle Java Platform and chair of the OpenJDK governing board, said the goal is to streamline access to the latest innovations that will be added to future versions of Java.
Specifically, via Project Amber, JDK 21 delivers improvements such as support for string Templates, record Patterns, pattern Matching for switch, unnamed Patterns and variables and unnamed classes and instance main methods.
In addition to the Vector API, Project Panama adds a Foreign Function & Memory API. Project Loom, in addition to virtual threads, adds scoped values and structured concurrency support.
There is also a more efficient garbage collection mechanism being proposed.
Oracle is committing to offering long-term support for Java 21 for at least eight years to encourage organizations to start working with this version of Java today. Oracle is also making available a Java Playground, an online sandbox on the Oracle Cloud Infrastructure (OCI) service where developers can create Java code using this edition without needing a local runtime or integrated development environment (IDE).
In general, Java has been experiencing a renaissance after more organizations began contributing to projects once the core platform became available under an open source OpenJDK license. In fact, the Java community is now adopting many concepts pioneered in other programming languages. That enables organizations to, for example, drive digital business transformation initiatives that depend on the modernization of legacy applications. As part of those efforts, the rate at which Java applications are being deployed in the cloud has substantially increased.
Despite the number of alternative programming languages, many enterprise IT organizations remain committed to Java. Rather than opting to retrain developers, it’s simply been more cost-effective to enable developers to continue to build applications using a programming language they already know.
Many organizations are currently in the process of shifting to Java 17. It’s not clear how many organizations will skip Java 17 and jump to Java 21 from previous versions of Java, but it’s clear Oracle is committed to making the lag time between moving from previous versions of Java to Java 21 much shorter.
Most organizations will find themselves continuing to deploy Java applications everywhere from the network edge to the cloud. These applications will be written in a programming language that has persevered for more than two decades and continues to evolve and mature for the next era of application development.
-
Summit Highlights Open Source Software Security Progress
More than a year after the Open Source Security Foundation (OpenSSF) summit kicked off an initiative to better secure open source software-based software supply chains, a lot of progress has been made—but much work remains to be done.
The OpenSSF this week hosted a Secure Open Source Software (SOSS) Summit 2023 event in Washington, D.C., during which it made available a Secure Open Source Software Vision Brief 2023. The brief describes the various efforts being made to improve open source software security, including, for example, providing maintainers of open source software projects with free DevSecOps tools.
This week, the Cybersecurity and Infrastructure Security Agency (CISA) also published an Open Source Software Security Roadmap that defines its role in providing more visibility into how open source is being used and the associated risks.
However, it’s not clear how much maintainers of these projects are embracing DevSecOps workflows. Many projects are led by a handful of maintainers that often lack the time and skills required to ensure the components they employ have not, for example, been compromised by malware that might find its way into downstream applications.
Sonatype CTO Brian Fox said the focus of the summit’s participants will be on education and securing public code repositories. While there is a lot of focus on maintainers, he added that there is not enough attention to all the organizations that have not updated open source components within a software build with known vulnerabilities. Many IT teams are reluctant to upgrade components for fear of breaking applications with complex dependencies, but the amount of code that is potentially insecure is massive. “There’s a huge mountain to climb,” he said.
The challenge is determining which code to upgrade first in a way that will have the most impact in terms of improving application security, added Fox.
Shawn Ahmed, chief product officer for CloudBees, also noted that organizations should evaluate the DevSecOps processes being applied by maintainers of open source software projects. CloudBees, for example, reviews the plugins provided by the open source community for the Jenkins continuous integration/continuous delivery (CI/CD) platform for vulnerabilities every month, he noted.
It’s also critical for organizations that consume open source software to contribute code to improve the security of open source software, added Ahmed. Too many organizations are focused on open source software security without devoting resources that would help resolve the issue. he added. “We need to make sure open source software is not a vulnerability threat,” said Ahmed.
Moran Ashkenazi, chief strategy officer (CSO) and vice president of security engineering for JFrog, said on the plus side, there is a lot more awareness of open source software security issues since the disclosure of a zero-day vulnerability in the Log4j shell tool widely used to manage logs in Java applications. But added that this issue is not going to be resolved overnight. “That was a wakeup call,” she said.
Tapabrata Pal, vice president of architecture for Fidelity Investments, noted that the fundamental problem is that cybercriminals have become more adept at injecting malware into software supply chains, so no one knows whether their application environments might be compromised. “It’s a serious issue,” he said.
It’s not known whether the open source software that many developers routinely incorporate into their applications might contain vulnerabilities that could be exploited, but the level of technical debt that has accrued over the last several decades is considerable. The issue is finding a way to address it without becoming overwhelmed by the enormity of the task.
-
Google De-Recruits 100s of Recruiters ¦ ARM Valued at $45½B in IPO
Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.
This week: Google fires hundreds of recruiters, and ARM gets a sky-high valuation. (more…)
-
CloudBees Unfurls Dual CI/CD Strategy at DevOps World Event
During a DevOps World event, CloudBees took the wraps off the first major update to the open source Jenkins continuous integration/continuous delivery (CI/CD) platform made in the past several years.
At the same time, the company also unveiled a proprietary DevSecOps platform based on Kubernetes that is optimized for building and deploying cloud-native applications based on portable Tekton pipelines.
Shawn Ahmed, chief product officer for CloudBees, said the updates to Jenkins make it possible to horizontally scale Jenkins controllers while at the same time ensuring high availability. The latest update also provides a workspace caching mechanism to simplify sharing them across multiple jobs in a way that significantly reduces the amount of compute and storage resources consumed, he added.
CloudBees also added a Baseline capability to the Jenkins repository to improve startup times without causing build storms.
Collectively, those capabilities will increase the number of jobs that can be simultaneously executed to improve overall productivity, said Ahmed.CloudBees has also added a Pipeline Explorer tool in the cloud-based edition of Jenkins that enables DevOps teams to visually debug and troubleshoot pipeline jobs. That tool will also provide the foundations for self-healing and artificial intelligence (AI) capabilities that CloudBees plans to add to the Jenkins platform later this year and into 2024, added Ahmed. For example, AI tools will surface suggestions on how to fix a problem in a pipeline, he noted.
The DevSecOps platform, meanwhile, leverages Tekton pipelines to provide DevOps teams with more granular control over how pipelines can be orchestrated, said Ahmed. It also provides access to GitHub Actions-style domain-specific language (DSL), support for feature flagging, security, compliance, pipeline orchestration capabilities, analytics and value stream management (VSM) tools that can be deployed by an internal IT team or accessed via a single or multi-tenant managed cloud service.
Those built-in capabilities will make it simpler for platform engineering teams to centrally manage building and deployment of cloud-native applications by enabling developers to self-service their own engineering requirements via an internal developer platform (IDP), said Ahmed. The overall goal is to provide a more integrated approach that breaks down software engineering silos that exist in many existing DevOps workflows, said Ahmed.
Each DevOps team will need to decide for themselves to what degree to move forward using Jenkins to build and deploy applications versus opting for a more modern alternative in the form of the DevSecOps platform. CloudBees, however, is committed to providing plug-ins and integrations across both platforms to enable DevOps teams to protect their investment in existing tooling, noted Ahmed.
The status of Jenkins X, an open source CI/CD platform based on Kubernetes that CloudBees previously donated to the Cloud Delivery Foundation (CDF), is not immediately clear. Regardless of the approach, however, there is a clear opportunity to revamp DevOps workflows by upgrading Jenkins or switching to a more modern platform such as DevSecOps. The challenge, as always, is determining the pace at which any such transition might be made.
-
Latest eBPF Advances Are Harbingers of Major Changes to IT
The open source community advancing the extended Berkeley Packet Filter (eBPF) gathered at a virtual eBPF Summit today that featured a demonstration of eBPF being used in a Windows environment. eBPF enables applications to run within a sandbox in the Linux microkernel.
Microsoft has signaled its intent to employ eBPF within Windows like the latest version of Linux already does today. That capability makes it possible to, for example, run networking, security, observability and storage software closer to the kernel to dramatically improve performance versus running that software in user space.
Liz Rice, chief open source officer for Isovalent, a provider of networking software based on eBPF, said that while it’s clear eBPF will have a major impact on IT, the transition to eBPF won’t be a single event. Instead, it will be more of an evolutionary process that will occur as organizations upgrade their operating systems and the various tools and applications they employ.
The eBPF Foundation, an arm of the Linux Foundation, was formed to expand the use of eBPF across all operating systems. Members of the eBPF Foundation include Facebook, Google, Isovalent, Microsoft and Netflix.
In effect, eBPF changes the way operating systems are designed. It bridges the boundary between kernel and user space by enabling developers to combine and apply logic across multiple subsystems that were historically completely independent. That approach enables, for example, a security tool to scale to the point where it can identify threats at much higher levels of throughput at a time when brute-force cyberattacks are overwhelming existing cybersecurity tools.
Similarly, networking, storage and observability tools should be able to run at much higher levels of scale. That’s especially critical for DevOps teams embracing observability platforms that need to dynamically process massive amounts of data in near-real-time.
Less clear is the degree to which eBPF might one day drive convergence across distinct categories once multiple tools and applications run at the microkernel level.
IT organizations would be well-advised to ask their vendors when they plan to support eBPF as they plan their upgrade cycles. It’s not likely IT teams are going to want to upgrade every tool and application that can benefit from eBPF all at once. In fact, updates to these applications that don’t support eBPF might even be postponed in the knowledge that a much larger and arguably more critical update is on the immediate horizon.
In the meantime, as the number of organizations running the latest versions of Linux continues to increase, more hands-on experience with eBPF will be gained. IT teams may not need to concern themselves with what is occurring in the microkernel of the operating systems, but they should at least understand how eBPF ultimately helps reduce the total cost of running IT and how it makes the Linux microkernel more programmable. Those IT teams, however, will soon discover how stark a difference in performance there will be.