devopsy.kinsta.cloud

Tag: open source

The Risks and Potential Impacts Associated with Open Source

Open source software (OSS) is built by communities of developers who contribute their knowledge and time to OSS projects they find appealing. That code can then be used by individuals, communities and organizations in their software products—the only obligation they have is to play under the rules of the license with which the OSS project was published.

This type of knowledge sharing brings many benefits to OSS users as it speeds up software development time and can help companies become more competitive in the market. Unfortunately, there is also a catch. Those benefits also come with certain risks which every OSS user needs to be aware of and take necessary actions to mitigate.

The OSS License

One specific risk to consider involves the OSS license. Not knowing what your obligations are under the license (or not abiding by those obligations) can cause an OSS user to, for example, lose intellectual property or experience a monetary loss.

Operational Impact on the Product

Another risk is the operational impact that the OSS in use will have on the product. It’s important to understand whether the community is consistently managing the OSS project and its issues in a timely manner, and that your organization has implemented the latest version and deployed patches have been applied. Using out-of-date, unpatched or unmaintained OSS could impact your release capabilities, product quality and customer trust.

Security Vulnerabilities

The third risk to consider when using OSS is that of security risk. As open source is software—and in many cases very complex software—it brings with it the possibility of introducing software security vulnerabilities into your source code. Some vulnerabilities can be introduced into your projects through dependencies that the OSS your organization is using has on other OSS projects. Additionally, vulnerabilities could be unintentionally introduced by developers as they code a functionality using open source. The reason for this is that software developers, whether professionals or hobbyists, oftentimes do not consider security while coding. This has a substantial impact on their work and their contributions to open source projects.

It’s not necessarily that developers intentionally skip the implementation of security procedures. Secure coding is a specific discipline in software development that guards against the accidental introduction of vulnerabilities—a discipline that must be learned.

Understand that many developers are self-taught or have been trained in schools and universities (where we are only recently seeing courses in software and application security emerging). Many use their software development skills as a hobby, to contribute to OSS projects or to pursue their own personal projects. Unfortunately, training programs, programming courses in schools and online tutorials often neglect to convey this knowledge or to stress the importance of building secure software.

It’s also important to note that many organizations find it difficult to introduce a secure coding discipline into their software development process. In other words, this struggle persists beyond open source projects. If you’re with an organization experiencing this pain point and unsure where to begin, you’ve come to the right place. There are several strategies that can be used to begin your secure coding journey.

Training

By giving developers the education required for them to become security aware, you’re setting them up for consistent, secure development success. There are a variety of in-person and computer-based training options on the market.

Software Composition Analysis (SCA) Tooling

Implementing an SCA tool will uncover incorrectly coded strings or vulnerabilities that have been introduced in code or OSS components. Not only can SCA tools assist developers in identifying potential software risks, some SCA solutions can help to train developers by offering remediation guidance or short training videos on how to avoid such vulnerabilities in the future.

It’s essential for software issues to be discovered early because the deeper into the process they’re identified, the more time-consuming and costly they are to remediate. As such, in recent years there has been a push to shift left, introducing security into the product development life cycle from the very beginning.

Companies and individuals need to also understand that this security mindset doesn’t start and end in the coding phase of the development process. A security mindset needs to be a part of every phase: from design to deployment to maintenance, all the way through to the product’s end of life.

Emphasizing a security mindset also has a long-term, broad reaching positive effects on the whole software development ecosystem, including the OSS world. As developers are trained to code securely (through university courses and other training institutions, coding portals or professional development through their employer) they pass this knowledge on to the projects with which they’re involved. This then leads to the production of higher quality proprietary and open source software, which in turn minimizes risk across the board.

— Boris Cipot

January 27, 2020
The Acceleration of Data and Data Storage Trends
There’s never been a more exciting time for data. Increasing demands for size, speed, performance and reliance have propelled the data market forward at a faster rate than could have ever been anticipated, and new technologies are making some dreams for data a reality.

From open source adoption in place of legacy databases to cloud-native app stacks and more, here are seven trends driving the acceleration of data and data storage.

Oracle’s Slow Fade Drives Cloud-Native Adoption

Organizations of all sizes continue to eschew Oracle for cloud-native databases, shrinking the user base of the monolithic legacy provider. Two major trends driving this activity are the ongoing move to microservices-based redesign of applications and the rise of cloud-native deployments running in public clouds or Kubernetes. Overall, Oracle has lost market share every year since 2013, and legacy relational database players have dropped about five percentage points per year. Major organizations, such as Amazon and Salesforce, have already figured out that it’s to their benefit to use less, not more, Oracle.

Open Source DB Usage Increases

Open source databases that claimed 0% of the market 10 years ago, now make up more than 7%. It’s clear that the market is shifting and an increase in open source database adoption is on the rise. This goes against the recent trend of database and data infrastructure companies abandoning open source licenses for some, or all, of their core projects. However, as technology rapidly advances it will be in the best interest of database providers to switch to a 100% open source model, since freemium models take a significantly longer period of time for the software to mature to the same level as a true open source offering.

Furthermore, Gartner predicts that, by 2022, more than 70% of new in-house applications will be developed on an OSDBMS or OSDBMS-based dbPaaS. It’s clear that open source has proven to be the most successful approach to developing and distributing business-critical infrastructure software, driven by the benefits of reduced total cost of ownership, access to community support and innovation and compatibility with commercial databases, making it easy to migrate towards an open source solution.

Kubernetes For Stateful DBs

There has been continued growth in the adoption of both microservices and containerization. With the combination of massive adoption and the support of a large and thriving open-source community, it is clear that the dynamic nature of containerization and microservices deployments will have huge implications on fundamental application architecture. To put it simply, this is the future of how applications will be built. This is also why the adoption of Kubernetes has continued to grow.

Multi-Cloud Goes Mainstream

Multi-cloud deployments are becoming the norm in today’s enterprise. A multi-cloud approach is critically important for organizations that run on-premise, since they need to stay in a hybrid mode when moving microservices to the cloud. As a result, enterprises are widely embracing distributed SQL databases to ensure agility without the availability constraints of traditional monolithic databases, such as Oracle.

Rise of Edge Data Centers

While the rise of edge data centers is assuredly a trend, Gartner predicts that three quarters of all databases will either be deployed or migrated to a cloud platform by 2022. There will also be an increase in multi-cloud adoption as a result of the geo-distribution of data. Edge data centers are already starting to offer data services with geo-distribution, a clear sign that demand is creating supply.

Cloud-Native App Stacks Gaining in Popularity

Modern microservices-oriented architectures are becoming increasingly popular. Over time, this leads to environments with lots of microservices that should be able to notify one another of changes and store these changes in a system of record. The application stack that is getting very popular for such use-cases is a combination of:
- Spring framework to build the applications.
- Kafka as the message bus to send notification between the microservices.
- A Distributed SQL database to store the data in a scalable manner and query it efficiently.
Other application stacks that are rising in popularity are the JAMstack ( Javascript, APIs and Markup) where there is no dependency on any particular web server. Javascript frameworks such as ReactJS and VueJS have become very popular for building the frontend portion of the app. GraphQL frameworks such as Apollo and Hasura are getting increasingly adopted as efficient ways to build the API portion of the app.

Advances in innovation for data and data storage will undoubtedly continue, driven by major trends across all areas of business, as long as organizations are prepared to embrace change and hit the accelerator on their data strategies.

— Karthik Ranganathan
January 21, 2020
IBM Open Sources SysFlow Monitoring Platform

IBM has announced that its SysFlow monitoring platform is now available as an open source project.

Fred Araujo, a research scientist in the Cognitive Cybersecurity Intelligence Group at IBM Research, said IBM developed lightweight SysFlow agent software and monitoring tools as a way to provide more context around the telemetry data being collected while simultaneously reducing the amount of data that needs to be stored.

SysFlow encodes a representation of system activities into a compact format that records how applications interact with their environment, Araujo said, noting that level of context provides deeper visibility in everything from container workloads to cybersecurity forensics. However, unlike existing monitoring platforms, SysFlow doesn’t require IT organizations to collect a massive amount of data to achieve that goal—it is intended to provide for a superset of the NetFlow framework used to analyze network traffic patterns to capture system events, he said.

Araujo noted IBM doesn’t envision SysFlow eliminating the need for legacy log analytics platforms, as they provide a way to analyze log data. However, SysFlow does enable IT organizations to apply analytics via a graph-like visualization to surface patterns that goes beyond a comparative simple rules-based approach, said Araujo. For example, SysFlow’s approach will make it easier to uncover the relationship between various events that make up a cybersecurity attack and subsequently to identify what countermeasures to employ to create the appropriate kill chain response. It also should substantially reduce the amount of fatigue cybersecurity teams experience from chasing down false-positive alerts, he said.

SysFlow is designed from the ground up to integrate with both open source frameworks such as Apache Spark and commercial analytics platforms via an open serialization format and associated libraries. IT organizations also can leverage a set of reusable components and APIs to make it easier to deploy telemetry probes. IT organizations also can take advantage of an extensible policy engine that can ingest customizable security policies described in a declarative input language, which then can be checked against records captured by SysFlow.

Araujo said IBM developed SysFlow to address the challenges associated with managing application workloads in public clouds, but the framework can be applied to any on-premises IT environment to create a single pane of glass through which multiple platforms can be monitored.

Regardless of the platform on which a workload is deployed, Araujo said as more responsibility for managing and securing applications shifts left toward developers in the age of DevOps, those teams need access to more sophisticated tools that are readily available and simple to deploy. There are several open source initiatives underway to provide that visibility, and one day many of them might coalesce into a single initiative.

In the meantime, as the dependencies between applications and IT infrastructure continue to increase, it’s clear IT monitoring as a core part of any set of best DevOps practices is now an absolute requirement to achieve and maintain observability.

— Mike Vizard

January 20, 2020
JFrog ConanCenter Extends DevOps Reach to C++

JFrog this week announced it has updated ConanCenter, a repository for storing and discovering software packages that can be downloaded for free, to include support for C++ artifacts.

Jagan Subramanian, vice president of community and partner engineering at JFrog, said ConanCenter provides a repository for the open source Conan package manager that JFrog includes as part of its continuous integration/continuous delivery (CI/CD) platform. By making ConanCenter available free of charge, JFrog is trying to encourage organizations that rely on C++ to build applications to embrace best DevOps practices, Subramanian said.

JFrog is also making available a free Community Edition of its Artifactory CI/CD platform for C / C++ available for download and later this quarter plans to integrate ConanCenter with other JFrog tools to make it even easier to securely access packages.

As a general-purpose programming language, C++ has been employed for decades to build applications in IT environments where resources are constrained, such as embedded systems. Subramanian said many organizations that rely on C++ to build those types of applications are experiencing the same pressures to accelerate application development as the rest of their software development colleagues.

To make it possible to achieve that goal, he said, JFrog is now extending its DevOps portfolio to include support for applications developed using C++ tools. ConanCenter not only provides a means to discover artifacts based on the metadata they generate, but it also provides a list of configuration options and a free binary building service that can be applied across multiple operating systems, platforms and compilers.

Despite its longevity and the rise of alternative programming languages, C++ continues to fulfill a unique role across a range of vertical industries. There are still plenty of C++ developers are that building some of the most challenging applications to develop. In fact, given the rise of robotics and internet of things (IoT) applications involving autonomous vehicles, interest in using C++ to build new classes of applications is on the rise. The ConanCenter repository promises to make it easier for C++ developers to share artifacts across those diverse projects.

In terms of adoption of best DevOps practices among organizations that rely on C++, Subramanian said it’s still very early, as most are still at the proof-of-concept stage. However, he said most regularly experience all the pain points that best DevOps practices address, which should drive more organizations to embrace DevOps in the months and years ahead.

It’s too early to hazard a guess as to how long it might take organizations that rely on C++ to embrace DevOps. Many of them have been relying on waterfall-based processes to build and deploy applications for decades. Nevertheless, just like in other programming environments, the need to build and deploy applications faster requires development teams to embrace change. As always, however, the biggest challenge when it comes to DevOps has little to do with the core technology platforms. Rather, the existing culture of the IT organization always proves to be the biggest hurdle.

— Mike Vizard

January 17, 2020
Software Supply Chain Attacks: How to Disrupt Attackers
Supply chain attacks—compromising an organization via insecure components in its software supply chain—are a growing concern for organizations. Throughout the past three years, an increasing number of open source software package repositories have been found to contain malware, making it clear that all installation and update pathways for software and library code must have security controls applied to prevent and mitigate supply chain attacks.

Growing Risks of Open Source Software to the Business

Digital transformation has made every company a software company. As a result, open source software has grown 75% over the past two years. Despite this growth, few companies accurately track the components used in their code. Most lack the policies, processes and tools to keep up with the choices made by their developers, which introduces risk. In fact, 60% of the codebases audited in 2018 contained at least one vulnerability.

Bad actors have turned to exploiting these weaknesses to compromise organizations. If production code is infected with malware or vulnerabilities, inadvertently sourced from the repository, it may contaminate all organizations that come in contact with it—whether by using the code already in their SDLC or by launching presumed trusted applications from third parties who failed to validate their own code.

Attackers are also shifting their techniques to mask malware among large packaged objects using obscure formats we refer to as destructive objects. They infiltrate at various points in the development processes opportunistically, knowing they can infiltrate systems outside a managed operational environment. They take advantage of third-party dependencies and use payloads that can sit and wait until an unsuspecting developer implements their software. Hence, software supply chain attacks are difficult to detect, and often achieve dwell-time of hundreds of days.

Assessing Supply Chain Surface and Sources

In order to assess weaknesses and identify what warrants continuous monitoring, we need to understand both the surface and sources.

When evaluating surface risks, consider the following questions regarding the development environment:

What technologies, repositories and packages are in use?
- Inventory technologies used in development: Python, Ruby, JavaScript, Node.js, .NET, etc. This can also refer to operating systems.
- Third-party repositories of code: PyPI, RubyGems, NPM, NuGet, PEAR or PECL, and OS repositories such as DebianRepository, CentOS, Ubuntu, AUR and more.
- Packages: There can be multiple packages from a single repository. How often are packages updated? How often are new packages added?
Are there update procedures in place?
- What are the dependencies for every package? How many other packages are pulled from a single package update, and how often are they updated and/or reviewed?
Who’s managing the verification processes across all these dependencies? What update procedures are in place if there is a personnel change?

Once these questions have been answered, they should be addressed regularly. The process must be iterative to be successful.

When considering attack sources, it’s important to understand the characteristics of repositories teams draw from in order to understand potential areas of weakness. Third-party repositories rarely infect a big project as they are typically found elsewhere (e.g. GitHub) and have thorough code reviews. Official repositories, unofficial mirrors and internal code repositories represent increasing risk potential.

Random malicious packages, typosquatting, bad code review and phishing developer credentials to hijack accounts are all less technical ploys used by attackers, but equally dangerous.

The Software Supply Chain Challenge

The size, breadth of formats, trusted third-party and open source risks, and misused certificates have made it nearly impossible to stop malware-infected files and objects with today’s signature based, sandbox and vulnerability-based solutions.

Threats evade dynamic analysis remaining unknown to defenders or lacking details when detected. The explicitness of signature blacklists misses new or never-seen-before code, and sandboxes or dynamic analysis solutions are resource intensive and unable to scale to the breadth and volumes of formats required. Without a way to inspect large or cross platform archives, installers and source code, it’s impossible to extract indicators of compromise. These destructive objects can also impact SOC triage, response and continuous improvement.

Rising to the Challenge

How can you address these risks, not only in the software development life cycle (SDLC), but as SOCs address vulnerabilities that may come through other applications and repositories? How do you evaluate and inspect wrappers, digital certificates, executables, scripts, files and libraries, resources, documents and icons?

It’s vital to establish security practices within SDLC, from training developers on secure coding practices using open source libraries to factoring in detection capabilities including static analysis, dynamic analysis, software composition analysis and manual penetration testing. Implementing a secure SDLC process ensures that the development effort is protected by these activities, augmenting code reviews and infrastructure analysis.

Teams can extend security controls into the SDLC with technology that provides deep visibility into all aspects of the supply chain, from open source dependencies through continuous integration and delivery of packaged applications. Capabilities that provide greater application security across code development and deployment activities include:
- File reputation identifies and classifies “known bad” and “known good” files by applying a file’s unique hash against a database of goodware and malware.
- Automated static analysis detects and classifies unknown malware in code repositories by deobfuscating and decomposing objects and files without executing, and extracting threat indicators.
- Dynamic analysis executes files in a safe environment and captures the behaviors and results of malware, allowing the security team to understand what is going to happen.
- Software composition analysis analyzes applications to detect vulnerabilities, dependencies and licensing requirements in software.
- Threat intelligence is served up at various stages of the analysis cycle, and represents evidence-based knowledge including context, mechanisms, indicators, implications and action-oriented advice about existing or emerging threats.
By incorporating these technologies into the SDLC, development teams can obtain continuous validation across the software supply chain. Further automation, implementing new security guidelines and advancing the activities within the existing framework will allow the enterprise to progress its security measures and close gaps in the SDLC and supply chain operations.

At the same time, it’s important to recognize that threats can still make their way into the software supply chain, and it’s equally important that the SOC remain vigilant about maintaining a clear triage and escalation process. Continuous monitoring of software repositories should be an integral part of good hygiene processes, and incorporated into best practices applied to operational infrastructure, including the SOC. The SOC serves as a centralized management infrastructure to respond, contain and remediate attacks. This includes aggregating and consolidating threat data, building a case for managing a response which can include actions to formulate triage and incident response plans, and moving beyond SIEM.

— Tomislav Pericin
January 17, 2020
Che: A Revolutionary IDE for the Mainframe

Tools such as the green screens of ISPF and the Eclipse desktop IDE, enhanced with proprietary plug-ins, have served mainframe application developers well over the years and, for those comfortable with them, will continue to do so. However, changes in the broader world of development are creating the conditions for a revolution in mainframe tooling.

First, mainframe developers are aging out of the workforce, leaving behind extensive code libraries and a workforce skills gap. Those likely to back-fill these roles know and love modern IDEs, especially Visual Studio Code, while the popularity of Eclipse is waning. For example, the annual Stack Overflow Developer Survey found that, over the past year, the popularity of Visual Studio Code grew from 35% to 51%, once again making it the most popular IDE, while Eclipse fell from 19% to 14%. According to Business Insider, 8.5 million developers worldwide use VS Code, creating a huge talent pool.

Second, application development has made great strides in productivity since the current mainframe dev tools were created, especially in DevOps automation with enablers like task runners, scripting and testing frameworks. The Stack Overflow survey cites scripting language Python as the most sought-after skill among developers, by far.

Finally, as the velocity of overall software delivery increases, mainframe is a critical component of digital transformation initiatives. According to 451 Research, 24% of companies are releasing application software daily or hourly while, similarly, DORA’s 2019 State of DevOps survey shows 20% of teams are deploying multiple times per day. Software delivery expectations have changed with continuous deployment becoming the new normal and, to remain a vital computing platform for the long term, mainframe app development needs to support this paradigm.

An Open Source and Cloud Development Technology

Rather than an evolution of the existing Eclipse IDE, Eclipse Che is a radical departure. According to the Eclipse Foundation, “Che 7 is the world’s first Kubernetes-native IDE that’s built from the ground up for cloud native application development. It simplifies and accelerates cloud development by allowing developers who are not Kubernetes experts to immediately contribute to cloud native application development efforts. In turn, Che 7 relies upon Eclipse Theia, which provides a highly modular and extensible IDE platform built on modern web technologies that runs on both your desktop or in your browser.”

For large enterprises, a good analogy for key value proposition of a hosted IDE is Git: usability and collaborative power have made Git the de facto standard for version control and Che’s workspace server offers comparable benefits.

Developers working on mainframe applications can use Che via the Che4z subproject. Che4z, whose primary goal is to bring a modern developer experience to mainframe software development, offers an extension pack providing access to the z/OS file system, remote debug and remote editing of COBOL programs with all the expected modern editor capabilities including syntax highlighting, code assist and real time syntax validation. After a beta period, Che4z is now GA.

Beyond the foundational benefits mentioned above, Che is revolutionary for the mainframe for many reasons.

Visual Studio Code and More

Che was architected around a VS Code-like experience from the outset which, with the growth of VS Code, means Che extends its value. Che is based on Eclipse Theia which provides an in-browser VS Code experience complete with the latest tooling protocols: language server, debug adapter and compatibility with VS Code extensions.

The Che4z sub-project, contributed by Broadcom, provides language support for both COBOL and High Level Assembler (HLASM). The language server protocol (LSP) enables developers to use these languages with Che, VS Code and any other LSP clients (e.g., IntelliJ, VIM, Emacs).

Bottom line: developers can use VS Code for mainframe development independent of Che adoption (the Code4z extension pack is available on the VS Code marketplace).

Cross-Platform Applications

For many Fortune 500 companies, the mainframe hosts critical system-of-record applications and data while the cloud and mobile applications have evolved from user-oriented systems-of-experience. With tech-first startups disrupting entire industries, large companies are deploying more cross-platform applications to maintain, if not win, market share by harnessing the best of both worlds. Cross-platform apps combine modern user experiences with the transactional power and data assets of mainframes.

Because cloud and mobile developers need to integrate mainframe applications, allowing them to use the tools they know and love is critical. This includes the popular code editors and IDEs they love today as well as the new innovations, such as browser-based, container deployed tools that make them more productive.

With its workspace server, Che also facilitates pair programming, which is a powerful approach when mainframe and distributed developers need to collaborate on cross-platform applications.

Bottom line: Che makes the mainframe less of a silo and more like other platforms.

Open Source-Enabled Vendor Independence

While open source is not new to the mainframe–zLinux has been around for years–access to off-platform toolchains became possible with the introduction of the Zowe open source framework (described in more detail below). Unlike the incumbent Eclipse vendors, and consistent with the open source vision, the Che4z extensions are open sourced and free to use. There is now an alternative to the proprietary, vendor-controlled model: true open source tooling for mainframe developers.

Furthermore, Eclipse locked mainframe developers into using a specific set of editors, runtimes and user interfaces for required capability and functionality–thus imposing a steep learning curve. Che, by comparison, offers flexibility via modular design, allowing individual developers to choose the components they need.

Whether developing for mainframe or other platforms, the Che community welcomes contributions from those passionate about software development. Want to incorporate a lesser-used language? No need to plead with a vendor. Anyone can seek community support for an idea or simply do it themselves. The Che/Che4z communities will ensure the proper procedures are followed before any code is ever released.

Bottom line: Che provides a more frictionless, fast-moving toolset allowing coding innovations to extend to the mainframe.

Onboarding and Cost Reduction

Consider the compatibility and maintenance headaches created by the heavyweight plugins required for Eclipse. For mainframe leaders, the need to reduce costs is an ongoing challenge as is the need to quickly onboard the next-generation of developers. As a container-based IDE with no client footprint, Che eliminates the need for local installation, maintenance and configuration. The potential of moving entire teams currently using Eclipse with customizations to Che provides an attractive opportunity to materially reduce operating costs while simultaneously improving developer productivity.

Because Che workspaces are containerized, new team members can onboard instantly–open the container, begin work. This means developers, who are high cost resources, can more easily move across projects, teams, departments, etc.

Bottom line: For mainframe leaders focused on staffing and costs, Che increases the talent pool while eliminating costs.

Built on Zowe

The extensions that open the mainframe to the Che IDE are powered by the Open Mainframe Project’s Zowe open source framework, which maintains mainframe-native security standards. Zowe, which was founded by Broadcom, IBM and Rocket Software and is the first open source project based on z/OS, enables developers to manage, control, script and develop on the mainframe just like cloud and other platforms.

Command line interfaces (CLIs), which come with platforms such as AWS and Azure, are popular with developers wanting the productivity lift of task runners, scripting and higher levels of automation (i.e., CLIs facilitate DevOps adoption). The Zowe CLI (the mainframe equivalent) magnifies the value of the Che4z stack as any set of commands that run through the CLI can be automated.

Bottom line: Using Zowe as the foundation for the Che4z mainframe extensions ensures the modern experience extends far beyond the IDE.

Path Forward

For leaders wanting to empower developers integrating mainframe code and resources with the most popular and powerful app development tools, Che provides the best of both worlds–the immediacy of VS Code use with the path to full-on, in-the-cloud team collaboration.

Expanded tool choice, greater alignment with off-platform peers and DevOps-fueled software delivery are all reasons why Che is not only the first Kubernetes-native IDE, but a real revolution for mainframe development.

— Jean-Louis Vignaud

January 16, 2020
SaltStack Extends IT Automation Framework

SaltStack today added three open source modules to its portfolio that make it easier to automate the management of IT at scale.

Heist is a plugin that allows organizations to create agents that will dissolve after a specific amount of time. Umbra is a plugin that gathers data streams to make it easier for machine learning algorithms to consume.

Finally, Idem is a declarative programming language that enables IT teams to create workflows based on stateful constructs that can be used in conjunction with any IT automation framework or systems management tool.

All three modules are based on the Salt Plugin Oriented Programming (POP) model. Company CTO Thomas Hatch said SaltStack created POP to make it easier to manage modules and libraries of code within a monolithic application without having to transition to a different microservices-based architecture based on containers.

That POP approach also serves to make it easier to test modules within a monolithic application environment, he said.

SaltStack used POP to create Salt, an open source IT automation framework that serves as the foundation for the commercial IT automation platform provided by SaltStack. Organizations that have embraced the open source Salt project include Juniper Networks, Cloudflare, Nutanix, SUSE, LinkedIn and eBay.

Heist extends the Salt project by giving DevOps teams an alternative to having to commit to either an agent or agentless-based approach to managing IT. Instead, DevOps teams can create an agent to collect data that will run only as long as it’s needed.

Umbra, meanwhile, prepares data to be consumed by machine learning algorithms transparently, which Hatch said is critical in providing visibility into how artificial intelligence (AI) models are constructed. That capability is going to be a requirement in any regulated industry that requires organizations to comply with IT audits, he noted.

IT automation frameworks are, of course, foundational to any set of best DevOps practices. While most of the initial IT automation management frameworks were proprietary, the entire category has gained more traction as open source implementations of these frameworks became available. Now DevOps teams have a plethora of open source and commercial IT automation frameworks on which to define a set of DevOps processes.

The biggest challenge in getting IT teams to embrace these frameworks historically has been the preference of many IT professionals for scripts they develop. However, as the scale of the IT management challenge has increased in the age of the cloud, relying on scripts that are often undocumented has become problematic. Industrial-scale IT organizations require IT automation frameworks that allow them to create reusable playbooks that are easily documented. Otherwise, every time an IT professional who authored a script leaves the company, the IT organization soon discovers the script they’ve been relying on that has broken can’t be fixed easily.

The day when all IT organizations rely on some form of an IT automation framework is fast approaching. In fact, the remaining IT automation challenges have a lot more to do with the culture within the IT organization than they do any of the technical capabilities of the platforms being considered.

— Mike Vizard

January 15, 2020
Perfect Sense Open Sources Gyro to Automate Cloud Provisioning

Perfect Sense has launched an open source project dubbed Gyro that provides an alternative approach to automating cloud infrastructure provisioning and management.

Gyro was developed as an alternative to other open source projects such as Terraform and Pulumi, said CTO Jeremy Collins. It is designed from the ground up around a programming language that allows DevOps teams to define infrastructure as code, and features logic built into configurations to make it easier to create more precise workflows, he said.

Adding logic to create equivalent workflows in rival platforms is more complex, Collins noted, adding the ability to add logic makes it easier to not only provision cloud infrastructure but also to roll it back when necessary.

Perfect Sense developed Gyro to make it easier to manage the infrastructure for its Brightspot content management system (CMS). As such, Gyro can support multiple cloud computing platforms.

In addition to making it possible to add cloud platforms to an existing configuration, Collins said it can be used to create custom commands and define multiple extension points directly within the Gyro language.

Perfect Sense developed Gyro to automate the provisioning of virtual machines. Since then, the company has extended its capabilities to add support for emerging cloud-native platforms such as Kubernetes.

Collins said Gyro provides a single extensible tool spanning a broad range of DevOps processes, from provisioning infrastructure to discovering what services already exist.

Despite the popularity of rival tools such as Terraform, it’s still early days in the adoption of infrastructure as code. Most IT organizations are a long way from standardizing on any one platform for automating the provisioning of infrastructure. In fact, it may not be uncommon for different DevOps teams within the same organization to employ different platforms for provisioning infrastructure.

Gyro’s ultimate success will depend much on Perfect Sense’s ability to foster an open source community. The Brightspot CMS is used by the Los Angeles Times, Walmart, Mattress Firm, Univision, Sotheby’s and US News and World Report, so Collins said there is a base of support for Gyro on which to build a community. Perfect Sense is looking to build a community that will continue to extend the reach and scope of the platform by adding additional plugins.

Regardless of the approach taken, as the number and rate at which workloads are being deployed on cloud platforms increases, it becomes critical for IT organizations to find a way to automate processes. It’s not clear to what degree those workloads will be provisioned by developers themselves versus traditional IT operations teams that have learned how to program. However, one way or another the provisioning of cloud infrastructure is going to be automated.

— Mike Vizard

January 13, 2020
Building a Real-Time App: 13 Open Source Projects You Should Be Tracking Right Now

Open source is the dominant model of software consumption in the modern era. Cutting-edge startups and entrenched incumbents alike find open source software development and community building a significant part of their overall business strategies.

For many open source projects, independent steering committees ensure that projects remain stable and are not beholden to a single company or profit motive. Alternatively, for open core projects where a single company financially supports and contributes the vast majority of open source code, open sourcing makes a project publicly available and accessible for potential new users. Opening a project to public contribution widens the pool of potential users and contributors, therefore ensuring more rapid innovation than commercial software offerings.

Choosing the Right Open Source Projects for Your Real-Time App

There are seemingly limitless open source projects to evaluate. Further complicating matters, each project has its own domain competence and cultural nuances. Considering a single distributed application will have tens or (many) more open source dependencies, development teams must continuously test new open source offerings and hone new skills to use these solutions, while simultaneously architecting their applications.

In order to help sort through the noise, I’ve looked at six categories of open source projects for this article: data storage, message systems, service meshes, REST frameworks and streaming frameworks. For each category, I’ll identify any dominant players as well as some other projects of note.

Here are the top 13 open source projects by category.

Data Storage: Cassandra, Hazelcast

Apache Cassandra, originally developed at Facebook, is one of the most widely used open source projects for real-time applications. Functionally, it’s a distributed NoSQL database management system. But practically, it’s become the linchpin of most at-scale, real-time applications. This is because Cassandra was designed to handle large amounts of real-time data across many servers, and has robust clustering support.

For applications that require extremely low latency or have exceptionally high data rates, Hazelcast may provide a useful alternative. Unlike Cassandra, Hazelcast is an in-memory data grid database, which enables development teams to benefit from the latency advantages of in-memory computing.

Message Systems: Kafka, Pulsar, NATS

Apache Kafka is the dominant open source messaging system used by application development teams today. Originating at LinkedIn, Kafka has grown into the de facto standard for high performance open source messaging systems. For streaming use cases, Kafka users can also turn to Kafka Streams.

Apache Pulsar, developed at Yahoo!, is another alternative messaging system which has been optimized for streaming data sources. Pulsar further differentiates itself with TTL message support and queuing via shared subscription.

For applications where latency is a critical consideration, NATS.io (a CNCF project) is one of the fastest pub-sub messaging systems available.

Service Meshes: Istio, Linkerd

Trends such as microservices, serverless and hybrid cloud deployment have contributed to the increasing architectural complexity of distributed applications. With hundreds or thousands of distributed application services, service meshes provide a mechanism for pulling all the disparate services together into a manageable and cohesive whole.

The most notable open source service mesh offering is Istio. One of the most popular open source service mesh projects, Istio is primarily used for applications built on top of Kubernetes. Another alternative is Linkerd, a flexible service mesh solution for Kubernetes or other frameworks.

REST Frameworks: Node.js, Spring

Node.js is a popular JavaScript framework for building scalable web applications. When comparing web development frameworks, Node.js typically ranks among the top three most (along with frontend frameworks React.js and Angular.JS). While Node.js can support large scale applications, it does not support real-time use cases well at scale.

Spring is another application framework, optimized for distributed microservices-style architectures. Like Node.js, Spring “focuses on the ‘plumbing’ of applications so that teams can focus on application-level business logic, without unnecessary ties to specific deployment environments.”

Streaming Frameworks: swimOS

SwimOS is a software framework for distributed streaming applications. Unlike REST-based frameworks, swimOS utilizes a unique and lightweight Web Agent architecture, optimized for composing and transforming many data streams in real-time. Whereas DevOps oriented projects, such as Spinnaker, seek to provide an abstraction layer to converge distributed cloud services, swimOS provides an abstraction across distributed data streams, so teams can build applications by composing many distributed data streams in a single mesh environment.

In order to ensure real-time consistency of distributed services, swimOS automatically ensures eventual consistency between all Web Agents. Complete with a UI toolkit and it’s own stateful messaging protocol called WARP, swimOS is the fastest way to build stateful streaming applications.

DevOps: Kubernetes, Spinnaker, RancherOS

Kubernetes is one of the most talked about open source projects in the world. Kubernetes describes itself as “an open-source system for automating deployment, scaling and management of containerized applications.”

While it may simply be a platform for container orchestration, rightly or wrongly, Kubernetes has become a stand-in for DevOps on the whole. In spite of its dominance, there are alternatives to Kubernetes, for example there’s RancherOS to manage Docker containers.

Furthermore, as application architectures continue to become more distributed and complex there is a need for platforms which can converge multiple distributed edge and cloud environments. Spinnaker provides an abstraction layer to connect the various environments and provides a simple mechanism for continuous integration/continuous delivery (CI/CD).

Final Thoughts

Your own particular use case should determine the appropriate open source technologies for your team’s real-time application. Whether you’re building an ultra-high performance low latency application or simply trying to tame large volumes of real-time data, knowing your options and choosing the right framework or platform can save you significant time and effort.

The open source software landscape is changing every day. If you have something to add about one of the technologies above or think I skipped over the next OSS breakthrough, I’d love to hear about it. Let me know your thoughts in the comments section below.

— Chris Sachs

January 8, 2020
GitLab Open Sources Monitoring Tools

GitLab has decided to make its monitoring and observability tools available as part of its core open source platform to help reduce the total cost of DevOps. Previously, the tools were available as part of the commercial tools in GitLab’s DevOps platform.

Kenny Johnston, director of product/Ops products for GitLab, said that as responsibility for observability shifts left, it is clear developers are now exercising more influence over which monitoring tools are employed. As developers begin to employ monitoring tools early in the application development process, most of them prefer open source tools because it’s unclear whether the application being developed will ever make it into a production environment, he said.

However, if the application does make it into a production environment, it can be challenging for developers to make a case for standardizing on whatever monitoring tools they’re using in place of an existing commercial offering employed by an IT operations team. In addition to reducing costs, the GitLab approach could serve to eliminate any potential friction in the DevOps environment, as applications move from development into production environments.

GitLab’s move to make its monitoring tools available as open source software comes at a time when more telemetry applications being employed to collect data are also based on open source software. In addition, developers of modern cloud-native applications based on containers are taking advantage of open source Prometheus monitoring software developed under the auspices of the Cloud Native Computing Foundation (CNCF) to monitor their applications. GitLab already provides integration with Prometheus.

GitLab is also making it clear it expects its monitoring tools to benefit from the contributions of a larger pool of developers. Specifically, GitLab is calling for help in the areas of ongoing integrations with Prometheus along with improved logging, tracing and alerting capabilities.

It’s not at all clear just yet to what degree DevOps teams will prefer to rely on open source versus commercial monitoring tools. While open source tools have some obvious advantages in terms of cost, many organizations have already standardized on commercial monitoring tools for a wide variety of existing applications. Open source tools often are more difficult to deploy and don’t tend to provide as much observability depth for IT operations teams. In addition, most of the providers of commercial monitoring tools are integrating their offerings with both open source telemetry software and Prometheus to bridge any divide that might occur between when an application is developed and deployed in a production environment.

The one thing that is clear is a lot more individuals within the IT environments will be accessing these tools now that developers increasingly are being held accountable for the performance of their applications in production environments. In addition, modern applications require monitoring tools to make sense of all the dependencies that existing between microservices. Previously, IT operations teams tended to limit usage of monitoring tools based on commercial software to their most critical applications because of the costs involved. With the rise of open source monitoring tools, the opportunity to apply monitoring more broadly becomes more economically feasible.

Regardless of the path forward chosen, the more visibility into IT environments provided, the easier it becomes to embrace best DevOps practices.

— Mike Vizard

January 3, 2020