devopsy.kinsta.cloud

Category: Enterprise DevOps

F5 Networks Planning Open Source Projects Beyond NGINX

F5 Networks today revealed plans to launch multiple projects in the months ahead that will extend its commitment to open source beyond the NGINX proxy software the company acquired in 2019.

The announcement was made during an online NGINX Sprint event; two of the projects are extensions of NGINX that will address networking requirements between microservices-based applications in addition to the management of fleets of NGINX instances. The third project is a platform-agnostic open source developer tool for building modern applications that does not necessarily require NGINX proxy software.

At the same time, F5 Networks also unveiled an open source NGINX Modern Apps Reference Architecture that gives IT teams access to a complete, fully operational microservices-based application that can be deployed in minutes on a Kubernetes cluster. Based on a modular approach made up of open source components that come pre-wired and pre-integrated, F5 Networks is committing to extending this framework in collaboration with multiple third-party vendors and organizations.

Rob Whiteley, general manager for the NGINX product group at F5 Networks, said the company is also committing to modernizing its NGINX proxy software to leapfrog rival open source Envoy proxy software being advanced under the auspices of the Cloud Native Computing Foundation (CNCF). NGINX is still more widely employed than Envoy largely because it is lighter-weight and easier to deploy, but Whiteley conceded that Envoy has more advanced capabilities.

Envoy, of course, is at the core of the Istio service mesh. F5 Networks supports both Istio as well as a service mesh it has developed on top of NGINX. Depending on the number of application programming interfaces (APIs) that an organization needs to support, an IT team might make use of proxy software, ingress controller or an API gateway for a service mesh to manage them.

Finally, F5 Networks said it will participate in the Kubernetes ingress project and that it is joining the gateway API community as its implements its own NGINX-based gateway controller to ensure its proxy software and Kubernetes remain tightly aligned in the future.

In general, Whiteley said F5 Networks also is making a commitment to be clearer about where the line between the open source and commercial editions of its offerings is drawn. As a rule, the open source editions are designed for developers and small teams while the commercial version is designed to scale within enterprise IT environments. The overall goal is to promote innovation without enabling cloud service providers to reap all the revenue benefits of F5’s research and development by creating a service based on that open source software, said Whiteley.

As part of that effort, F5 Networks is expanding the number of open source projects it supports to make it easier for developers to contribute code beyond the core NGNIX proxy software, said Whiteley.

It’s not clear to what degree F5 Networks will be able to rally the open source community to advance additional projects beyond NGINX. The company is promising to provide additional details at an unspecified future date. However, Whiteley said there is still plenty of room for innovations not only in emerging technologies such as service meshes but also proxy software that is already widely employed within many enterprise IT organizations. Especially since most of them are embracing microservices-based applications that will ultimately change every aspect of IT.

August 23, 2021
Edge Computing: Tech’s Next Trillion-Dollar Opportunity

Even as technology continues to move into the cloud, a simultaneous shift is already underway in the opposite direction as computing increasingly moves to the edge. One key reason: by 2025, the number of connected devices in use is expected to exceed 56 billion units, according to IDC. Edge computing is not a successor to the cloud, but another expansion of technology’s reach that represents an exceptional opportunity for investment and growth.

As edge-device and computing adoption continues to skyrocket, organizations need to process the proliferating data from these devices locally so they can act on it in real-time. This will require investments in edge-adapted infrastructure and related tools and platforms. Enterprise technology is about to be transformed yet again — the need to process and analyze some data at the point where it is both created and consumed is becoming a business requirement as mandatory as moving other parts of the technology stack into central locations in the cloud.

Organizations already must deliver compelling customer experiences with edge devices based on data that must be continuously refreshed. The alternative—uploading and downloading data to and from a cloud service or local data center—often takes too long. Whether the data is on a consumer’s smartphone or a manufacturing floor, organizations need to process and analyze data as near as possible to the point where a real-time data-driven response is required.

The competition to provide these capabilities is already intense. Many vendors—ranging from cloud service providers to telecommunications carriers to longtime providers of server and storage infrastructure—view edge computing as their next trillion-dollar opportunity.

Why We Compute at the Edge

Edge computing has been around for decades in various forms. There are countless examples of standalone software being deployed on, for example, devices used on factory floors that automate various aspects of a manufacturing process. In recent years, many of those systems have been connected to the Internet to enable organizations to collect and share data more easily. The challenge now is that the amount of data being generated at the edge often exceeds the point where it is practical to transfer that volume of raw data across a wide area network (WAN). Platforms that can process and analyze large amounts of data at the network edge are required. The aggregated results of data processed on an edge platform can then be shared more efficiently with other applications running in the cloud, in an on-premises IT environment or even on the network itself.

Low-latency requirements are today’s biggest challenge. In many scenarios, an application running on a gateway at the edge of a telecommunications network needs to be able to respond in a fraction of a second to a mobile computing application’s request for data. In other circumstances, an edge application is driving an automated factory operation that needs to dynamically adjust to analytics processed at the edge of an extended enterprise.

There are even more advanced use cases, some of which are described below. These include in-car computers that process computer-vision data in real-time for driverless vehicles and robotics applications that promise to transform manufacturing.

Hardware Advances Enable Edge Computing

The often competing demands of low-latency network connections, high-power computing and large storage requirements are driving innovations in edge hardware. Electric carmaker Tesla, for example, has added custom-made devices to its processors capable of pre-processing massive amounts of data, performing machine learning inference to make quick driving decisions and predictions and even do some deep learning model training on the vehicle itself–before sending a subset of the data to central systems to do even more training.

Earlier this year, Tesla unveiled a processor with six billion transistors. According to the company, its device boosts performance by 21 times compared to the Nvidia GPU it replaced in its Tesla Model S, Model 3 and Model X models. These powerful devices are used in addition to CPUs from traditional providers such as AMD and Intel for workhorse applications like infotainment.
With this computing power at the edge, a Tesla car can process sensor data that allow it to recognize and self-pilot around pedestrians, other cars on the road, emergency roadside signage and potentially hazardous moving objects. This sensor data is also processed with pre-loaded mapping data and GPS connectivity. Additionally, Tesla’s in-car computers relay information over Tesla’s network to massive storage facilities where the data is analyzed to improve Tesla vehicles’ self-driving and other functions. Improvements to the vehicle’s software stack are then downloaded to it over the network.

Healthcare is an example of a sector with similar needs for advanced machine learning-taught devices and software. Images from MRI and other scanning devices with inference-computing capabilities deployed at numerous locations would also require computing power and network infrastructure similar to that which Tesla’s cars need. Most hospitals, however, don’t have the interest or wherewithal to build their own powerful edge computing hardware and will need to consume this capability from technology vendors.

As advances in semiconductor circuits approach seven nanometers and smaller, the ability to run applications at the edge in an energy-efficient manner becomes more realistic for a wider range of organizations. With each new iteration of processors, the total cost of edge platforms will continue to decline while the amount of horsepower available to run applications steadily increases.

Networking at the Edge

Regardless of how much data is processed and analyzed at the edge, the sheer volume of data creates more demand for network bandwidth and, consequently, the potential for bottlenecks. One main bottleneck is at the backhaul—the stretch between the edge towers and central servers and network. Whether tomorrow’s edge network infrastructure will consist of 5G, more ultra-powerful WiFi connections and devices, low-latency daisy chain systems, fiber systems such as those that Google is developing, satellite connectivity, yet-to-be-created technology or a combination of these is still unclear–but one thing is certain: Data volumes are increasing faster than network bandwidth.

Today, organizations have a choice of connecting edge platforms among several available wired and wireless network alternatives. The number of use cases involving edge computing is expanding in part because wireless 5G networks make it possible to share data across multiple interconnected edge platforms. For example, manufacturing employees wearing augmented reality headsets that can share data and analytics in near-real-time not only with their coworkers on the floor but with colleagues hundreds of miles away.

As telecom carriers continue to virtualize the network infrastructure they employ to deliver 5G services, the cost of delivering them should steadily decline across a widening range of geographic areas. Today, a 4G network can support roughly 4,000 devices per square mile. A 5G network could pack up to three million devices in the same area.

It’s still early days as far as 5G is concerned. A Capgemini survey of 1,000 industrial organizations that plan to incorporate 5G into their operations finds less than a third (30%) have progressed to trials and real-world implementations. Yet, carriers are already at work defining the next generation of 6G wireless standards that promise to deliver another giant leap for wireless networks in another decade.

Regardless of the network employed, organizations will still need to strike a balance between how much data needs to be processed, stored and eventually disposed of at the edge versus the amount of aggregated data that needs to be transferred over a WAN and incorporated within various other platforms that process data centrally or function as systems of record.

Factoring in Security

Arguably, the biggest obstacle to edge computing is security. Each time an organization deploys an edge platform, the attack surface they need to defend grows. Many of these edge platforms are also managed by operational technology (OT) teams that don’t have as much cybersecurity expertise as does, say, the data center security team at Google. Most organizations are trying to meld their OT and traditional information technology (IT) teams in an effort to deploy, manage and secure edge platforms. However, the cultures of these two teams are significantly different. It takes time for organizations to harmonize the efforts of these two groups.

More challenging still, cybersecurity expertise is hard to find and retain. Cybercriminals have discovered how vulnerable OT environments are. Attackers affiliated with various nation-states have already demonstrated that industrial control systems controlling electricity grids and other critical infrastructure systems are primary targets should war erupt. From their perspective, each edge computing platform connected to a network is yet another vehicle through which malware can be introduced into an IT environment. It only takes a few minutes for malware to propagate laterally across an entire platform.

Ultimately, organizations will soon spend billions of dollars securing edge platforms. Once the edge systems are locked down and secure, edge processing can improve overall cyberdefense—processing more data at the edge without moving it around as much can dramatically reduce the attack surface.

How the Edge Computing Ecosystem is Evolving

Once the right level of infrastructure is in place, the next challenge is building applications. The widespread adoption of container technologies has allowed developers to build applications using artifacts that require much less memory and storage. Most container environments today, like those based on Docker and Kubernetes, are decentralized, in a sense, but are run in centralized environments like data centers. But work is progressing on even smaller container formats that make it even easier to deploy application software on, for example, an oil rig hundreds of miles from any server or gateway.

Lighter-weight instances of the open source Kubernetes container orchestration engine are starting to gain traction. Kubernetes, by surfacing a consistent set of application programming interfaces (APIs) regardless of what platform it is deployed on, provides an opportunity to centralize the deployment and management of modern microservices-based applications across highly distributed computing environments.

Industry API specifications and frameworks are also beginning to emerge for machine learning, data science and other technologies that empower edge computing. An example is oneAPI, supported by Intel and others, which provides a framework for developing edge applications.

Providers of development platforms are also racing to create frameworks that make it simpler to build applications within edge architectures that seamlessly invoke backend application services running in the cloud. Distributed event-streaming platforms like Apache Airflow and Kafka can play a role in allowing edge computing platforms to share data with those backend platforms at scale. Some organizations will also invoke the services of content delivery networks (CDNs) to improve the performance of applications deployed across a global network of points-of-presence (PoPs) that have already been constructed.

Finally, stateful applications deployed at the edge can require local databases that provide access to persistent forms of data. The bulk of container applications deployed today are stateless—they store data on an external storage system. Building stateful applications that access data in containers on a local Kubernetes cluster is more challenging.

There are already millions of developers familiar with container platforms who are standing by until improved hardware and software simplifies and secures deployment of highly portable applications as far out on the edge as possible.

The Coming AI and Data Management Challenge

The most exciting edge innovations ahead will likely be driven by the rise of machine learning and AI. Today, AI models are typically trained within centrally located cloud computing platforms. An AI and machine learning inference engine can then be created and deployed in a production environment, where a trained AI engine deduces what actions to take as new data is discovered.

Over time, those inference engines are subject to drift as the amount of new data collected exceeds the parameters on which the original AI model was based. Data science teams working in collaboration with IT and OT teams then need to train a new AI model to replace the inference engines at the edge. Advances in new AI and machine learning capabilities, in conjunction with lower-power and higher-performance processors and other components to help lower costs, can improve the accuracy and performance of prediction systems, but typically doesn’t reduce the degree to which this drift happens.

If AI models can be at least partially retrained at the edge, this can help to reduce prediction accuracy drift. However the software and hardware infrastructure required to retrain an AI model efficiently within an edge platform is still very early-stage. That said, as the cost of increasingly high-powered hardware declines and edge machine learning software systems improve, the ability to train at the edge will become more feasible.

The Economic Impact and Investment Opportunity

The global edge computing market was valued at approximately $3.6 billion in 2020 by Markets and Markets. The research firm forecasts this market will reach $15.7 billion by 2025, representing an astounding 34.1% compound annual growth rate (CAGR).
McKinsey forecasts that the economic value generated by use cases involving internet of things (IoT) applications alone will be somewhere between $3.9 trillion and $11.1 trillion per year by 2025. Add in all the other use cases for edge platforms and the potential for positive economic impact becomes enormous.

Increased computing at the edge will be a huge technology market in its own right and will also drive increased demand for centralized cloud computing. Gartner notes that by year-end 2023, 20% of installed edge computing platforms will be delivered and managed by hyperscale cloud providers. The other 80% of this massive emerging market opportunity is up for grabs.

Lonne Jaffe co-authored this article.

August 23, 2021
The Pace of Change Requires Better Testing
The COVID-19 pandemic in 2020 created a pressing need to give people additional ways to do their jobs and handle the tasks of day-to-day life via mobile devices and forced the growth of app innovation at a never-before-seen rate. Thanks to new application development technologies and agile processes along with open source and embedded libraries, organizations are delivering greater functionality, faster, and for multiple devices (think web, mobile, tablet, etc.). And while a never-ending stream of new apps and updates has been great for consumers, it’s proven to be a pain point for many enterprises whose testing processes aren’t up to speed, literally and figuratively.

While test automation is an essential component of agile development, it’s not addressed in published agile methodologies such as Scrum and Kanban. Of course, testing tools and methodologies have changed along with modern development practices, but they haven’t always kept up with today’s rapid-fire pace of AppDev, and the past 18 months have only served to elevate the need for improved testing. For companies to keep up with and move beyond their competition, testing must be conducted in parallel with development.

Automated and Continuous Testing: Analogous, not Synonymous

There are many ways to test software, including manual, user interface (UI), automated and continuous testing (CT), but most enterprises today employ some form of automated testing and, increasingly, continuous testing. All too often, people throw the terms automated and continuous around as if they are synonymous, but they aren’t. So, before we delve into the merits of one over the other, it’s important to have a clear understanding of what each of the terms means.

Automated testing refers to conducting software tests programmatically with little to no human interaction. It measures actual outcomes in relation to predicted outcomes, and, as it’s less time-consuming to execute than manual testing, has become the norm for modern enterprises. Continuous testing, meanwhile, uses automated tests to provide uninterrupted feedback at every stage of the software development life cycle and, therefore, plays a key role in DevOps.

Automated and continuous testing work hand-in-hand to ensure software quality, cost-savings and speed time to market and should be conducted in lockstep with development. Whereas it’s possible to have one without the other, it’s far from advisable, however.

The Upside of Continuous Testing

Continuous testing takes automation to the next level and, as such, holds several advantages over automated testing alone.
- Early and actionable feedback at each development stage means issues can be addressed sooner—before they evolve into larger, more complex issues.
- Agile teams’ velocity is measured by how much they can complete in a sprint. Because of this, a team’s testing focus is narrowed to the sprint they are immersed in, meaning that problems affecting functionality previously deployed often aren’t discovered until later in the development life cycle, or they are discovered by users after deployment.
- An automated and continuous approach means more problems are identified and fixed before deployment.
- Release cycles are accelerated as organizations can respond to market changes faster and in real-time.
- Correcting problems early reduces rework, which translates into cost and time savings.
- Continuous integration requires continuous testing.
- CT tests not only the parts of the application that have changed but also how those updates work within the app’s entire functionality.
- Unlike automated testing, CT requires no human intervention.
What’s the Holdup?

Despite numerous benefits, CT isn’t as widespread as one would hope. One of the reasons for the disconnect comes down to investment, whether it’s hiring the right people, training or tools.

Perhaps the greatest stumbling block comes at the hiring stage. CT requires a specific set of technical skills coupled with a natural aptitude for test development and a mindset focused on resiliency. Beyond investing in staff, there are upfront (and ongoing) automation costs to consider. Unfortunately, it’s not as if you can implement CT and never touch it again—CT becomes its own software delivery pipeline that must be developed, maintained and enhanced to ensure it doesn’t become obsolete.

Continuous Testing: What Could go Wrong?

As the pace of AppDev quickens, so does the imperative to integrate it into DevOps. Failure to do so has serious implications, the most obvious of which is that without CT, companies fail to test as much and as frequently as they should. Regression tests aren’t done as deeply, for instance, so when a new app or update is released, it’s more likely that something will break. Further, without the right skill sets, teams have a tendency to examine whatever is new and fail to see how it fits into the whole. Moreover, without proper testing, you may inadvertently introduce bugs or discover too late that functionality doesn’t work as planned. Relying on manual and slow, fragile UI testing will only hinder progress in organizations that don’t adapt.

Negative consequences go beyond the confines of an agile team or enterprise as poor software design and implementation feed into the all-important issue of customer (dis)satisfaction. As everyone knows, unhappy customers aren’t loyal customers, and that translates to a lower bottom line. Another equally serious (perhaps more serious, depending on your perspective) ramification is the potential impact on data security, exposing organizations to a very different but significant risk posture.

Thanks to DevOps, ideas are moving from deployment to business value at lightning speed. It stands to reason, then, that testing also must be faster than ever.

When you consider that what you see today is the least amount of change you’re going to experience and that tomorrow is guaranteed to bring more, it should be abundantly clear why continuous testing is so important: The less testing you do, the more problems will occur downstream.

Businesses develop software for a reason—to get business value. If your app isn’t working, you aren’t reaping its full value. So, instead of asking yourself if you should implement continuous testing, you need to be asking yourself why you haven’t done it already. By failing to invest in CT now, you will undoubtedly pay for it later.
August 20, 2021
Yeah, You’re Doing DevOps

The Stack Overflow annual survey of developers came out this month, and it has a lot of interesting data points in it. There is a lot that I wish was different about their questions/methodology, but one thing caught my attention. Docker, Kubernetes and similar tools (Ansible and Terraform are in there at lower percentages, for example) make up more of the responses than can be accounted for by people with “DevOps” in their title.

Sometimes by a lot.

That means we have a fair number of developers that are doing DevOps, but who aren’t DevOps team members. Yeah, there are a few systems admins in the ranks of the respondents also, but not enough to cover the discrepancy. If every respondent that was in DevOps and sysadmin roles used Kubernetes, there would be just enough to cover the number of respondents that said they used Kubernetes “extensively” over the past year. It’s unlikely that all of them are Kubernetes specialists. And that doesn’t touch on Docker, with a whopping 40% of respondents saying they use Docker extensively. Any developer with recent experience can explain away Docker—though “extensive” is a bit strong for your average developer’s usage, it is becoming part and parcel of development, using containers to host dev environments, test, even build tools. And, of course, some deployments. But 40% is still a lot.

So, if your title is developer, and you’re using deployment tools on a regular basis, go ahead and call yourself DevOps. That’s not the only criteria, but if you were going to pick one, the breadth that using deployment tools represents combined with the implication that there isn’t an ops person worrying about that type of thing makes “uses deployment tools” a good stand-in test.

To the nearly 18% (20% of pro developers) of you using Yarn, I tip my hat. I used it for a couple of years early on in its life, and am happy to say I do not need to use it any longer. I assume your usage is not tied to Hadoop and that it has greatly improved over my 2015-2016 experience. Still, y’all rock, just by definition of using it to get resource scheduling down. It ain’t easy.

Across the board, the number of DevOps/Ops tools that make these lists is amazing for a developer survey – nearly 9% of respondents use Ansible, and nearly 9% use Terraform. There is undoubtedly overlap, so the actual number that uses one or the other is less than 18%, but still, nearly one in 10? That’s huge. And that’s DevOps or Ops, not purely developers. Regardless of title.

And that’s been a hallmark of development since the field took off—doing what needs to be done to deliver software. This is just another iteration, and you’re rockstars, regardless of title. You keep delivering apps that run the business—let them call you whatever they like. Money talks; titles are pretty much meaningless. Not like you’ll retire on a title, but bonuses or stock grants? Those you might retire on.

August 18, 2021
How CIOs Can Measure Digital Transformation ROI

To fully grasp how to measure success and gauge progress toward goals in any digital transformation, it’s important to understand the shifting expectations of CIOs who must report these objectives and outcomes. The pandemic has substantially changed the role of the CIO, giving them heftier goals and a shift away from IT.

Today’s CIO is no longer solely responsible for the IT department. Instead, the expectation is for them to have a greater influence in solving strategic business challenges. CIOs are now held to higher standards and have great responsibilities, with the assumption that they excel at communication and facilitation. Today’s CIO role also includes business planning and monitoring, transformational development, testing and security and compliance.

Gartner states that, in a technology-driven workplace, 41% of tech producers are within other business units, not in IT. Solutions are being created from those in charge of the bottom line, who make organizational and business decisions. Further, the rise of low-code platforms has accelerated the shift from IT to the business. Finally, as Generation Z moves into the workforce, there will be a significant shift in what this workforce wants and needs to be their most productive and fulfilled. Typically, this generation of workers are masters of autonomy and they strive to impact everything they touch and have greater control over what they handle and take on.

Within this shifting landscape, it can be difficult to understand how to measure and set goals.

Acquire the Right Tools and Frameworks

The first thing CIOs need to provide to this decentralized organization is the right tools and frameworks. These tools, such as value stream delivery platforms (inclusive of CI/CD, testing, artifact management, etc.) coupled with agile frameworks like SAFe provide a consistent method to deliver apps within the organization.

CIOs need platforms that allow them to visualize, manage and monitor progress on an ongoing basis. While this was largely a manual process before, solutions have vastly improved and, luckily, we no longer need to sit in a room and manually whiteboard progress.

Leverage Value Stream Management

This is often new territory for CIOs and can be daunting. CIOs need to be able to protect the organization, facilitate the quality of its solutions and take accountability for the tools being used and sold to its customers. Fortunately, this new power and influence can be managed with value stream management. It is what allows CIOs to keep track and monitor the bottlenecks in the development process. Further, the CIO can provide collaboration tools to help connect users around common processes.

Value stream management can be a powerful improvement tool for organizations. It can serve as a material and information flow map that helps workers and executives continually view their performance and improvements. “Value” emphasizes that the purpose of the work process can prove the value and worth to the customer, while the term “stream” means that the flow of this value is being measured throughout the process.

Set Business Goals, Not Just IT Goals

Digital transformation success for the CIO not only means that technology works, but that it works for the business. That means the goals CIOs set for technology need to meet whatever bottom-line goals are set for the business, too.

Define KPI objectives that align specifically with the business’ goals and focus on relevant metrics that matter. There are a number of types of metrics that CIOs should consider, but each organization will have to choose which are most relevant to their unique business, industry and their business goals: User engagement, customer experience, reliability, availability, active usage (think: Conversion rates, repeat customers/users, daily active users) are all examples.

When choosing your metrics, find those that you can accurately benchmark against past performance. A baseline will prove your worth; without it, it’s difficult to understand value gained. Ultimately, the CIO must ensure that these metrics are valuable to everyone, and stakeholder buy-in is critical. Finally, everyone in the organization needs to be on board with these goals and metrics.

August 17, 2021
DataOps Vs. DevOps: What’s the Difference?
There is a mindboggling amount of data today; to even measure it requires using a byte measurement called a zettabyte, which is one sextillion bytes (that’s 21 zeros). Currently, because such a ridiculous amount of data exists, there is a growing urgency to end wasteful data processes. From this environment, DataOps was born. Similar to the way that enterprises adopted DevOps to formalize and streamline wasteful development practices in the past, today, many large organizations turn to DataOps to formalize modern data management practices.

What, Exactly, is DataOps?

Primarily, many companies are adopting these principles because they are either trying to avoid or rectify data debt, which is the amount of money required to fix data problems due to the mismanagement of data processes. Data debt is a strong motivator for revamping outdated processes and policies, particularly when decision-makers and stakeholders require metrics before implementing change. Unpaid data debt can be detrimental to a business; the longer it remains unpaid, the more it costs to maintain a data landscape.

By implementing DataOps principles and data governance, an organization can effectively reduce its data debt and prevent it from growing any larger. Moreover, DataOps practices and software engineering can be used to detect inefficiencies, minimize knowledge loss and capitalize on missed opportunities related to data usage.

Similarities Between DataOps and DevOps

Much of the processes that enable DataOps were borrowed initially from the same foundations that built DevOps. Likewise, just as companies need DevOps to provide a high-quality, consistent framework for software and feature development, data enterprises also rely on these same features to realize rapid data engineering and analytics development. For organizations that already have a DevOps framework in place, leveraging DataOps is relativity straightforward. Several important DevOps concepts adopted by DataOps include:
- Agile development
- Focus on delivering business value
- Continuous integration and continuous delivery (CI/CD)
- Automated testing and code promotion
- Reuse and automation
The Differences

Despite the similarities between the underpinnings of DevOps and DataOps, there are several major differences.

The human factor: The people using DataOps and DevOps have divergent personalities and skillsets. DataOps participants may be tech-savvy, but often their knowledge is theoretical. DataOps professionals can include data engineers, data scientists and analysts who focus on creating models and visual aids. DevOps, however, was made for software developers and engineers–coding is in their DNA.

The process: The life cycles of DataOps and DevOps do share similar interactive properties. But, the former deviates in that it consists of a data pipeline and an analytics development process, both active and intersecting. While conceptually, the pipelines of DataOps resemble the development processes of DevOps, typically, experts note that the DataOps process is more challenging.

Orchestration: In the DevOps process, application code does not require complex orchestration. However, for DataOps, both the data pipeline and analytics development orchestration is an essential component. Although orchestration in the DataOps pipelines occurs frequently and drives data flows, there is usually no such coordination of pipelines in application development and DevOps processes.

Testing: Again, the two pipelines of DataOps create a significant difference from DevOps; testing in DataOps occurs during both the data pipeline and the analytic development process. These tests attempt to catch anomalies, flag abnormal data values and–unlike DevOps–validate new analytics before deployment. Likewise, these tests get embedded into a data quality framework for continual monitoring.

Test data management: In most DevOps environments, test data management hardly takes priority; with DataOps, it’s vital to accelerate analytics development so that innovation keeps pace with agile iterations.

Tools: DevOps is the ‘father’ of DataOps, and as such, the tools needed to support the latter are still in their infancy. While testing in DevOps is primarily automated, DataOps doesn’t have the same luxury – most users modify testing automation tools or build their own from scratch.

Exploratory environment management: Generally, data teams use more tools than software development teams. Moreover, exploratory environments in data analytics are more challenging from a tools and data perspective; data teams also naturally depart from data islands across the enterprise.

While foundationally, the concepts of DevOps serve as a starting point for DataOps, the latter involves additional considerations to maximize efficiency when operating data and analytical products. Nevertheless, both serve their intended audiences, reducing data debt and evolving data products or shortening systems development life cycles or providing continuous delivery. For businesses looking to make internal data-related processes more efficient, they should start by examining best practices associated with DevOps.
August 11, 2021
Should You Hire a Clojure Engineer?
Quite often, when developing applications, it becomes necessary to execute long-running code at the same time as performing other tasks. More recently, to run a program on the Java virtual machine (JVM) it was necessary to write the program in Java. Those days are long gone due to the wide variety of programming languages for the JVM. Clojure is a general-purpose dynamic language developed by Rich Hickey for the JVM. One of the main goals of an engineer who speaks this language is to simplify the process of writing a program with multiple threads that are accessing data simultaneously. What, exactly, is the role of a Clojure software engineer and how can they improve the functionality of its products?

What Problems can a Clojure Software Engineer Fix?

A good programming language should help us write transparent and straightforward programs. The software systems that humans create are inherently changeable. In development, it is impossible to foresee all situations in advance, especially when the system’s complexity only grows over time. Even the slightest code change can potentially lead to significant changes in overall system behavior. As a developer, it’s necessary to study carefully, analyze the code and predict each shift’s side effects. Over time, and with greater complexity, it becomes close to impossible to thoroughly analyze all the possible consequences of our changes.

The Clojure language was created to overcome these challenges. Clojure has a more convenient and modern syntax that orients the user to the functional paradigm as much as possible. Despite the dynamic typing of the language, Clojure programs are compiled into JVM bytecode and integrate seamlessly with existing Java programs. Clojure has now been implemented for several platforms, the most notable of which are the JVM, .Net, and JavaScript. Because it runs on the JVM, Clojure code is portable, stable, reliable and secure. Each operation in Clojure is implemented in three forms: function, macro or special form.

Typically, complexities in computer programs cause side effects. This language helps to localize those and makes writing functions cleaner. This makes testing easier for the developer, too, since they don’t have to try different queues and find the correct state of the inputs.

What to Look for When Hiring

Large companies in particular value senior full-stack Clojure software engineers who can work across a number of tasks from analytics to coding, testing and DevOps.

Among the problems that a Clojure software engineer can solve for an enterprise are:
- Development of front end and back end systems
- Creation of algorithms for load distribution by competencies/user workload
- Design of modules for forecasting the load users create on systems
- Machine learning in Clojure.
Here is the example of a Clojure engineer’s résumé:

Clojure is also an expression of a developer’s ideas about how simple a programmer’s tool should be. Clojure typically uses only four data structures: list, vector, display and set. The main API is built around these four structures. The main abstraction in Clojure is a pure function that has no side effects and is therefore much more reusable.

To date, many libraries have been written for Clojure, including those for developing simple and modular web applications. A Clojure program is an open architecture constructor, which means developers can choose for themselves which libraries to use regardless of which others they may have chosen previously.

Clojure Engineer Salary

Salaries for Clojure software engineers in the U.S. average $129,887, while a Clojure software engineer’s salary in the UK is $69,910. In Canada, the typical Clojure engineer’s salary is $61,780 and in Germany, $69,089. Clojure software engineers in Ukraine earn an average salary of $38,000.

Value to the Enterprise

Enterprise software performance requirements dictate the need to use parallel computing to interact with each other and work at the same time. Existing languages are poorly suited for parallelizing tasks. They contain many object and state dependencies, which translate to systems with unpredictable behavior in parallel interactions. Solutions based on locks and traffic lights do not greatly simplify the implementation of these tasks; the fundamental problem of the system’s state’s dependency on the sequences of object interaction remains.

By using Clojure in web development, a programmer can develop using other minor, task-focused languages. ClojureScript, for example, compiles quickly to JavaScript and runs in the browser. All of this provides an entirely new approach to building client applications.

Since Clojure seamlessly integrates with any Java code and can be deployed on standard Java application servers, it is well-suited to web application development, too. The flexibility of the language and its incredible expressiveness allows developers to create fully functional web applications very quickly.

All a developer needs to work with Clojure is the JDK and the Clojure library which are contained in a single JAR file. There are two standard ways to develop and run Clojure programs: The most common way is to use a simple interactive REPL, a read-compute-print loop.

Since Clojure is a general-purpose language, it is not difficult for a programmer to find a use for it. Since it is based on the JVM, there is already a massive selection of libraries available, which significantly lowers the entry threshold for developers skilled in other languages and removes the need for developers to write code from scratch.

Clojure can be used wherever modern, efficient, stable data structures are needed. Of course, there is no one-size-fits-all tool that can handle all types of development tasks. However, the fact that a Clojure developer can use the language for client-server communication is yet another advantage.
August 10, 2021
The SolarWinds Query Report 2021: Database Priorities and Pitfalls
The rise of hybrid IT and the accelerated adoption of cloud-native apps have evolved traditional database design, management and monitoring. Workloads are rapidly shifting to the cloud and application performance is heavily dependent on the performance of the underlying database. Meanwhile, the global COVID-19 pandemic has forced companies to make significant changes in the last year. Although it was and is a necessity, the transition to remote work and many organizations’ rapid pivots in strategy posed significant challenges for the technology professionals tasked with supporting new employee demands and shifting business priorities.

As a result, organizations are increasingly acknowledging the importance of database performance management and strategy to ensure their most valuable asset: Data. SolarWinds recently revealed the findings of The SolarWinds Query Report 2021: Database Priorities and Pitfalls, which examines the current state of database management, evaluates the primary challenges database professionals face today and outlines how database professionals can best overcome these challenges.

The North American results are based on responses to an online survey fielded in February 2021, which included participation from 331 technology professionals responsible for database performance management in the U.S. and Canada from public and private sector small, mid-size and enterprise organizations.

Among many other key findings, this year’s report revealed the following:

Increasing Complexity

Database design and management are evolving in tandem with increased complexity, and IT professionals are on the verge of a significant shift in strategy that embraces cloud database-as-a-service (DBaaS) platforms as critical database performance success factors.
- Though nearly 80% of the respondents said Microsoft SQL Server is the most critical platform in use today, there’s a relatively even split between the second, third, and fourth most-used platforms: cloud DBaaS (54%), Oracle (47%) and MySQL (43%).
- As tech pros keep pace with advances in database platforms and indicate an intended shift toward cloud DBaaS adoption (33% of the respondents ranked cloud DBaaS as the number-one priority database platform to adopt over the next three years), the new mix of traditional/cloud DBaaS platforms will also require new skills.
- Complexity increased in response to COVID-19 and the transition to remote work over the past year, according to 31% of respondents, who cited increased data demands from users, increased volume of data and security/compliance issues (such as GDPR) as the top three drivers of added complexity.
The industry’s acknowledgment of cloud DBaaS platforms as critical success factors for database performance is now underway. When it comes to cloud migration (especially amidst a pandemic dramatically accelerating planning and execution of cloud computing and digital transformation), database professionals should start with a basic data and code cleanse and ensure only what’s truly needed is being migrated to reduce complexity and overall cost (we know cloud computing costs can add up). A comprehensive, successful migration strategy should also include a detailed assessment of the data estate—what dependencies or complex coding features exist? Remember, everything interconnected with a database has to go with it to the cloud, so taking these preliminary steps will help organizations avoid surprises down the road.

Database Security/Compliance Influence Priorities

The cloud and security/compliance are key factors influencing priorities and resource allocation in 2021 and beyond.
- Nearly 60% of tech pros surveyed stated that of the data/systems currently managed within their organization, less than 25% are in the cloud. But this is expected to change, as 36% of respondents reported integrating cloud DBaaS is one of their organization’s top priorities in 2021.
- Two-fifths of the respondents anticipated bigger budget allocations in 2021 (compared to 2020) for cloud services and security/compliance (42% and 39%, respectively).
Managing database platforms is a big undertaking—a balancing act between proactive and reactive responses. But tech pros must make sure they’re covering all their bases, especially when it comes to security and compliance (and, on occasion, cost and licensing). Time and time again, we see these four big-ticket items pushed to the bottom of the priority list, and falling at one of these hurdles can do a great deal of damage. The new generation of legal compliance has real teeth if you fail to comply with the law. A simple cost-benefit analysis can be the proof point you need to highlight the potential impact of not upskilling in one of these areas and can give you a reason to move it back to the top of the priority list.

Maintenance Constraints

Database professionals devote more than a third of their day to database maintenance. This constrains both time and opportunity for activities such as innovation and upskilling in response to increased complexity and data-driven initiatives.
- Maintenance work accounts for a greater portion of database professionals’ time when compared to other tasks. One-third of the tech pros surveyed stated one of the top areas where they spend the most time as a database/data/tech pro is maintenance (33%) and database development (22%).
Daily database maintenance consumes time tech pros could otherwise be spending on things like upskilling or innovating new ways of working. For example, nearly half of those polled stated performance/query tuning was one of the top three skills/tools they believed they could develop/improve to better manage increasing database complexities, followed by security management and data engineering (45% and 41%, respectively). One of the best ways to work smarter, not harder: Mitigate downtime spent on maintenance and enable efficiency is to implement appropriate automation and monitoring tools. When implemented properly, automation and monitoring solutions can improve overall performance and eventually lead to reduced costs. Additionally, they can free up tech pros’ time, allowing them to focus on more proactive database performance management.

The report reveals just how quickly the database architecture and operations tech pros are managing in today’s organizations are changing—and the web of management complexity accompanying this change. With application availability and performance as critical business drivers, the database has long been the linchpin of today’s IT environments. However, against the headwinds of increasingly diversified platforms and reduced resources, tech pros must be equipped to do more than reactive maintenance in environments that keep adding more databases with each passing day.

Tech pros must capitalize on the opportunity ahead to maximize the performance potential of cloud-hosted databases and ensure the successful optimization of their organizations’ data.
August 9, 2021
A New DevOps Test Environment Platform is Needed
Test environment management is bottlenecking DevOps transformations and is driving the need for a new test environment management platform solution. On-demand test environments in which developers and testers do not have to wait for a shared environment are sorely needed.

The evidence is clear. In my book Engineering DevOps I indicated that finding and setting up test environments is a primary source of bottlenecks in DevOps value streams.

In my February 2021 webinar with Plutora, I presented Best Practices for Developing Test Management Environments. I indicated that orchestrating and managing test environments is often a primary source of delays and waste within value streams.

The State of Testing Report 2020 by PractiTest indicates that DevOps practices increased 57%; from 26% in 2017 to 41% in 2020. The report revealed that 46% of testing has shifted to using CI/CD processes, 55% are actively involved in shifting testing left and 36% are testing in production. This is clear evidence that DevOps testing is on the rise.

Wolfgang Platz’s December 2019 article Why Software Testing Remains a Bottleneck indicates that testing is responsible for more delays than any other part of the development process – accounting for 52% of the bottlenecks in the development process. And 53% of respondents to the World Quality Report 2020-21 indicated that test environments are not available when needed, 67% lack the capability to book and manage test environments and 71% lack visibility of available test environments.

An article by Velocity echoes the problems with pre-production environments. “With the rise of microservices and cloud-native applications, environments are becoming more complex. Often you need to wait or compete to get access to develop on a specific microservice, or can spend time waiting to deploy to a shared staging environment only to realize something already changed, and you’re back to square one.”

The Continuous Testing Report – 2020 by Capgemini defines test environment management (TEM) as managing infrastructures, test data and applications to support testing on demand. These environments are one of the greatest impediments to continuous testing. In this report, 36% of respondents said they are spending more than half their time building and managing test environments; 44% of respondents indicate they spend weeks of their time searching, managing and generating test data.

Test environments should be regarded as functions to be spun up or destroyed at will, with virtualization, infrastructure-as-code (IaC) practices, service virtualization, containerization and improved test data management.

Given all the above evidence, it is clear that a new approach to DevOps test environment management is needed.

The following are examples of benefits that DevOps test environment management platforms need to offer:
- Compatible with current tools:
  - Automatically map and configure application infrastructure to develop and test multiple services in an environment that runs just like it would in production. This needs to be fully agnostic to cloud and tech stacks, such as Kubernetes and other environments.
  - The platform needs to work on applications in parallel without disrupting others, with full access to data and third-party dependencies.
- Automatic configuration and maintenance:
  - The platform needs to automatically configure and update any changes within the production infrastructure’s mapping in real-time so ephemeral environments will always be synced with what is in production. Instead of having to spend time making sure any changes are configured, the platform will ensure the right environment that replicates production, without having to wait.
- Data and third-party dependencies:
  - The platform needs to load the relevant data and third-party resources that are vital parts of the environment to test, work and debug against, so test environments will, as closely as possible, mimic working in production.
- Team collaboration:
  - The platform needs the capability to share ephemeral test environments with other stakeholders so they get visibility and can collaborate for faster feedback cycles. This is also important for collaborating for debugging.
What This Means

DevOps testing is on the rise, and test environments are a major bottleneck for DevOps testing. A new DevOps test environment management platform is needed to reduce the bottlenecks. On-demand test environments in which developers and testers do not have to wait for a shared test environment are needed. Such a solution would improve agility and reduce wasteful orchestration bottlenecks in test, improve quality and risk management through comprehensive and automated testing and enable continuous improvement by providing more intelligent insights and feedback.
August 9, 2021
5 Steps to a Seamless Multi-Cloud Migration

With digitization the topic of discussion in almost every boardroom since the onset of COVID-19, it’s no coincidence that migration to the cloud has skyrocketed in tandem. In fact, at the onset of the pandemic, 87% of IT decision-makers noted that the remote restructuring of the workplace would push their organizations to accelerate cloud migrations.

That trend shows no signs of slowing down. In fact, Gartner predicts that cloud spending will increase by 23% in 2021 to reach $332B. Driving this historic shift is the adoption of multi-cloud infrastructures. Businesses are increasingly separating and distributing their data flows and critical functions across multiple clouds from different service providers. According to Flexera’s 2021 State of the Cloud Report, an overwhelming 92% of enterprises say they now have a multi-cloud strategy, and 82% say they have a hybrid cloud strategy.

There are many reasons for this increase in adoption. Developers that use containers, for example, can test their applications across multiple systems in the multi-cloud environment to speed up innovation, due to containers’ agility and portability. This use of multiple clouds also allows businesses to tackle cloud migrations in a more measured and deliberate manner rather than a ‘big bang’ approach.

What this also means, however, is that it multiplies IT’s stress as they deliberate the benefits, costs and risks of moving their business to a multi-cloud strategy. For example, they must consider whether or not to keep their data on-premises, migrate it to a public cloud or some combination of both. They must also ensure their application’s ironclad security, which has become a mission-critical consideration during the unrelenting cybersecurity emergency that is the pandemic. Meanwhile, they may worry that such a large-scale cloud migration will result in lost revenue and increased downtime.

With these key considerations and challenges in mind, here are five tips to help enterprises seamlessly migrate their resources to a multi-cloud environment.

Assess Your Application’s Ripeness

Before any application is migrated, it’s necessary to do a complete audit of current applications to assess their readiness for the migration. Businesses should look at the application’s dependencies and the ecosystem it lives in. This ecosystem could be its physical and virtual network infrastructure, including configurations, policies, load balancers, security and compliance requirements.

Plan and Prepare

This is a particularly challenging phase, as IT teams often linger here; wasting valuable hours and productivity. Therefore, the entire organization must outline what their migration journey will look like, discuss which departments will be affected during the process, and delegate responsibilities to facilitate a smooth transition to the cloud. The IT operations team, the development and quality assurance teams, key stakeholders and leadership should all be involved in these discussions—and should share the goal of ensuring a predictable migration process.

Carefully Weigh Your Cloud Options

Choosing the right cloud depends entirely on the unique needs, challenges and opportunities for the organization. While there are a number of different providers available to every enterprise, the best option will be influenced by financial constraints, goals for cloud adoption and even the systems already in place.

Of course, when a business is taking a multi-cloud approach, these decisions become that much more difficult—especially in light of the fact that the average cloud company uses 2.6 public clouds and 2.7 private clouds, according to the Flexera report. Connectivity between these providers is crucial, with 88% of businesses enabling their multi-cloud strategy with a private connection.

With this in mind, it’s also absolutely vital that decision-makers consider the security risks associated with each provider, as well. The number of cyberattacks on enterprises’ cloud infrastructures has increased significantly since the onset of COVID-19. With the stakes so high, a migration partner can help businesses assess which cloud providers are the right fit for them; accounting for their cost-effectiveness as well as which options will pose the least friction for the migration.

Construct Your Migration Checklist

The migration of an enterprise’s workloads is the most critical step of its cloud journey, and every company’s objective is to make a move swiftly and seamlessly. Creating a checklist can help businesses achieve this goal, as it helps them to prioritize the most critical components of their journey and understand the repercussions of a stagnant or inefficient migration.

For example, every business should aim to achieve minimum downtime during their migration process and little to no disruption of service. Both the loss of network access and system downtime can cause immediate pain in the form of lost productivity and opportunities and even more harmful damage to a company’s brand. In fact, according to Gartner, the average cost of IT downtime is $5,600 per minute, meaning a well-practiced and efficient migration strategy has a significant impact on a company’s financial health.

Since each migration is unique to the business and, often, unforeseen challenges arise that are native to the environment, it’s beneficial for organizations to have an expert migration team or partner to help them develop their checklist and ensure their migrations are streamlined and airtight.

Validate Your Cloud Journey

As the final step in a business’ migration journey, companies must validate that all their resources from their previous environment are fully migrated and operational in their target cloud environments. This step is crucial, as it marks the completion of a successful multi-cloud migration journey! This is primarily a manual process ensuring that nothing is neglected or forgotten and is often followed by an entire testing lifecycle.

Post-Migration Pointers

Cloud migrations can be complex, especially for companies relying on a years-old traditional and highly manual approach. Thankfully, in the age of automation, enterprises today are experiencing substantial time and cost savings and deploying migration strategies that move their workloads across multiple clouds in seconds instead of days.

But even when an organization’s migration journey is complete, the management of several clouds at once creates its own set of challenges and risks—especially true in today’s landscape of skyrocketing climate and cybersecurity risks. It’s therefore vital that business leaders heighten their focus on organizational resiliency and minimizing downtime through things like tightened security and effective disaster recovery strategies.

After all, with hybrid work forcing C-suites to scale their digital transformation plans even further, it’s becoming increasingly clear that the future of work is in the cloud. It’s therefore imperative that as cloud spending soars, enterprises allocate substantial resources not only to migration and management but protection, too.

August 6, 2021