Tag: data

What is MLOps? DataOps? And Why do They Matter?

Let’s look at three distinct disciplines—DevOps, MLOps and DataOps. In 2011, Marc Andreessen famously proclaimed that software was “eating the world.” A little more than a decade later, it’s all but impossible to argue with his premonition as software has embedded itself into virtually every industry.

But now another shift is underway and AI is gobbling up software as we know it. Our emails can finish our sentences for us, connected cars are helping us stay safer on the road and predictive technology is helping minimize supply chain disruptions caused by COVID-19. In the near future, almost every piece of software we interact with will have intelligence built-in, and as AI gains acceptance and new use cases are rolled out, applications will only continue to become more intelligent. The widespread move toward ML-enabled software has the potential to be as transformative and far-reaching as what Andreessen predicted.

But innovation is rarely easy or simple—and intelligent applications are no exception. While conventional software involves one thing—code—and not much else, intelligent software relies on a complex relationship between three interconnected variables or legs of a three-legged stool.

A Three-Legged Stool: DevOps, MLOps and DataOps

Model: The one or more AI/ML models (e.g., linear regressions, rules, deep neural networks, etc.) trained to recognize patterns in data and make decisions are what makes an application “intelligent.”

Data: Most often, a model is trained on historical data and programmed to emulate it. As a result, an application’s behavior depends heavily on the data—the raw input data, labels and features, as well as the new data to which the model is applied and ground-truth for this data.

Code: Code is the language an application uses to function. When used in the context of intelligent applications, the code pillar might refer to business logic, calls to models, receiving outputs, decision making and calls to other data systems.

A Real-World Example

To see how these pillars play out in real life, consider the example of a perception system for a self-driving car. Such systems use data (provided by offline sources and real-time cameras and sensors), models and code to determine myriad ever-changing variables (for example, if there are obstacles on the road, what lane markings look like, how fast a car can safely travel, etc.) While the following illustration is greatly simplified, it outlines the basic steps needed to build and deploy such a system and indicates which pillar each step belongs to.

Regardless of application type, the legs that intelligent applications stand on are always the same: Model, data and code. And if you try to remove any one of them, the application will topple over.

With the three distinct pillars that make up intelligent applications, three distinct disciplines have emerged to keep each individual pillar functioning as efficiently as possible: DataOps, MLOps and DevOps. The rest of this post examines the details that make each discipline distinct and necessary to the model life cycle.

DevOps

Once upon a time, companies used waterfall processes to develop software. The process often moved so slowly that products didn’t realize their potential before they died or were replaced. In 2000, Agile was introduced, allowing developers to work quickly and iteratively in tight sprints where they were constantly delivering. By 2009, Flickr, for example, was the envy of the industry, routinely performing 10+ deploys per day. Companies seeking to increase their own productivity combined software development with established operations principles—and the term “DevOps” was coined.

As the familiar loop graphic below illustrates, the DevOps cycle goes from the planning stages to the creation, testing/verification, packaging, release, configuration, monitoring and then starts over again with planning. Over the past two decades, a host of unique tools have emerged to support individual steps in the DevOps life cycle—helping developers ensure quality by building in continuous integration and continuous testing from the start and helping speed up time-to-market with workflow automation. The DevOps ecosystem is robust, well-thought-out and always iteratively improving.

MLOps

MLOps helps teams move more quickly and efficiently through the model life cycle—a loop that involves six distinct steps, each of which contains unique considerations.

Since MLOps is related to DevOps, businesses sometimes try to adapt the processes used to write conventional software to execute the unfamiliar task of machine learning operationalization. But the approach doesn’t work, as the two disciplines are actually quite different.

MLOps primarily deals with models, not code—and models are quite different from code. For example, model training and testing look very different from code building and testing. (What does “test case” even mean for a model?) And monitoring model performance is completely different than monitoring traditional software. (DevOps looks at things like CPU utilization, latency and throughput, while MLOps looks at model quality, drift and data quality.)

Furthermore, the people that MLOps engineers are building for are quite different than their DevOps counterparts. As one might expect, the primary users of models are data scientists while DevOps generally serves software developers. Most often, MLOps engineers come from a background in ML and data science or software engineers who picked up ML.

As with DevOps, no single all-inclusive MLOps platform exists. Rather, different tools support different individual steps in the model life cycle and provide some amount of workflow automation to speed up processes. Different tools support specific pieces of the ML life cycle—but the ecosystem is still somewhat fragmented. Ideally, MLOps will evolve to offer more DevOps-like tools that work well together and offer more automation. For companies looking to build an MLOps platform, build a suite of supporting tools with speed, safety and automation in mind.

DataOps

We’ve seen very well-defined life cycles for the other two pillars discussed above; however, when it comes to DataOps, the life cycle is still being defined. But the overall objective of DataOps should sound familiar: DataOps involves processes and tools to ship high-quality data frequently—which requires a combination of data engineering, data quality, data security and data integration.

As with DevOps and MLOps, what the DataOps platform or DataOps engineer is doing is supporting the tools that help perform one of these activities and help build workflows. For example, a workflow may take data from a database, apply some transformation to it and then make it available for a business intelligence (BI) tool. That’s the kind of automation that is involved in DataOps.

As you might imagine, the skillset and the tools required for DataOps are quite different from the ones needed in MLOps and DevOps—so expecting a DevOps engineer to also be an expert in DataOps seems a bit unrealistic. While MLOps requires a much tighter integration with data than DevOps historically has required, MLOps is not the same thing as DataOps. More so than the other pillars that support intelligent applications, DataOps tends to be quite separate and distinct.

Conclusion

From perception systems to enterprise-scale fintech software, every intelligent application depends on three distinct disciplines: DevOps, MLOps and DataOps. And therein lies the complexity of developing intelligent products. Each of these fields is distinct, dealing with a different set of questions and objectives within the ML life cycle and requiring different kinds of people and tools. However, they are all fundamentally united by a common goal: Optimizing quality and speed of iteration of the ML life cycle. The tools, practices and organizations that will rise to the top are the ones that will enable more seamless collaboration between different ML teams, helping them move through the ML life cycle quickly and efficiently.

March 10, 2022
Apollo GraphQL to Provide More Control Over APIs

Apollo GraphQL today previewed an update to its Federation platform for tracking and managing application programming interfaces (APIs) based on the GraphQL query language to give organizations more control over how APIs are reviewed and ultimately approved.

Matt DeBergalis, founder and CTO, Apollo, said the alpha edition of the Federation 2 platform provides a unified view of distributed APIs based on GraphQL so they can be more easily governed.

Federation is based on a unified graph created by Apollo GraphQL to initially provide more visibility into how APIs based on GraphQL are employed within an organization. That capability makes it simpler to increase collaboration among development teams that typically build APIs in isolation from one another, noted DeBergalis.

The latest version of Federation also adds a shared ownership model that empowers teams to evolve unified graphs across smaller teams, he noted.

In general, DeBergalis noted that Federation reduces the chances developers are going to create redundant APIs because they can more easily discover existing ones. They can then either employ those APIs as-is or modify them to suit their requirements as they increasingly compose an application using tools such as Apollo GraphQL Studio versus building every capability from the ground up themselves.

Overall, Apollo GraphQL claims that Federation is currently being downloaded more than one million times a week, with more than 30% of Fortune 500 companies such as Netflix, PayPal, Zillow, Walmart, Peloton and Priceline already using its tools.

The company is also now making Federation and the rest of its tool portfolio available under the Elastic License 2.0 which provides the right to use, copy, distribute, make available and prepare derivative works of the software, but prevents organizations from offering a managed service, circumventing license key functionality or removing/obscuring features protected by license keys or remove or obscure any licensing, copyright or other notices.

GraphQL, originally created by Facebook, is gaining traction as an alternative to REST APIs because it gives developers more control over how data is accessed via an API using a set of query capabilities. That capability is especially appealing when constructing digital business processes that require access by specific classes to large volumes of data. In fact, DeBergalis notes that, in many cases, it’s the existence of a rich set of APIs that enables organizations to construct digital processes that bridge the various isolated processes organizations previously had in place.

Most IT teams will not be replacing REST APIs with GraphQL-based APIs overnight. However, there will soon be enough APIs based on GraphQL to have a significant impact on how applications are built and deployed. Many applications may wind up invoking external services via both REST and GraphQL-based APIs for many years to come.

In the meantime, the rate at which applications are being built should substantially increase as tools such as Federation make development teams more productive. The challenge many DevOps teams will soon face is managing a large volume of deployments and associated application updates as more applications are deployed in production environments.

November 3, 2021
A New DevOps Database Platform is Needed

The diverse number of database platforms, deployment environments and data variants needed to support the plethora of applications and services required for rapid DevOps continuous delivery lead times, coupled with the challenges of integrating database changes for application pipelines, is a growing challenge for many organizations as they strive to accelerate their DevOps transformations.

When you stand back and look at the bigger picture, an enterprise with many databases and various data requirements for DevOps continuous delivery presents a problem. At best, it makes things chaotic; at worst, it is a gross impediment to overall organization agility—which undermines the very purpose that DevOps is meant to serve.

In my book, Engineering DevOps, I explained that databases offer several challenges for CI/CD pipelines.

In the simplest case, when a database is used solely by one application or microservice, the database pipeline can be integrated with the application pipeline to ensure its changes use the same version management system and change controls to ensure the database changes are kept in sync with the application. However, many database platforms are not as responsive to changes as the applications themselves and become a bottleneck for the pipeline.

If a database is used by several applications, each with its own DevOps pipeline, then changes to the database must be coordinated with all applications that depend on it. Maintaining a separate database pipeline linked to the application pipelines through CI/CD toolchains, as a strategy, is overly complicated and can often introduce additional bottlenecks and synchronization issues.

Another challenge with database pipelines is the management of data in the database during the CI and CD stages. Data in production needs to be persistent and is ever-changing, even while the underlying database code is changing. During the CI process, sample data from production needs to be used to ensure testing is realistic, and new test data must be introduced to verify new database capabilities are working.

During the CD stages, rollbacks to prior versions of database code must avoid rolling back to versions that are no longer compatible with production data. This is a key reason why database changes must follow a disciplined approach that fully tests database changes with applications prior to full deployment. Rolling deployments, in which applications and database changes are deployed gradually in separate clusters, can minimize the blast radius in case of problems. The required coordination of separate systems adds complexity and bottlenecks.

So, why not simply integrate database changes as artifacts within the same DevOps pipeline as the application?

Gilad David Maayan’s September 2020 article DevOps Database Strategies and Challenges: Automation, Scalability, and More explained some of the specific challenges that arise with integration of traditional databases with DevOps.

Database tools were not designed for integration with DevOps tools. These tools tend to be database- and environment-specific, which can make it difficult to incorporate into pipelines designed for flexibility.

Database performance and ease of management often do not perform well in DevOps environments where multiple applications need to use the database in a short time, such as in the accelerated lead times for DevOps continuous delivery. This is especially true with larger-scale databases. If an application change must sync with a database schema, changing the pipeline can become complicated especially if there is a need to sync across multiple application pipelines to meet continuous delivery goals.

While DevOps processes are proven for continuous delivery of applications, incorporating database changes into the DevOps pipeline is hindered by large data sets, limited database automation and security concerns. As a result, database development and testing continue to rely on outdated processes, with fixed and often shared instances refreshed from backups. A new DevOps Database platform is needed to provide highly dynamic and efficient cloud-native database solutions for modern enterprise digital transformations, and that platform must include DataOps use cases for DevOps, DevSecOps, continuous testing and governance. This new kind of data repository must be designed to integrate seamlessly with DevOps CI/CD pipelines through flexible APIs and cloud-native architectures.

Given the points above, traditional database solutions and parallel data pipeline strategies are not sufficient to meet the challenges of enterprise DevOps transformations. What is needed is a new kind of DevOps database platform that allows the database, and database changes, to be as agile as any other artifact in the DevOps pipeline.

Paul Stanton, VP of product management at Windocks, in his June 2021 YouTube video Windock DevOps Data Repo, said, “Imagine terabyte class database environments managed as DevOps artifacts with entire versioned databases available in seconds.” That’s what we need.

The following are examples of the capabilities required for such a new DevOps database platform.

• Automated database provisioning
• Distributed repositories support
• Simple declarative builds
• Dynamically scalable
• Virtual hard drive images for a wide variety of DBs – Linux, SQL, Oracle and others
• Distributed versioned database environments working on standard file systems and storage systems delivery writeable data at the speed of DevOps – in other words, in seconds.
• Deploys equally to all operating systems and cloud environments, Docker containers and Kubernetes.
• Many clones per virtual machine (VM)
• Security features including access management, data masking and encryption
• Compatible with and easily integrates with CI/CD platforms
• Compatible with multi-cloud (AWS, Azure, GCS)
• Built-in data security and governance into DevOps data.

I will explain these more completely in a subsequent article.

What This Means

Digital transformations are imperative for modern enterprises and organizations if they are to remain competitive. However, the transformations of many enterprises and organizations are being seriously impeded by having too many database platforms and tools that do not integrate well and that bottleneck continuous delivery goals. A new kind of DevOps database platform, in which the database is treated as any other artifact, is needed to resolve technical problems with database integration to DevOps application pipelines. A new DevOps database platform solution is needed to accelerate time-to-value and realize efficient, safe operation for digitally transforming enterprises and organizations.

July 8, 2021

What Data Catalogs Don’t Reveal

In today’s data intensive world, data catalogs, as the name implies, are simply catalogs of information (some human-entered and some extracted from metadata in systems). The importance of data in the enterprise has grown, and analytics plays an increasingly important role in generating value. So, while it’s important to collect and store and organize all this data, data catalogs are missing a major piece of the puzzle – they fail to draw accurate relationships between data assets and how they relate to (or are calculated from) each other.

It is like having a catalog that contains a description of all the electronics components in the world, but without an understanding of how to assemble those components into a clock, a radio or even a computer. Or how to diagnose what needs replacing when the device stops working.

Data catalogs cannot answer a whole host of questions along the lines of, “What are the consequences if I make a change to this dataset?” Even if the information in the data catalog is religiously updated, if a change was, in fact, made, users impacted by that change, whether directly and indirectly, have no way of knowing that a change was made. As a result, risk of unidentified issues is high.

The table below summarizes the strengths and weaknesses of traditional data catalog products.

Issue	Data Catalog Strengths	Data Catalog Weaknesses
Data Discovery	Collect metadata information from a wide variety of systems Give users flexibility to manually update information Simplify searching for data assets across a wide variety of systems Make it easy to identify the subject matter expert (SME) related to a dataset	Rely on manually maintained documentation Do not automatically relate data sets to each other Do not understand how data assets are calculated from each other Fail to identify who uses a particular dataset (and how they use it)
Data Governance	Provide a robust framework for managing the availability, usability, integrity and security of the data	Rely on manually maintained user-defined rules for dictating data access Cannot automatically propagate access rules from one dataset to antecedent datasets
Infosec Compliance	Make it easy for users to label personally identifiable information (PII) and other infosec risk data sets	Do not automatically infer the implications of data processing on the movement of PII inside the enterprise
Data Quality	Data catalogs do not actually inspect data – so they do not assist with data quality issues	Complete failure to identify issues and communicate who those issues impact downstream from a data quality issue

Data Discovery

Data catalog solutions provide powerful frameworks for collecting metadata and facilitating documentation of datasets. The collection and aggregation of metadata is automated, and data engineers/analysts are, hopefully (but rarely), compliant in documenting their work so others can leverage it.

Unfortunately, data catalogs fail to identify how datasets and processing rely on each other. There is no way to reliably identify which users access and rely on particular datasets, both directly and indirectly.

Tasked with this objective, a data engineer can manually answer these questions by searching and analyzing the relevant data processing code. This approach uses expensive data engineering resources, takes time and results in reduced analyst and data engineering productivity.

Data Governance

When new data sets are created, the authors must tag the data to control data governance. Not only does this rely on compliance from the author, but the rules must be propagated to determine who can use the newly processed data. How should the rules be propagated? This is a complex issue. For example, the city in an address may be restricted, but aggregates across that city may not be. Defining these rules, in and of themselves, is complex — applying them is even more complex.

Reliance on data engineers and analysts to comply with data governance documentation and systems for enforcement is error-prone and expensive. Sensitive datasets are at high risk of being exposed.

Infosec Compliance

Current solutions to information security compliance rely on:

Users manually documenting if their data processing activities use (or produce) infosec-sensitive data.
Automated identification of infosec-sensitive data by inspecting the actual data and using algorithms to identify how the data needs to be secured.

Data catalogs are used to facilitate user documentation of data processing activities to comply with infosec compliance requirements. This process relies on humans manually maintaining documentation and systems (and performing actual data inspection.) These processes present significant security risks in and of themselves, because both data inspection – which itself presents a risk – and skilled manual work is required.

Data Quality

Data engineering already has processes to identify when there is an issue landing data in a table. Automated systems are available that inspect data values and make intelligent decisions (some even based on machine learning) about whether or not the data is acceptable. These systems exist outside of traditional data catalogs.

The question is, who will that problem impact? You must identify who are the immediate users of the data. You must also identify the indirect users of the poor-quality data.

The current approach to this problem is to have skilled data engineers review all data processing code and manually identify users to warn – this is both expensive and time consuming. Beyond this, the risk is that a user is not identified; what if the error impacts a report that lands on the CFO’s desk and they are not warned?

Data catalog solutions are good at solving the problem for which they were initially designed – finding datasets that are of interest to data engineers and data analysts.

The risk in DevOps is that data catalogs are relied upon for data discovery, data governance, infosec compliance and data quality issues that they fail to address in today’s complex data processing environments.

May 12, 2021

Out of the Silo: Big Data and DevOps Meet in the Middle

Given the myriad benefits they provide, it seems crazy that big data and DevOps have been implemented independent of each other

If you have been working from home during the recent lockdowns, you probably don’t feel like you are making many new connections, but when we come to look back on this time in a few years, it might prove to have been one of the most effective networking sessions of the past decade.

Nowhere is this more true than in the twin worlds of big data and DevOps. It seems that being forced to work from home, far from locking engineers in both disciplines deeper within their silos, has brought them out of their shells. In recent months, we’ve seen an outpouring of speculation on how DevOps and big data can work together, and even a number of impassioned manifestos on why it is time to bring DevOps ideas to big data.

In this article, we’ll look at why this crossover is suddenly so popular, how these two disciplines can engage in a symbiotic relationship with each other, and how they are already doing so.

The Pandemic and the Silo

For consumers, the experience of lockdown has accelerated internet usage trends. While some of the changes that have recently swept the online world were easy to guess, the speed at which they have taken hold has been astounding. It’s therefore no exaggeration to say, as the New York Times put it recently, that the virus has changed the way we use the internet.

These consumer trends are driving changes throughout the software development industry, forcing firms to adapt to rapid release, testing and update cycles.

One of the biggest changes in the way that developers collect data over the past decade has been the source of the data. This year, and largely due to the COVID-19 pandemic, we passed a notable watershed: Mobile devices and smartphones now account for more than half of all internet traffic, resulting in an increased data flow that boggles the mind and creates issues of its own. This increased data load from consumer mobile devices, rather than from corporate data acquisition systems, is forcing teams to work more closely together.

In the words of Bill Detwiler, editor in chief of TechRepublic, in a recent CBS interview: “The COVID-19 pandemic has accelerated the blending of data analytics and DevOps, meaning developers, data scientists, and product managers will need to work more closely together than ever before.” For most firms, the way in which these consumer trends are fed into the development life cycle is via analytics teams. This has meant that, during the pandemic, analytics teams were generating actionable insights at an increased rate, and this has been the driving force behind the newly intimate relationship between big data and DevOps staff.

Waterfall vs. DevOps

To understand the way in which big data and DevOps teams can best work together, it’s instructive to visualize the way in which developers, operational staff and data analytics agents work.

A decade ago, most software developers were using some form of waterfall development methodology. As the name suggests, this approach was essentially linear—the specifications for software were set in advance, and after this point multiple teams worked in tandem to deliver a continuous “stream” of software updates.

The DevOps framework changed that by encouraging communication and collaboration between operations and development staff. The idea, in short, was to ensure that software underwent continual testing—not just in terms of usability and security, but also against the requirements of the operations staff who were charged with maintaining it.

Now, we might be facing a shift of similar scale. Many firms have now adopted big data frameworks, but these systems generally have been seen as a way of harvesting and collating market insights. This was always a waste of potential for big data because as long as it was regarded as part of marketing, it’s utility in other parts of an organization were difficult to discern.

Not any longer, it seems. The changes caused by the pandemic—both in consumer behavior and in the way that organizations communicate internally—have given developers an appreciation of the value of the data that analytics teams hold.

Pioneers Show the Way Forward

It’s also worth noting that big data and DevOps teams working in tandem is not a utopian vision; rather, this way of working was already being explored within leading companies even before the pandemic.

In retrospect, in fact, it seems crazy that these two teams ever worked independently. If you’re a transportation company, the ability to track your loads on the road and the health and safety of the cargo that they’re carrying has become mission-critical. If you’re in the armed forces and using drones on the battlefield to conduct and report reconnaissance in real-time flyovers, the ability to adapt software parameters on-the-fly can be the difference between success and failure.

The military is, in fact, the place to look for examples of how big data and DevOps will shortly become one process. Big data in the military has long been more advanced than that in the civilian sector and is beginning to inform everything from IT predictive analytics to real-time usage analytics.

The Future of Big Data and DevOps

There remain, of course, significant challenges to be overcome before the promise of DevOps working with big data teams can be realized. Perhaps the most thorny of these is the fact that computing performance is crucial for big data in a way it hasn’t been for development: The latter teams may need to build capabilities to make use of the insights generated by big data teams.

Still, communication can only be a good thing, and if the pandemic has forced some of us to emerge from our bunkers and start to work together, then at least one good thing will have come from it.

December 4, 2020
To Accelerate Digital Transformation, Accelerate Data Transformation

Successful digital transformations are not possible without successful data transformation. After all, it’s data—and the ability to analyze that data—about an organization’s customers, employees, infrastructure, market dynamics, supply chain, business-technology systems and more that make digital transformation possible.

However, last year Gartner said that “information as an asset” is still in its infancy. “Data and analytics are the key accelerant of an organization’s digitization and transformation efforts. Yet today, fewer than 50% of documented corporate strategies mention data and analytics as fundamental components for delivering enterprise value,” the analyst firm wrote in its report, “Why Data and Analytics Are Key to Digital Transformation.”

Further, Gartner predicts change is coming and quickly: Gartner says that 90% of corporate strategies will unequivocally list information as a critical enterprise asset and analytics as an essential competency within two years.

Getting there and being successful with data and data analytics won’t come without significant challenges.

According to a survey released by IDG (sponsored by Matillion, a cloud data transformation provider for cloud data warehouses), the challenges are steep. The survey polled more than 200 IT, data science and data engineering professionals at North American organizations with at least 1,000 employees.

What are the challenges enterprises are facing? According to the survey results, 47% said data control issues are the biggest challenge to data analytics projects. In contrast, other top challenges indicated are lack of a scalable, reliable technology platform to process large data sets (45%), having too many manual processes (38%) and challenges cleansing and preparing data (36%). Additionally, 40% cited a lack of visibility and control of data silos as a challenge when collaborating with business users.

With that, it’s not surprising to learn that it takes on average one week to pull together and prepare data for useful analysis, and 45% of all time spent on data analytics initiatives is consumed by data preparation. About 60% said they lack the scalability and flexibility they need when preparing data for analysis.

It was almost unanimous when it came to the number of companies that sought ways to accelerate their approaches to data analysis improvement.

So how are enterprises accelerating their efforts to improve data analysis to better power their digital transformations? Respondents said data portability (57%), ease of onboarding (57%) and cost-effectiveness (52%) are the top features of an analytics platform that can help them move past current obstacles. The IT professionals surveyed favored user-friendly (50%) and easier connections to data sources (50%) as top features; however, data engineers, data scientists, data architects and others cited time-to-value (57%) and ability to provide self-service (51%) as essential capabilities.

A sizeable portion of those surveyed indicated they are already on their way, adopting the technologies and processes they need to improve their ability to analyze their data for better decision-making rapidly. According to the survey results, 38% are already using cloud data warehouses. And in the future, 43% said they expect to have all of their data in the cloud, with 57% planning to use both cloud and on-premises data strategies.

Those that master their data analytics more rapidly may enjoy an early mover advantage. “A company’s ability to compete in the emerging digital economy will require faster-paced, forward-looking decisions,” said Douglas Laney, distinguished VP analyst at Gartner. “Data and analytics leaders need to assert themselves into corporate strategic planning to ensure that data and analytics competencies are incorporated within the highest-level public-facing enterprise plans.”

November 25, 2020
DataOps: The Key for Real-Time Data Application Development

Data engineers and developers face challenges every day to help their organizations digitally transform. To do this, they must deliver real-time data applications faster, better and cheaper. With businesses in every industry now data-driven, data professionals must work more efficiently and accelerate time to market for their products. That’s where DataOps comes in.

Development of real-time data applications slows down when strategic business project teams are dependent on small, centralized engineering or IT teams for basic data operations such as exploring data or even building most data processing applications. Needlessly, engineering teams are notorious for being a bottleneck for strategic digital transformation initiatives. Rather, they can become known as strategic project enablers that provide the best tooling to their customers—instead of building it and being known as gatekeepers.

Business users, outside of IT and engineering, need access to data but lack the necessary deep technical knowledge to work with the latest data technologies. DataOps brings people closer to data, helping organizations accelerate the delivery of their strategic projects, reduce costs and minimize the need for hard-to-source engineering skills.

To accelerate digital transformation initiatives, teams and organizations should onboard new users directly onto a data platform quickly with the right tooling and resources required to get to work on their projects. DataOps empowers these business-focused users to work with data tools on their own to self-serve. A DataOps approach makes it easier for more people across an organization to work with its data, resulting in better alignment for data engineers and developers with their business and technical partners and improved data access.

Serious new security, compliance and governance risks are introduced when less-technical users increase their interaction with technology, such as onboarding to a data platform. More people now understand that personal data has tremendous value. Initiatives such as the General Data Protection Regulation (GDPR) and California Consumer Privacy Act (CCPA) have brought the need to respect data privacy to the forefront. The stakes for safeguarding data are increasingly high and regulators are unforgiving—costing some organizations several billion dollars in fines, lawsuits and loss of market cap for some data breaches.

When providing access to data, stringent management, control and real-time monitoring must be applied from a governance perspective. A surprise compliance auditor visit can result in questions such as which applications are reading credit card data, how an organization is keeping customer data safe, who has access to data and what are they doing with it. While locking down for compliance, you must help your business stay agile and avoid any slow down in data initiatives.

Build vs. Buy to Streamline Operations

In addition to adopting a DataOps approach, a key step is to onboard users to a data platform. This will optimize your data strategy and will help take the worry out of managing data projects, putting you on a path to faster execution and accelerated time to market.

When onboarding users to a data platform, you need to choose between building your own or buying. It takes time and resources to develop your own system to onboard users and clients and can take years to build your own technology in-house. If your engineers are spending all their time working with open source technology to build tooling for onboarding, they’re not focusing on better ways to use data and get solutions to market fast—which is an inefficient approach.

When your users are onboarded, the cycle begins again, as users request access to more data sets. More time at the command-line console means less time for your engineering teams to focus on higher-value work and strategic initiatives.

Get a Leg Up on Competition With a DataOps Approach

Buying a solution requires an upfront investment to support onboarding and, when combined with DataOps principles, is formidable to remain competitive. Together they enable support and manage the onboarding of clients to a data platform at scale and to maintain security and compliance. A DataOps approach fosters improved communication between data and business professionals to share data more effectively and to build a data mesh architecture.

A Swedish digital bank, Avanza, needed to ensure the continuous improvement of the customer experience of its digital services to remain the leader in the Swedish market. Doing so required giving developers direct and constant access to real-time business data in Apache Kafka.

The organization applied a DataOps approach and intelligent data masking rules to meet GDPR and protect customer confidentiality, featuring a large single data platform to address how it delivered data access and applied data governance to support more than 20 strategic projects. Using the platform, tenants who build streaming applications gained visibility into the health of their Kafka clusters and improved insight into data flows. This self-service approach means fewer support calls to data engineers, minimizing bottlenecks and freeing them up to work on more strategic initiatives.

Data is the fuel that propels insights by tying back to analytics, decision-making and business systems. The better the shared data, the more it can be applied to supporting better business decisions, better customer experiences and execution of strategic business objectives and goals.

DataOps also builds compliance and governance into your processes. It provides the visibility needed to understand who is accessing data and how to maintain governance and ensure people are acting ethically with the data they are enabled to access.

The daily challenges faced in data-driven organizations only continue to grow. A DataOps approach helps data engineers and developers meet these challenges to digitally transform and build real-time data applications that help organizations gain a competitive business edge in their respective marketplaces.

November 5, 2020
AppDynamics Expands APM Data Consumption

AppDynamics, a unit of Cisco Systems, this week extended the observability capabilities of its namesake application performance management (APM) platform to ingest data from open source tools and agentless services alongside metric collected via its agent software.

At the same time, AppDynamics has added a Cloud Data Collector that automatically captures data from public cloud services generated by tools such as Amazon CloudWatch from Amazon Web Services (AWS).

Finally, AppDynamics has added a visualization interface optimized for monitoring cloud-native applications and integration with Cisco’s Intersight Workload Optimizer tool, which organizations can employ to better optimize IT infrastructure resources usage to reduce costs across a hybrid cloud computing environment.

AppDynamics CTO Ty Amell said the need for deeper levels of observability into IT environments has become more pronounced in the wake of the COVID-19 pandemic. Organizations have launched multiple digital business transformation initiatives that depend on the ability to correlate events occurring in IT environments in real-time. To provide that capability AppDynamics has been investing in machine learning algorithms to analyze data that can now be more easily collected, he said.

Cisco, meanwhile, has been investing in various machine learning technologies to automate the management of compute, networking and storage infrastructure, which Amell said also serves to provide deeper insights into the overall IT environment when combined with the AppDynamics platform.

The shift toward a more holistic approach to observability justifies Cisco’s acquisition of AppDynamics, which provides a widely used APM platform. However, much of that shift is being made by DevOps teams, where roles tend to be a lot more conflated than in traditional IT environments. Nevertheless, Amell noted the shift will continue to accelerate as IT organizations realize they can also reduce costs by rationalizing legacy monitoring tools.

As IT environments continue to become more complex, there’s no doubt the tolerance for spending days and sometimes weeks navigating multiple tools to uncover the root cause of an issue is declining. It’s especially frustrating when the root cause of an issue turns out to be something trivial that can be fixed in a few minutes.

Of course, it will be up to each organization to strike a balance between observability and automatic remediation of an issue uncovered by an observability platform. Many IT teams prefer to review changes before they are applied.

Regardless of their level of comfort with automation, the amount of metrics, logs and traces that can be collected and analyzed has increased dramatically. As more of that data is collected, the more precise the observability platform becomes. The challenge now is finding the most efficient way to instrument an IT environment as the number of platforms running either monolithic or microservices-based applications continues to expand. In fact, at this point collecting all the relevant data is still often a bigger challenge than analyzing it.

October 29, 2020
From ‘Cloud Zombies’ to Multi-Cloud Experts
A significant risk associated with cloud migration is overspending. For years, zombie-like obedience to AWS has resulted in price lock and escalating fees at many organizations – this is a tale for fighting the cloud zombies.

Cloud costs around data egress or application bandwidth are often not transparent and thus highly unpredictable. Complicating this problem is the need to rapidly develop and scale large application ecosystems, leading to snap judgments by project engineers to buy tools. Or, cloud purchasing decisions fall to the CFO, who may not be qualified to understand technical nuances. Without cloud spending forethought, it’s easy to enter unsavory pricing commitments.

However, all of this is changing. As cloud service providers (CSPs) mature their offerings, competitors have risen to take on AWS, unlocking more consumer choice throughout the cloud landscape. In a multi-cloud world, cloud economists are leveraging lift and shift capabilities to optimize performances. Ironically, companies are also seeing cost benefits in retaining some on-premises servers.

I recently spoke with Scott Leatherman, chief transformation officer at Virtana, to get his take on preventing cloud cost escalation and how IT should streamline multi-cloud environments to stem the cloud budget.

Zombies in the Cloud

When the cloud arrived, it promised high performance at a low cost. Companies were happy to offload on-premises servers for the ease of cloud arrangements. For years, there was a “zombie-like loyalty” to cloud platforms, said Leatherman.

Not much was thought about costs rising in parallel with the astronomical growth of the web, mobile and IoT. Executives penned long-term pricing contracts without a second thought. Then eventually, reality hit. And it came with a hefty price tag.

Spooky Cloud Expenses

As cloud-enabled businesses took off, they encountered many hidden fees. To name a few:
- Egress: Fees for moving data in and out of the cloud.
- Application: Fees for moving applications to new instances.
- Workloads: Mounting costs for increasing bandwidth and workloads.
These and other add-ons could quickly raise a cloud bill from $200 to $200,0000 a month. Complicating this was the fact that many large organizations purchased multi-year cloud subscriptions locked at a specific price.

Counterintuitively, such cost-savings guarantees are “completely backward,” Leatherman said. As computing power increases exponentially per Moore’s Law, servers become cheaper to maintain.

“You’ve literally committed yourself to something that is depreciating in cost,” he said. Yet, those savings don’t trickle down to consumers and SLAs have remained pretty flat. “What’s the most frustrating thing for me is that the best way to run this business is to lock people in.”

The Zombies Become Aware

In the wake of COVID-spurred digital transformation, cloud zombies are waking up to slash the budget, finding cost-effectiveness in multi-cloud strategies. Some companies are even reaffirming ties with on-premises servers.

If performance isn’t worth the cost, it defeats the benefits of moving to the cloud entirely. So, how can we smartly optimize our environments for quality performance with leaner financial margins? Leatherman shared some steps to become cloud-smart:

1. Baseline Your Footprint

The first step toward reducing cloud costs is maintaining a baseline of all cloud dependencies and private data centers. This should cover a deep cost and performance analysis.

“You need a baseline in your data center—what can I move? what should I move?” said Leatherman. Unless you have a baseline of what your data center is doing (both private and public), you can never understand what resources should move to the cloud. Powerful apps for monitoring the public cloud could help audit architecture.

2. Optimize, Optimize, Optimize

Only after mapping topologies can you can begin to optimize performance. Since usage differs depending on the season, cloud optimization should respond to yearlong application usage trends. For example, ERP software may pull significantly more requests at the end of the quarter and e-commerce spikes during the holidays.

Organizations should ensure the elastic cloud truly scales to application needs, but only when needed. Ensure systems of orchestration and routing function properly, and diligently search for hidden fees. “The cost that sneaks up on you in the elastic cloud always comes up,” said Leatherman.

Heavy computation processes, such as machine learning over big data, will require enormous server consumption. For resource-intensive scenarios, Leatherman recommended keeping code lean. Whereas CPU relies on the laws of physics, “application developers are only limited by creativity,” he said.

Application developers should whenever possible optimize code to avoid redundancies. Doing so yields a positive environmental impact, too—eliminating operational inefficiencies can decrease cloud server usage, which reduces carbon footprint.

3. Lift and Shift

GCP, Azure, Oracle and other CSPs mature, the zombie-like loyalty to AWS is fading. “Now that there is parity across cloud vendors, things are changing,” he noted. “The flexibility in the marketplace is really dynamic.”

In a multi-cloud world, organizations can leverage a hybrid cloud and “lift and shift” toward the best cost optimizations for their specific needs. This is where maintaining a strong baseline will help continually test cost efficiencies among cloud competitors.

4. Refine Your Tool Procurement Process

Another solution involves scrutinizing your cloud procurement process and the personnel involved. As mentioned above, new cloud tooling purchasing may occur at the DevOps level or large contracts may be escalated to the CFO office.

Unfortunately, neither side is in an ideal situation to make smart financial decisions—one faces an urgency to deliver and the other can’t see the trees from the forest (and especially not the termites eating the valuable wood). Many believe a new role is required—The Other CFO—to help optimize the cloud stack.

5. Retain and Optimize Private Data Centers

By now, most enterprises are accustomed to having both a data center and a cloud center team. Leatherman argued that ultimately these two teams must work together to avoid outages and reduce the exponential costs of moving to the cloud.

From a security perspective, there are still significant advantages to maintaining on-premises private clouds and data centers. By leveraging private assets, you aid data protection efforts and have increased control. “Datacenters have value,” he said. “Unless you started there, no one will be 100% cloud.”

Although cloud migration seems inevitable, through software automation and Moore’s law there is value in on-premises. As a result, Leatherman predicts a rise in private cloud data center optimization and management services by third parties.

6. AI for Cloud Cost Awareness: DevFinOps

Finally, AI soon may be available to automatically scan for cloud cost impacts before applications are even deployed.

Most DevOps teams use container security scanning or automated compliance validation as part of their DevSecOps process. Leatherman suggests future DevOps tooling will similarly evolve to display automated notices that estimate cloud costs for your actions in real-time.

For example, as engineers are pushing code, a popup could display server utilizations and a cost impact breakdown. Or, inline code validation suggestions could find redundancy to decrease waste. Perhaps we’ll need a new abbreviation for this Developer + Finance + Operation focus—DevFinOps, anyone?

Such an AI would likely not negate the need for a cloud economist to help manage multi-cloud complexity. “We’ll still need them,” said Leatherman.

Surprises in a Multi-Cloud World

COVID–19 is accelerating deployments tagged for much shorter timeframes, forcing many more instances to public clouds. Businesses need to get performance up and meet their SLAs. To reduce expenditures, they must quickly lift and shift to the most performant and cost-efficient environments.

The gravity is apparent, yet hasty moves now could be disastrous for security and data privacy in the future. Besides rising cloud costs, there are other risks associated with the cloud that increase exponentially with multi-cloud environments. “It’s not just the financial spend, but the security and risk that goes along with it,” Leatherman said. “It becomes really scary how many people have access.”

Not only must cloud spending decisions shift left, but security must arise earlier in the development cycle as well. Compliance with regulations such as GDPR or HIPPA should also influence how teams manage data between clouds.

The Cloud is Relatively New

An IBM-endorsed study found enterprises are only 20% of the way through their cloud journey. Many enterprises are still just at the beginning of their cloud transformations, and 90% plan to expand their cloud footprint in the next year.

The cloud journey is quite long, and many have quite a way to go. Knowing this, cloud economics will only rise in importance. As I recently covered, cloud expertise is now superior to a university degree.

Current forces, like the COVID–19 crisis, will no doubt continue to “drive people to the cloud,” said Leatherman. While responding to the imperative to deliver more and higher speeds, it’s essential to be smart, leverage existing assets, and optimize cloud usage at every level possible. “It shouldn’t just be cloud-first,” he said. “It should be cloud-smart.”
October 27, 2020
Strengthen API Security With These Tips and Patterns
If you haven’t noticed, digital organizations are building more and more APIs. ProgrammableWeb tracks more than 23,000 public web APIs at the time of writing, and the API market is estimated to be worth $5.1 billion by 2023. Building with APIs increases internal interoperability, reduces development time and can extend product functionality tremendously. In short, the value of APIs is rising. However, opening up with APIs brings security caveats that, if not addressed, could result in serious breaches that negate these benefits.

I recently spoke with Roey Eliyahu of Salt Security to get his take on the top security concerns for API-enabled companies, and how organizations can safeguard their services. In short, he recommends thoroughly cataloging and understanding your API inventory, designing for legal challenges and mitigating OWASP’s top 10 API security vulnerabilities.

Challenges for API Security

APIs have been called “the next frontier in cybercrime.” Rightly so, as API breaches continue to pop up nearly every day. Take the recent API vulnerabilities at Cisco Systems, Shopify, Facebook, U.S. presidential campaign apps, and GCP as evidence. The most infamous was likely the Equifax breach—not enforcing formats on incoming API calls resulted in a massive data breach, which cost the company a $700 million lawsuit.

So, why have API breaches increased recently? Well, a few years ago, there were simply fewer APIs. And, these older services exposed less sensitive data. Nowadays, companies typically have many more APIs. Banking applications, for example, might expose hundreds of highly valuable endpoints that are very appealing to hackers.

Companies also build APIs for different audiences: A 2020 study by SmartBear found 72% of companies develop both internal and external-facing APIs. These APIs could easily hold hundreds of parameters, and with every new method added, a company’s potential attack surface grows considerably.

3 Steps To Secure APIs

To mitigate the above API security issues, Eliyahu suggested three main steps: Understand your API inventory, audit against privacy policies and data privacy regulations, and track the most pressing API vulnerabilities like a hawk.

Know Your API Inventory

“You need a great inventory of APIs to know your attack surface,” said Eliyahu. Many API providers document their APIs with OpenAPI definitions. But, these files don’t always match the production realities—they may be manually generated or leave out certain methods. “In every organization, there is up to a 40% gap between APIs that are actually deployed versus what has been inventoried,” he said. By maintaining a clearer API inventory, organizations can avoid unprotected shadow IT.

Include Legal Auditing

A change to an API could result in a change in how it collects user information. Updates involving sensitive data could break privacy policies or regulations such as GDPR. “After you have an inventory, you need to map all parameters manually,” said Eliyahu. He recommended performing a legal audit against your API inventory and establishing a repeatable testing process that plugs into a continuous delivery pipeline.

As API teams release new updates, they must consider security and legal implications earlier on in the development flow. “Now, with CI/CD, these changes have to happen very frequently,” he noted.

Watch the OWASP API Security Top 10

There are many known API security gaps. For example, there may not be proper authorization in place, keys could be exposed, unexpired tokens could be hijacked and so on. Thankfully, OWASP has compiled a resource of top API security vulnerabilities, which Eliyahu and many others in the API economy have rallied behind. “The world should move to adopt the OWASP issues,” he said. These vulnerabilities are as follows:
1. Broken Object Level Authorization
2. Broken User Authentication
3. Excessive Data Exposure
4. Lack of Resources & Rate Limiting
5. Broken Function Level Authorization
6. Mass Assignment
7. Security Misconfiguration
8. Injection
9. Improper Assets Management
10. Insufficient Logging & Monitoring
Using AI To Go Beyond ‘Whac-A-Mole’

Plugging these above holes is a good start. However, these are only defensive measures against known patterns; they aren’t helping you keep guard for nuanced attack types. Ideally, security systems should stop cyberattacks before they even get started.

Yet, Eliyahu recognizes a big gap in how APIs are monitored. Organizations “don’t have proper tools to know who is looking for vulnerabilities,” he said. “Tools are typically not that advanced. They mainly look for injections and known patterns.”

Imagine a hacker probing an API. They will likely do so by trial and error, testing undocumented endpoints, sending malformed requests and so on. “They need to probe for hours or days before they find something,” said Eliyahu. If an AI could detect these odd attempts in minutes or seconds, IT could significantly reduce risk.

Security solutions should thus leverage big data and AI to create baselines of typical behavior, then deter malicious activity the millisecond any nefarious probing begins. Eliyahu said such a security AI must consider dozens of behaviors such as, How is the API being accessed? What parameters are being used? What are the relationships between parameters? What is the flow of API calls? What type of data can be exposed?

Instead of waiting for a breach to occur, an AI-enabled API security monitoring could instantly pause behavior deemed “out of the norm” by referencing a knowledge base that’s constantly refined using machine learning. Because, if you are only blocking malicious actors without knowing how you need to improve API security posture, “you are just playing Whac-A-Mole,” said Eliyahu.

DevSecOps Requires API Security Forethought

Mark O’Neil, analysis at Gartner, predicted that by 2021, 25% of organizations with public APIs will have discontinued or rebooted their public API strategy, citing security concerns as a common breaking point. APIs are now at the core of many application developments. Yet, without the proper DevSec proposition, teams risk wasted efforts and costly breaches.

To summarize, here are our tips on API security:
- Ensure everything is well-documented: API inventories cannot be audited if undocumented. Shadow APIs increase the likelihood of an attack.
- Watch the top 10 OWASP vulnerabilities: These points are foundational to API security. Though not a cure-all, they present a starting point.
- Build a security and legal review process: Ensure APIs are never deployed without the proper security review. Create validation systems that plug into continuous development processes, so new versions are never breaking privacy policies.
- Each API is unique: Another thing to keep in mind is each API is unique. Therefore, API security monitoring also requires more context into unique business logic.
- Continuously monitor your APIs: Continuous search for anyone looking for vulnerabilities.
- Remove deprecated APIs: Remove old API versions. “You need a mechanism to identify and remove old API versions.” Otherwise, “zombie APIs will bite you!”
- Learn from the attacks you block: Don’t just play Whac-A-Mole. “You need a mechanism to learn from attacks you block.”
October 6, 2020