devopsy.kinsta.cloud

Tag: big data

  • Observe, Inc. Dives Deeper Into Observability

    Observe, Inc. Dives Deeper Into Observability

    Fresh from picking up an additional $70 million in funding, Observe, Inc. announced this week it has added multiple tools, dubbed Data Universe Maps, to its observability platform to visualize all datasets and surface underlying event data.

    The company also added pre-built applications for Kubernetes, Amazon Web Services (AWS) and Jenkins environments with editions for Google Cloud Platform (GCP) and Microsoft Azure planned for later this year. Observe Apps are intended to make it easy for IT teams to ingest data that is surfaced via pre-built dashboards that will generate alerts when issues arise.

    Finally, Observe, Inc. has added dashboards that provide transparency into its usage-based pricing model. IT teams can now more easily determine how many credits are consumed by both users and specific datasets to help contain costs.

    Observe, Inc. CEO Jeremy Burton said those tools are crucial because in the absence of tools to control data costs, most organizations will soon discover that observability is unsustainable.

    Of course, it’s not clear yet how many organizations are moving beyond monitoring metrics to embrace observability and launch queries to discover the root cause of performance issues. Many organizations don’t have the level of expertise required to structure such queries. However, as more organizations struggle to investigate and remediate issues impacting customer experience, interest in observability platforms is rising, noted Burton.

    Burton said thus far, as a startup, Observe, Inc. has expanded its customer base by a factor of three to now include 50 organizations, which has resulted in a five-fold increase in the number of monthly active users of the platform. The platform is now ingesting more than 40TB of data each day, with more than 25 million queries each day launched across nearly one trillion rows of more than 10 petabytes of data. Capital One is included among the investors in Observe, Inc.

    At the core of the Observe platform is a graph engine that makes it simpler to uncover the relationships between machine data that is ingested into the platform.

    Going forward, Burton said Observe intends to add support for distributed tracing using the open source agent software created by the OpenTelemetry project that operates under the auspices of the Cloud Native Computing Foundation (CNCF).

    Change within enterprise IT organizations happens slowly. Many will be using legacy monitoring tools alongside observability platforms for years to come. However, there is an opportunity to rationalize monitoring tools as the capabilities of observability platforms continue to expand.

    In the meantime, IT environments will continue to become more complex as more modern applications based on microservices and constructed using containers are deployed. These applications, over time, are generally more resilient than monolithic applications; given all the dependencies between microservices, determining the root cause of a performance issue can be a major challenge. As a result, it may be just a matter of time before this new class of applications ultimately forces the observability issue in the enterprise.

  • Using Data Fabric for Real-Time Processing

    Using Data Fabric for Real-Time Processing

    Charlene O’Hanlon and Scott Gnau from InterSystems talk about silos in organizations and how data fabric enables organizations to process, transform, secure and orchestrate data from disparate sources in real-time. The video is below followed by a transcript of the conversation.

    Announcer: This is Digital Anarchist.

    O’Hanlon: Hey, everybody, welcome back to Tech Strong TV. I’m Charlene O’Hanlon, and I’m here now with Scott Gnau, who is the VP of Data Platforms at InterSystems. Scott, thank you so much for taking the time and getting on the call with me, I really appreciate it.

    Gnau: And likewise—thanks for having me. It’s great to be here.

    O’Hanlon: Alright. So, tell me a little bit about InterSystems.

    Gnau: InterSystems is a creative data technology company, we’re based in Cambridge, Massachusetts. We’ve been around for about 40 years, and we actually build data integration, data platform technology that’s used by a lot of very sophisticated applications around the world—in fact, more than half of medical records in North America go through our systems, 15 percent of global equity trades touch an InterSystems data platform or interface along the way.

    So, we’re known for solving very large, complicated, business critical problems that are centered around data.

    O’Hanlon: Well, data is the new oil, right, is that what I’ve heard? Time and time again, so many companies now rely on the data to make their business decisions and find areas of opportunity and so much more just, you know, ways to do their business better. So, data—you guys are definitely in the right space, for sure.

    Gnau: Absolutely.

    O’Hanlon: [Laughter] So, let’s talk a little bit about, then, the issue of silos when we’re talking about data and access to data, especially when we’re talking about things such as customer experience or uncovering those opportunities within the business, whether it’s improving internal processes or external applications and issues and things that do touch end users.

    So, we’ve been talking for a long time about breaking down silos when it comes to data, but do you think that that’s something that’s feasible or do you think that it’s kind of one of those wishes that will never really come true?

    Gnau: [Laughter] I’ve kind of given up on the breaking down the silos, and I’ll back into that with, you know, I’ll hit on a couple of things that you said. Because I think some of the things you said are actually really, extremely relevant, right? Data being the new oil and, you know, people have been saying that, and it’s absolutely the case. If you look at the wealth that was created in the 18th Century, it was railroads, and if you look at the wealth that was created in the 20th Century, it was energy and oil, right? Look at the wealth that’s been created in the last 20 years. It’s all about data centric kinds of companies, right?

    O’Hanlon: Right.

    Gnau: And that’s a really cool thing. And the difference really is, you know, companies can leverage that data not only to understand their business and have a rearview mirror view of what happened in the business and be able to make better decisions about it, but actually synthesize that data into building new businesses for themselves and new ways to interact with customers and so on. So, it’s absolutely critical and important.

    And sadly, I’ve been in industry, I guess, about 35 years. It feels like just five, but a long time. And throughout that career, we’ve been talking about how do we break down silos in data and silos in data processing? And, you know, silos crepe up for many different reasons. They can creep up because you’ve deployed an application that’s very specialized to go solve a problem that’s extremely important and the data that gets collected is kind of in that application and not shared easily, and that becomes, eventually, a silo in and of itself.

    Certainly, there are business model and political silos that get created, right? I’ve got a business unit model and the mortgage folks don’t wanna talk to the consumer lending folks don’t wanna talk to the deposit folks and so on. And it’s also very easy for silos to build up because of geographic differences in the business, right?

    O’Hanlon: Mm-hmm.

    Gnau: And so, this has been going on for a long time, and for a long time, we in technology have tried to solve and advise our clients, “Hey, don’t let silos control you.”

    O’Hanlon: Right.

    Gnau: I would say that the problem of silos and how they come up has actually gotten larger and is increasing at almost an unimaginable pace. Certainly, with the advent of the cloud and microservices and easy to spin up applications, more and more silos can come up very, very quickly. The fact that data and relevant data, the oil of our businesses used to be created inside the business, inside the firewall by our application, so we actually had some semblance of control over it. Now, a lot of the data that we need to use, understand, consume, and drive decisions from may live its entire life cycle outside the firewall of our business and out of our control.

    So, you think about one of the drivers of silos, and it just has now gone kind of exponentially very different.

    O’Hanlon: Mm-hmm.

    Gnau: Certainly, the pandemic has created more geographic silos, but also, the need to be able to deploy data and decision making very quickly as many of us have been remanded to our home offices and trying to manage customer relationships and run our businesses remotely in a new and a different way.

    So, I think kind of the drive for silos is going to continue to increase. And so, I think it’s also then unimaginable to think that we will live in a world without data silos. So, if that were the case, it’s kinda like, how do we come up with an approach that lets us leverage and build beyond that?

    O’Hanlon: Yeah. So, you know, it’s a very good point that you make, the fact that we’ve had so many silos, the number and the velocity of silos has increased multifold, I would imagine, since the onset of the pandemic. And with this remote and hybrid work environment that we’re now entering into, I don’t think those silos are gonna be going away.

    So, short of I don’t know what, short of whatever an organization needs to do to break down those silos, how can an organization work effectively to access that information and to get the information they need and really kind of make the most of it without being able to break down those silos? Is there a way that they can still utilize that information as effectively as they could if there was—you know, if everything was kinda linear and they could just kind of skip across the data meadow, if you will, and kinda pick the flowers as they went along? I don’t know where that analogy came from, but [Laughter].

    Gnau: Well, listen, it would be less interesting if it was equally kind of a serial thing when you go pick up those flowers.

    O’Hanlon: Right, right.

    Gnau: And certainly, we talk and work with our clients, really, about this concept that we call the data fabric and a lot of technology ins and outs and all that kinda stuff with data fabric, which we probably won’t get into a lot today, but it’s almost at a mindset. And the way I describe it, being a pilot myself, is—think about being able to have a control tower kind of impact to how you manage and integrate your silos.

    So, think about control towers in flight, right? What is the purposes of having control towers? Well, obviously, the mission is to provide safe flights. How do they do that? Well, they connect different kinds of data together and synthesize that data for decision making, right? They don’t consolidate it, they don’t fly the planes, necessarily, but they collect that data and then disseminate it in a way to kind of control and coordinate the flow.

    O’Hanlon: Right.

    Gnau: They also, it’s not a single point of contact kind of consolidation, right? There’s a control tower at every airport, right? And they connect to each other as well to manage traffic between their segments and sectors in that grid. And so, the notion of control tower is a way to think about, how do you apply that technologically speaking into thinking about your infrastructure more as a data fabric and the things that are necessary from a mindset—again, not a specific technology or a vendor, necessary—but just a mindset of how to manage through the notion of silos.

    And so, the first thing, really, is something that’s also kind of nontraditional in data processing, at least during my career, it’s all about connecting data and not consolidating data, right? And for many years, it was like, “How do I consolidate all this stuff into one place to make it more manageable to take cost out?” and all that kinda stuff. Again, in a cloud-centric world, almost unmanageable.

    O’Hanlon: Mm-hmm.

    Gnau: So, instead of thinking about consolidating it, really think about how do you connect the different data across different places? And that obviously becomes a key part of your data fabric—the connectivity, the pipes, and the widgets, and how they fit together. Connectivity isn’t enough without context.

    O’Hanlon: Mm-hmm.

    Gnau: So, data that’s created in the silo may have a certain context inside of the silo, but that context needs to be reflected externally just like the control tower needs to understand what kind of plane you’re flying, because they’ll understand the operating capacity of that plane—how fast does it go, how high can it go, and what are its maneuvering capabilities, right?

    Same kind of thing in a control tower kind of aspect to data fabric—connect the data and provide context for the data so that, as the data get connected, there’s some relevance to what is happening and something could actually be derived from it.

    O’Hanlon: That makes a whole lot of sense, and to your point, having the data remain kinda siloed and just being able to kinda select the data that you’re looking for, you know, that’s most relevant to you at that time, I imagine actually, is a little bit easier than kinda sifting through an entire haystack for one little needle. So, that’s a really interesting concept.

    Is this a technology that is currently being used within the data management space, the data science space?

    Gnau: You know, there are more and more people talking about it, and I’m sure there’s some folks that maybe have different words to describe it. We’re thinking, and some of the analysts that we work with as well, are talking about the data fabric. I’d say first and foremost, it’s a mentality, but like I said, connect, don’t consolidate. Provide the context. Have the control towers communicate amongst themselves so it’s not a single control tower but multiple different control towers for the context or the application that’s required and kinda think about the network effect of being able to build that out.

    Once you have that mentality, then it comes down to a checklist of, you know, certainly a list of what is the existing technology that we have and what is the existing technology that we use, how are the data created, how are the data stored—you know, kind of just a roadmap of what ya got. And then, think about the connectivity and the context provisioning that you want to create and look at other technologies that might need to be added to the mix to make that work in an end to end state.

    So, it’s a mindset first, and then it’s a collection of some technologies to enable that, and it’s a combination of the existing stuff that you’ve got, and certainly, an application that’s been deployed in a business-critical nature, you’re not gonna yank that out for some technology decision. So, again, it’s all about that connectivity and fitting it together and piecing it together.

    The other thing that most of the clients that we’re talking with about this are very pragmatic, it’s not a—like anything ever really should be, it’s not a 10-year plan, but it’s an, okay, how do I map this sector? How do I put a control tower at this airport and this airport and how do I leverage that, right? So, how do I go connect this silo and that silo, and then let’s go get the next silo and let’s go get the other business unit and figure out how to kinda build that out over time.

    And what’s really cool is, the value accelerates as you start to collect more of these things together. And, you know, you’ve got three or four and then you add the fifth one—the value doesn’t go up 20 percent, the value actually, of what you can do with that can go up dramatically, much higher, because of being able to understand all of the customer interaction points and create new programs, or leverage that data to actually go build a new line of business that your customers weren’t expecting.

    O’Hanlon: Mm-hmm. Right, right. Interesting. That’s really, really fascinating to me. Now, does this also kinda democratize the data? Does it enable other parts of an organization to access the value from a lot of this data by using this data fabric without the need for maybe a data scientist or somebody who’s a specialist in this in an organization?

    Gnau: Yeah, absolutely, absolutely. And again, it’s a mentality and needs to be kind of thought of with a design going in, right?

    O’Hanlon: Mm-hmm.

    Gnau: So—right, in today’s world, thinking about security and privacy on one plane and then democratization of data on the other plane, they’re almost kind of at odds. So, there needs to be some forethought that goes into what is appropriate for the appropriate eyeballs in these different spaces, and then again, making sure that the technology stack that’s been built out has the capability to deliver that. But certainly, companies that—in my experience, at last, companies that have the most success in kind of refining their oil, their data, are the ones who are actually getting data anywhere that the data need to be for a decision that’s local, right?

    So, you know, people talk about real time decisions and real time AI models and all that kinda stuff. All really compelling, all really valuable, but all completely worthless if you can’t leverage that to deliver a yes to a customer when the customer asks a question, right? And so, democratization and the architectural democratization of the data is an important key component to making your data fabric really successful and really showing up with a lot of value for the business.

    O’Hanlon: And I would imagine that AI, machine learning—you know, those technologies can actually add a whole lot to making this a much more intelligent way of gathering data as well.

    Gnau: Yeah, and it’s a symbiotic relationship, right? So, you know, machine learning and AI are all the rage and all really good stuff. A lot of the math behind it is not particularly new. What’s interesting is, the math behind it dictates that the more data you add to the model, the more accurate the prediction becomes, right? And that is very different than some of the previous kind of data mining and data science stuff that went on.

    And so—okay, that means you need lots of data. How do you get lots of data? Well, you don’t if it’s in a bunch of silos that don’t talk to each other with no context and no connectivity.

    O’Hanlon: Right.

    Gnau: So, I actually think they’re kind of symbiotic in that an effective data fabric could actually enhance the capabilities of your AI programs, and your AI programs can actually enhance the value of your data fabric in return.

    O’Hanlon: That’s really, really fascinating. [Laughter] I love this topic. I think this is definitely something that more organizations should be considering, because you know, especially if you think about just the sheer volume of data that’s out there and, I mean, really, what organization today is not basically putting their entire business on the shoulders of data without having that information that they need to make those business decisions to uncover those opportunities to just basically survive and thrive in today’s environment. I really—short of having that data, I just don’t know how organizations today can do it.

    So, I think anything that’s going to make it easier for them to uncover those areas of opportunity and find those nuggets of information that’s in all of that data, I think, is gonna be a huge, huge value add for organizations.

    Gnau: Yeah. I mean, and the last piece, really, of success for a data fabric and why I think data fabric is so important—I tried to come up with something that started with a C, because I said connect and then context. But the last one really is lineage, because lineage creates trust.

    O’Hanlon: Mm-hmm.

    Gnau: So, you know, it’s interesting, the other interesting thing about some of the AI models out there is that the answers they create are actually sometimes not intuitive.

    O’Hanlon: Right.

    Gnau: And as human beings going to execute these things, we’ve gotta really trust that the models and the data are effectively connected, so that we’re gonna risk our business on these decisions, right?

    O’Hanlon: Yep, yep.

    Gnau: And so, you know, connectivity, context, and ultimately lineage or trust of the data are kind of the three key components, again, of the mentality. Nothing to do with technology, but you’ve gotta kinda build in what’s your framework, what’s your appetite in your business for these pieces and then go look at the technology landscape to start building it out.

    O’Hanlon: That’s a very intelligent way of going about that, too, so—I love that. Scott, thank you so much for having the conversation with me about data fabrics. I know we’re gonna be hearing more about it in the future. I think it’s a great mindset and a great way to approach data access and data use in an organization. So, thanks again for your time and your expertise, I really do appreciate it.

    Gnau: Thank you.

    O’Hanlon: Yeah. Alright. Alright, everybody, please stick around, we’ve got lots more Tech Strong TV coming up, so stay tuned.

    [End of Audio]

  • How Developers Can Handle the FinServ Data Deluge

    How Developers Can Handle the FinServ Data Deluge

    A decade ago, if you were to ask developers what the biggest challenges in commercial banking were, they would likely cite the difficulty of accessing and collating high-quality data on the day-to-day operations of banking institutions. Today, that’s completely changed.

    Or it has to some extent, at least. Developers working in the banking industry certainly have access to more data than ever before. But this has created challenges of its own. With so much data being collected, stored and at least partially processed, some developers have started to talk of a “data deluge”—an unstoppable torrent that overwhelms the developer’s ability to process and use it.

    In this article, we’ll try and make sense of this. We’ll also explore what it is being used for and what this means for the future of the banking industry.

    Promises and Challenges

    Before we get too deeply into the specifics, however, let’s take a moment to note that the data deluge is not entirely new and that it will be with us for quite some time to come.

    In fact, most guides to technology in the banking sector note that many of our current issues are caused by the sheer amount of data being collected. It has become acutely apparent that our data harvesting capabilities have grown far more rapidly than our ability to ensure accuracy or to label it in accordance with legally-mandated compliance frameworks.

    And, of course, this is only going to get worse. No one knows yet how the global COVID-19 pandemic and ensuing economic downturn will affect the global payments market, but it was previously predicted to reach $2 trillion by the end of 2025, with a compound annual growth rate of 7.83%. Even realizing a portion of that growth will create real challenges for developers in the banking sector.

    Using the Data Deluge

    With all that said, however, let’s also recognize that this is a ‘good problem.’ It’s not necessarily a bad thing that we have access to so much data—the issue is making sure that it’s accurate, safe and useful.

    And when it comes to doing that, three opportunities are already apparent. Here they are.

    1. Increased Velocity

    Many of the new approaches to leveraging data in the banking sector are focused on increasing speed. Indeed, the ability to access real-time intelligence in banking operations was one of the original promises of the big data revolution, albeit one that has taken some years to come to fruition.

    There are several reasons for that, but one of the most important is the fact that banks have lagged when it comes to updating their hardware and software to correlate with the amount of information they are collecting. Most banks now undertake significant levels of quantitative analysis on business and customer data, for instance, but few do so in real-time.

    Instead, many banking systems are still run on monthly data-processing schedules, meaning that it can take even the most technologically advanced bank 30 days to spot consumer or business trends. Implementing new technologies such as Apache Spark is one potential solution to this problem. Like Hadoop, it’s an open source big data analytics engine, but it’s faster, more scalable and easier to use.

    2. More Reliable Governance

    Another challenge when it comes to managing the deluge is that of governance. With so much data flowing in, it can be a challenge for developers and operations staff alike to keep track of its source and how it should be managed within data compliance frameworks.

    This is one reason, as we’ve long argued, that banking should consider DevOps. By uniting developers and operations staff, both can come to a deeper understanding of data compliance and how this must be integrated into every aspect of the contemporary banking industry.

    This is one area, however, in which more data is not necessarily a bad idea. Rather than pausing data harvesting because of concerns that it will create compliance issues, developers should aim to build systems that can carefully and reliably label this data throughout the custody chain.

    3. Focus on the Customer

    Finally, developers shouldn’t lose sight of the ultimate aim of collecting data—improving the customer experience. Data can be used to do this in two primary ways.

    The first is to put data to work in your customer-facing systems. This is the approach recently pioneered by Western Union, which offers an omnichannel approach that tailors and personalizes customer experiences by processing more than 29 transactions a second and integrating all that data into a single platform for statistical modeling and predictive analysis.

    Secondly, data can be used to inform strategic decisions. Just as consumers are now more savvy about the way that their data is used, leadership needs to understand that this type of data-based decision-making is the future of the industry.

    In other words, banks must be able to move faster to transform their data into intelligent insights and then put those insights into action to improve customer service, connect customers to information and products when and where it’s needed most and protect sensitive data and customer accounts from threats.

    The Bottom Line

    The data deluge creates challenges for developers and leaders alike. These challenges are only going to grow as our ability to harvest consumer and industry data continues to expand. If we are going to realize the promises of technology-driven banking, we need to improve the way in which data is managed and processed and ultimately improve our ability to extract meaningful insights from that data.

    And just as this challenge is not new, neither is the outcome—a better banking experience for the customer.

    If you want to learn more about financial services and FinTech development, operations, architecture and leadership through expert-led talks, panel discussions and keynotes, check out FinCon DX—now available on demand!

  • Does Your Organization Need a Data Diet?

    Does Your Organization Need a Data Diet?

    The scenario is all-too-familiar: There’s a security breach, and afterward, the affected organization asks what it must do to better protect its data.

    But what if that organization never collected and stored that sensitive information in the first place? Often, the best defense against an embarrassing and costly breach is to collect only data that is essential to an entity’s mission.

    Some who work in computer privacy circles refer to this as “going on a data diet.” They know the temptation is great for organizations to bulk up on data of all kinds. After all, storage costs are low. With the move to the cloud, an organization doesn’t even need to invest in hardware and upkeep to store the information it collects. So why not grab whatever data a customer or client is willing to provide?

    Because we live in a time when the only sensible approach to computer security is to wonder when your entity might be breached—not if. That’s why it pays to not only go on a data diet, but to adopt a regimen for keeping your organization’s databases lean and, as a result, your customer relationships healthy. 

    Too often, companies collect data just because there’s a chance it might be useful or they think it’s harmless. I’ve seen this play out firsthand. Sales departments seek any small edge they can find; who knows what small shard of data might prove useful in the pursuit of potential customers? The same is often true of an organization’s marketing unit, whose people are working hard to reach a broader audience for their product or message. If information flows freely from the potential and current customers, they collect it in pursuit of any leg up in the competition for people’s attention.

    But there are costs to an organization beyond the price tag of data storage. That includes reputational risk and exposure to lawsuits and the distractions these and other problems bring. More info might seem better—until hackers have broken into a system and the head of an organization is forced to explain why that data was gathered in the first place.

    Weaponizing Seemingly Harmless Data

    It was not that long ago that there seemed to be two distinct types of data: In one bucket there was personally identifiable information (PII) such as a name, address, phone number and email address and critical financial information like bank account and credit card numbers. Throw in personal health information (PHI) and other obviously sensitive information; everyone knew this sort of data must be zealously safeguarded.

    The second bucket was everything else: The seemingly extraneous bits we all leave behind living in a digital world such as what time you visited a website, the language you choose for a website, what products you viewed. Think of it as a mop bucket—messy if spilled, but not vital to corporate security.

    But those distinctions have blurred in recent years. An individual piece of data might seem innocuous, but not when combined with other sets of data that do not fall under the definition of PII as set out by law. What makes Big Data so promising to an organization’s sales and marketing teams is also what makes it potentially dangerous to collect: The connections that a clever algorithm and today’s awesome computing power can deliver crawling through vast data stores. If you link information that is personally sensitive (a person’s tastes or habits or activities) but not technically PII, it can be weaponized to harm individuals.

    As an example, think of the tracking data offered by people’s cell phones. If a person leaves the same street address at around 8:30 every morning and makes the same drive back and forth, it’s quickly obvious where someone lives and where they work. Given other open records, a hacker with dark motives can find our commuter’s name and other information. It’s not a big jump to connect this information to crimes like theft or even stalking and blackmail.

    Adopting a New Data Discipline

    Given the risks, forward-looking organizations are now adopting privacy programs that force their teams to ask the hard questions before collecting information. For these privacy practitioners, any time someone internally proposes that a new piece of data be grabbed or used in a new way, a conversation ensues about its usefulness and the validity of collecting it.

    Maybe the policy is conducted through an internal privacy committee that meets to debate its appropriateness. Maybe it’s a rigorous process that means checks and balances throughout an organization. But whatever the method, only data that can be justified as fundamental to an organization’s operation is collected.

    Another idea gaining popularity among those focused on privacy concerns: A process for jettisoning old data. (An example: Does an organization need to know a person’s last several addresses or just the current one where they can be reached?)  Under such a data diet, all information that an organization continues to store beyond its immediate use requires a specific justification.

    After all, the information you don’t have is the easiest to protect.

  • Why DataOps is Critical for Digital Transformation

    Why DataOps is Critical for Digital Transformation

    Data lies at the heart of all digital transformation. In fact, today, as organizations look to further accelerate their ongoing digital transformation initiatives, companies are looking to extract even more value from their data, to better drive corporate decision-making, help grow the business, improve customer satisfaction and gain a competitive advantage. 

    Inspired by DevOps, DataOps, which Gartner defines as “a collaborative data management practice focused on improving the communication, integration and automation of data flows between data managers and data consumers across an organization”, has grown in popularity in the past decade as organizations seek to accelerate the delivery of business value from their data. But taken even one step further, enhanced data operations (i.e. more than DataOps) can help organizations maximize the business value of the data they own and also the infrastructure in which the data resides to increase data quality, reliability and availability in a hybrid cloud environment. This enables organizations to further improve business resilience and digitalization through collaborative, data-centric operations in an increasingly complex, interconnected world. Particularly as hybrid and multi-cloud environments proliferate and it becomes harder for teams to understand, analyze and streamline data usage.  

    DataOps Enables Digitalization

    By 2023, IDC predicts that 52% of worldwide GDP will come from products and services delivered by digitally-transformed enterprises—and according to Gartner, 91% of enterprises are already involved in some sort of digital initiative. This means that organizations already engaged in digitization are paving the way for a new future of business—commerce, production and innovation. But digitization is only half the battle—the other half (i.e., digitalization) comes from investing in understanding those digitally-driven insights and using those learnings to fuel business growth. 

    In the face of stifling market competition and evolving customer demands, the key differentiator in market leadership today is data and operationalizing that data. Hence, the difference between the market leaders and followers boils down to this: Data operations. Data operations not only plays a vital role in empowering ongoing digital transformation initiatives (think hybrid cloud operations, DevOps optimization and application modernization; data is at the core of all of these efforts) but also serves as a foundation for faster, better-informed innovation—fueling organizational growth and propelling businesses ahead of their competition.

    Who is Driving Digitalization?

    To gauge where your organization stands on your digitalization or digital transformation roadmap, you must first identify who within your organization is driving it and who’s involved. The latter is an easy enough guess—because, essentially, it’s everyone. 

    Although digital transformation is defined as the transformation of the digital operations of an organization, it reaches far beyond IT to all major functional departments and requires a cultural change to reach its fullest potential. That includes everyone from the chief information officer (CIO) and chief data officer (CDO)—collaborating with key stakeholders to map out and define digital transformation strategies and efforts and communicating goals and objectives—to the technical and professional teams—working closely with business units to design and operationalize new architectures and technologies to advance digital transformation initiatives. At the core of all of these modernization efforts is data; the metric for business success in the digital age. As such, integrating DataOps into your digital transformation efforts from the start is mission-critical. 

    As the market becomes increasingly competitive and business leaders look for new ways to grow and expand their business, data will remain a key catalyst in achieving those lofty digital transformation goals. By gaining intelligence from their data, companies can realize the ultimate goal of data empowerment—where all employees have simplified, secure levels of access to the high-quality data they need to do their jobs—enhancing operational efficiency, improving business outcomes and fueling business transformation at scale. 

  • DataOps: How to Turn Data into Actionable Insights

    DataOps: How to Turn Data into Actionable Insights

    Enterprises‌ ‌have‌ ‌struggled‌ ‌to‌ ‌collaborate‌ ‌well ‌around‌ ‌their‌ ‌data, which can impact everything from digital transformation to advanced concepts like AI and ML.‌ ‌DataOps is not without challenges; building, managing and scaling data pipelines requires careful thought around reusability, portability across infrastructure and applications and long-term maintenance and governance. And these are just a few of the issues facing enterprises. For this reason, DataOps technology stacks need to focus on providing key capabilities include data extraction, integration, transformation and analysis.

    The Importance of DataOps

    The enterprise is undergoing a seismic shift from siloed application development and data repositories to more composable and reusable architectures. Additionally, there is a growing demand for speed and agility as well as the influx of disruptive technologies such as the cloud, IoT and AI. Organizations that want to win big must act and adapt quickly to deploy and scale new software and solutions that provide customers with a superior experience and satisfy their rapidly evolving needs. To do that, they also must be able to rapidly aggregate, integrate and analyze data sources.

    Even with this need for speed, the tools and processes for the creation and processing of data are not standardized‌ in‌ a way that promotes rapid innovation‌ and ultimately helps organizations transform. DataOps is critical to address the challenges involved in acquiring, storing and governing data. In addition, companies are dealing with increasing complexity in their IT environments. In a recent survey, more than 80% of enterprises have a hybrid cloud or multi-cloud strategy. In order to have cost-effective and secure management of increasingly large amounts of data, enterprises must adopt DataOps.

    Managing Data Growth

    DataOps has been around for about a decade, but has recently gained momentum because of the overwhelming challenges companies face today in dealing with large, complex sets of data that are being generated at increasingly faster rates. With new technologies like the internet of things (IoT), cloud computing and the power of big data now integrated into everyday use, companies are generating at least 50 times more data than they were just five years ago. And with more data comes a need for greater efficiency and higher demand for data experts.

    DataOps potential impact on businesses might lead you to think of it as a radical new methodology. But many companies have already been using similar practices to deal with some aspects of data management, particularly around data warehousing and analytics. And just like DevOps, DataOps isn’t a product, but rather a cultural shift supported by many products—many existing products. So it’s important for businesses looking to adopt DataOps to consider what they already have, such as enterprise data warehouses and ETL tools, and what they may need to acquire, replace or modernize. In the end, companies will end up with several systems to support the data pipeline.

    DataOps and Big Data Challenges

    Most enterprises have invested in significant data infrastructure to extract, load and store their data to take advantage of Big Data analytics and technologies. However, these infrastructures are often layered, hard to manage and full of legacy tools that hinder the transfer and integration of data. Additionally, organizations have invested significant resources in tools such as data warehousing and data marts. These data warehouses have been used to model data but have been typically implemented with a predefined data model. A business intelligence (BI) system, an enterprise data warehouse or any other set of tools might assist in the execution of a data transformation job.

    The technological revolution is creating unprecedented opportunities to make things happen with data. By analyzing the data that powers modern business, we are able to provide value in every dimension of our organizations. As is the case with every transformation, the benefits of building an integrated, self-service analytics environment will be realized by users rather than IT. Leveraging the power of data science, business leaders can achieve operational excellence and compete effectively against the growing waves of competitors. For the data science and analytics staff to be successful, they must be exposed to all of the necessary technology and processes to support effective use of data.

  • How to Optimize Data in Software Development

    How to Optimize Data in Software Development

    Data can drive significant business value for every company. Globally, we are collecting more of it than ever before, and it keeps growing—from 45 zettabytes in 2019 to a projected 175 zettabytes by 2025, according to IDC. However, companies are struggling to monetize all the personal, behavioral, engagement and transactional data from customers and operations and turn it into actionable intelligence to drive their digital strategies. Leveraging insights from your data is critical for success and the development of new digital solutions to better serve customers, grow sales, improve operations and more.

    Data helps organizations develop the right software for the right customer at the right time. Data-driven organizations are 23 times more likely to acquire and retain customers than their peers, and are growing at an average of more than 30% annually, according to Forrester.

    Organizations with a deep and dedicated focus on data and analytics tend to follow three common paths to successfully drive their software development initiatives.

    Discovery Matters

    It is nearly impossible to draw value, understanding, and insights from the massive amount of data businesses collect without a proper discovery system in place. Having a comprehensive process of collecting data from various databases and silos and consolidating it into a single source that can be easily and instantly evaluated is critical for any software development project.

    During client engagements, data scientists perform deep-dive research into clients’ data pools and workflows to support client roadmaps. We recently partnered with a global e-commerce company to create a recommendation system for their customers. The discovery process allowed us to verify the feasibility of recommended products using their existing data before moving ahead with the project. Using machine learning, the team was able to clean and analyze information from multiple angles. With that knowledge and confidence, the rest of the development project proceeded quickly knowing the data was in place and the project was viable.

    Align KPIs

    Software development metrics help organizations and their partners track and measure how an application is performing and how effective the development team is in its work. IT organizations rely on a range of KPIs to fully understand software engineers’ progress, as well as gauge software quality including performance, engagement and user satisfaction. While each engagement is different and requires a different set of KPIs, ensuring that the organization and development team are on the same page is crucial.

    During the e-commerce project, success was measured through conversion and user retention rates. Other KPIs that are regularly used include developer productivity and performance metrics like cycle time and team velocity, and software developer metrics like response time, usability and UX data, among others.

    Deploy Agile

    By using agile methodologies at all stages of development, developers are able to focus on business results, not just architecture and new technologies. A similar agile approach can also be applied to data management. When it comes to data science, it’s all about extracting useful information from raw, structured and unstructured data and implementing machine learning models to analyze vast amounts of information. This process requires more creativity and is a key driver for understanding potential risks and failures over the course of any given project.

    With agile, data models and analyses are broken down into manageable bite-sized chunks that can be built and tested quickly—the same way software is developed. If the analytics do not yield the expected results, data scientists see the failures immediately and can correct their course. They could ask the team to analyze different data sets, adjust the models, tweak algorithms or make an educated decision about the project’s viability.

    A multinational credit company took this approach to the next level when it used agile to develop a new, big data hybrid cloud platform. After anonymizing, aggregating and storing information in a data lake, the team quickly realized the database it was planning to use was limited to a certain number of partitions and entities. By running agile sprints and developing and testing the minimum viable product (MVP) continuously, the team was able to seamlessly switch to a new database that had more scale and functionality.

    The value of data is only going to increase with time. Data-driven organizations embrace the use of data and analytics to gain unique business insights to support their software development projects and overall business. By focusing on the data in each and every development project, companies have a tremendous opportunity to gain a competitive advantage and grow their businesses.

  • DataOps Vs. DevOps: What’s the Difference?

    DataOps Vs. DevOps: What’s the Difference?

    There is a mindboggling amount of data today; to even measure it requires using a byte measurement called a zettabyte, which is one sextillion bytes (that’s 21 zeros). Currently, because such a ridiculous amount of data exists, there is a growing urgency to end wasteful data processes. From this environment, DataOps was born. Similar to the way that enterprises adopted DevOps to formalize and streamline wasteful development practices in the past, today, many large organizations turn to DataOps to formalize modern data management practices.

    What, Exactly, is DataOps?

    Primarily, many companies are adopting these principles because they are either trying to avoid or rectify data debt, which is the amount of money required to fix data problems due to the mismanagement of data processes. Data debt is a strong motivator for revamping outdated processes and policies, particularly when decision-makers and stakeholders require metrics before implementing change. Unpaid data debt can be detrimental to a business; the longer it remains unpaid, the more it costs to maintain a data landscape.

    By implementing DataOps principles and data governance, an organization can effectively reduce its data debt and prevent it from growing any larger. Moreover, DataOps practices and software engineering can be used to detect inefficiencies, minimize knowledge loss and capitalize on missed opportunities related to data usage.

    Similarities Between DataOps and DevOps

    Much of the processes that enable DataOps were borrowed initially from the same foundations that built DevOps. Likewise, just as companies need DevOps to provide a high-quality, consistent framework for software and feature development, data enterprises also rely on these same features to realize rapid data engineering and analytics development. For organizations that already have a DevOps framework in place, leveraging DataOps is relativity straightforward. Several important DevOps concepts adopted by DataOps include:

    • Agile development
    • Focus on delivering business value
    • Continuous integration and continuous delivery (CI/CD)
    • Automated testing and code promotion
    • Reuse and automation

    The Differences

    Despite the similarities between the underpinnings of DevOps and DataOps, there are several major differences.

    The human factor: The people using DataOps and DevOps have divergent personalities and skillsets. DataOps participants may be tech-savvy, but often their knowledge is theoretical. DataOps professionals can include data engineers, data scientists and analysts who focus on creating models and visual aids. DevOps, however, was made for software developers and engineers–coding is in their DNA.

    The process: The life cycles of DataOps and DevOps do share similar interactive properties. But, the former deviates in that it consists of a data pipeline and an analytics development process, both active and intersecting. While conceptually, the pipelines of DataOps resemble the development processes of DevOps, typically, experts note that the DataOps process is more challenging.

    Orchestration: In the DevOps process, application code does not require complex orchestration. However, for DataOps, both the data pipeline and analytics development orchestration is an essential component. Although orchestration in the DataOps pipelines occurs frequently and drives data flows, there is usually no such coordination of pipelines in application development and DevOps processes.

    Testing: Again, the two pipelines of DataOps create a significant difference from DevOps; testing in DataOps occurs during both the data pipeline and the analytic development process. These tests attempt to catch anomalies, flag abnormal data values and–unlike DevOps–validate new analytics before deployment. Likewise, these tests get embedded into a data quality framework for continual monitoring.

    Test data management: In most DevOps environments, test data management hardly takes priority; with DataOps, it’s vital to accelerate analytics development so that innovation keeps pace with agile iterations.

    Tools: DevOps is the ‘father’ of DataOps, and as such, the tools needed to support the latter are still in their infancy. While testing in DevOps is primarily automated, DataOps doesn’t have the same luxury – most users modify testing automation tools or build their own from scratch.

    Exploratory environment management: Generally, data teams use more tools than software development teams. Moreover, exploratory environments in data analytics are more challenging from a tools and data perspective; data teams also naturally depart from data islands across the enterprise.

    While foundationally, the concepts of DevOps serve as a starting point for DataOps, the latter involves additional considerations to maximize efficiency when operating data and analytical products. Nevertheless, both serve their intended audiences, reducing data debt and evolving data products or shortening systems development life cycles or providing continuous delivery. For businesses looking to make internal data-related processes more efficient, they should start by examining best practices associated with DevOps.

  • Open Source Vs. Proprietary DataOps

    Open Source Vs. Proprietary DataOps

    Core DataOps concepts are making their way into data engineering teams and, from there, into the broader enterprise. Data engineers are retooling how they create data products, and much of this work revolves around creating data pipelines.

    DataOps pipelines offer the kind of observability that traditional data integration and ETL processes don’t or can’t. They allow you to continuously integrate and test new data sources and deliver data in streaming or batch contexts with higher levels of quality and reliability than traditional, siloed approaches. They can also support machine learning efforts by preparing data for training, testing and deployment.

    DataOps pipelines shorten the cycle time between a data consumer’s request and how quickly and reliably the data producer can fulfill it, all while observing service quality throughout its life cycle. The challenge teams face when building pipelines is how to construct them. The two approaches, as I see it, are do-it-yourself using open source, or through commercial product acquisition.

    Open source certainly has its vocal adherents and some visible successes, but there are trade-offs to taking the open source route.

    Building Or Buying Your Pipeline

    One advantage of open source is in its flexibility and availability. Open source licenses, excluding the SSPL, gives users incredible freedom over what they can do with the software. If you have the skill, you can compose a DataOps pipeline that can take any data, enrich it and route it to the right place. That flexibility, though, is also a downside. While you can do anything you want, you also have to do it. Open source projects like Kafka, Pulsar, Spark, Airflow and Flink don’t know anything about the data they’re handling. That’s up to the developer.

    This may not sound like a problem, but today’s data engineers are handling dozens of data types in hundreds – or even thousands – of different formats. If you add in operational data, you’re also looking at data flooding in from firewalls, containers, SNMP traps and HTTP sources. And that’s just what’s coming at you. You also need to fetch data from object stores, multiple activity hubs and other messaging sources. No open source project natively supports the variety and volumes of data required in a modern DataOps pipeline. Every new data source you add means starting from primordial components rather than higher-level abstractions, adding time and undifferentiated effort to your work.

    You’ll also need essential features like per-source backpressure, support for a range of protocols and data stores, role and permission management and so on. That’s a lot for overworked data engineering teams to build and maintain over the life of a pipeline.

    There’s no ‘easy button’ for proprietary products, either. Many commercial products only support a handful of sources and destinations and may have excessive licensing costs related to data ingest rates or per-connector licensing fees. Also, they can be challenging to scale, from an operations perspective. Seeing a cool drag-and-drop interface may look great during the demo, but it quickly becomes a cluttered mess when you’re looking at thousands of similar data processing pipelines.

    The advantages to commercial products center around the features enterprises need, like governance, management and security. They may also integrate with other components more readily than open source. Finally, having someone to call when you need support is also essential when running in production, although open source core companies also offer support.

    Take a Higher-Level View

    As you’re evaluating DataOps pipeline options, keep two things in mind. First, regardless of which path you choose, ensure your chosen technology offers observability and monitoring features. DataOps is about reacting to change faster, and you can’t react if you don’t know what’s going on. Building observable DataOps pipelines ensures you are delivering a valuable data product to the range of data consumers in your enterprise.

    Next, get past the technology. DataOps is more a people discipline than a technology practice. A key component of a successful DataOps implementation is aligning metrics and incentives across teams. If the staff running the DataOps pipeline are disconnected from the business outcomes their data products inform and influence, they’re not incentivized to run the pipeline as a mission-critical infrastructure component. It’s key to align all of your teams with shared metrics and incentives.

    DataOps Determination

    Before you decide on your DataOps pipeline implementation, there are some things you should do.

    • Determine who your customers are and what their expectations are around data delivery, enrichment, user experience and observability requirements. Are you only serving data scientists, or are you also serving infrastructure and operations, business intelligence and marketing? Knowing your customer allows you to make better decisions.
    • Test the software in a real-world environment. Everyone dislikes being told to perform a proof of concept before you buy/build, because POCs are time consuming. Remember – they’re less time-consuming than picking the wrong tool and looking foolish later.

    Engineering teams sometimes choose technologies that are in fashion rather than selecting the right tool for the job. Taking a pragmatic view of performance, manageability and engineering effort, as well as involving a wide array of stakeholders, will give you a higher chance of success with DataOps.

  • Out of the Silo: Big Data and DevOps Meet in the Middle

    Out of the Silo: Big Data and DevOps Meet in the Middle

    Given the myriad benefits they provide, it seems crazy that big data and DevOps have been implemented independent of each other

    If you have been working from home during the recent lockdowns, you probably don’t feel like you are making many new connections, but when we come to look back on this time in a few years, it might prove to have been one of the most effective networking sessions of the past decade.

    Nowhere is this more true than in the twin worlds of big data and DevOps. It seems that being forced to work from home, far from locking engineers in both disciplines deeper within their silos, has brought them out of their shells. In recent months, we’ve seen an outpouring of speculation on how DevOps and big data can work together, and even a number of impassioned manifestos on why it is time to bring DevOps ideas to big data.

    In this article, we’ll look at why this crossover is suddenly so popular, how these two disciplines can engage in a symbiotic relationship with each other, and how they are already doing so.

    The Pandemic and the Silo

    For consumers, the experience of lockdown has accelerated internet usage trends. While some of the changes that have recently swept the online world were easy to guess, the speed at which they have taken hold has been astounding. It’s therefore no exaggeration to say, as the New York Times put it recently, that the virus has changed the way we use the internet.

    These consumer trends are driving changes throughout the software development industry, forcing firms to adapt to rapid release, testing and update cycles.

    One of the biggest changes in the way that developers collect data over the past decade has been the source of the data. This year, and largely due to the COVID-19 pandemic, we passed a notable watershed: Mobile devices and smartphones now account for more than half of all internet traffic, resulting in an increased data flow that boggles the mind and creates issues of its own. This increased data load from consumer mobile devices, rather than from corporate data acquisition systems, is forcing teams to work more closely together.

    In the words of Bill Detwiler, editor in chief of TechRepublic, in a recent CBS interview: “The COVID-19 pandemic has accelerated the blending of data analytics and DevOps, meaning developers, data scientists, and product managers will need to work more closely together than ever before.” For most firms, the way in which these consumer trends are fed into the development life cycle is via analytics teams. This has meant that, during the pandemic, analytics teams were generating actionable insights at an increased rate, and this has been the driving force behind the newly intimate relationship between big data and DevOps staff.

    Waterfall vs. DevOps

    To understand the way in which big data and DevOps teams can best work together, it’s instructive to visualize the way in which developers, operational staff and data analytics agents work.

    A decade ago, most software developers were using some form of waterfall development methodology. As the name suggests, this approach was essentially linear—the specifications for software were set in advance, and after this point multiple teams worked in tandem to deliver a continuous “stream” of software updates.

    The DevOps framework changed that by encouraging communication and collaboration between operations and development staff. The idea, in short, was to ensure that software underwent continual testing—not just in terms of usability and security, but also against the requirements of the operations staff who were charged with maintaining it.

    Now, we might be facing a shift of similar scale. Many firms have now adopted big data frameworks, but these systems generally have been seen as a way of harvesting and collating market insights. This was always a waste of potential for big data because as long as it was regarded as part of marketing, it’s utility in other parts of an organization were difficult to discern.

    Not any longer, it seems. The changes caused by the pandemic—both in consumer behavior and in the way that organizations communicate internally—have given developers an appreciation of the value of the data that analytics teams hold.

    Pioneers Show the Way Forward

    It’s also worth noting that big data and DevOps teams working in tandem is not a utopian vision; rather, this way of working was already being explored within leading companies even before the pandemic.

    In retrospect, in fact, it seems crazy that these two teams ever worked independently. If you’re a transportation company, the ability to track your loads on the road and the health and safety of the cargo that they’re carrying has become mission-critical. If you’re in the armed forces and using drones on the battlefield to conduct and report reconnaissance in real-time flyovers, the ability to adapt software parameters on-the-fly can be the difference between success and failure.

    The military is, in fact, the place to look for examples of how big data and DevOps will shortly become one process. Big data in the military has long been more advanced than that in the civilian sector and is beginning to inform everything from IT predictive analytics to real-time usage analytics.

    The Future of Big Data and DevOps

    There remain, of course, significant challenges to be overcome before the promise of DevOps working with big data teams can be realized. Perhaps the most thorny of these is the fact that computing performance is crucial for big data in a way it hasn’t been for development: The latter teams may need to build capabilities to make use of the insights generated by big data teams.

    Still, communication can only be a good thing, and if the pandemic has forced some of us to emerge from our bunkers and start to work together, then at least one good thing will have come from it.