devopsy.kinsta.cloud

Tag: data management

  • Cisco Unveils 800G Networking Platform to Advance DataOps

    Cisco Unveils 800G Networking Platform to Advance DataOps

    At the Open Compute Project (OCP) Global Summit conference, Cisco announced it developed an 800-gigabit switch that consumes significantly less power than the previous generation of its networking equipment.

    Thomas Scheibe, vice president of product management for cloud networking for Cisco’s Nexus and ACI product line, said the throughput provided by the latest 7-nanometer iteration of Cisco One ASIC processors is primarily needed by organizations processing massive amounts of data to train artificial intelligence (AI) models either in a local data center or in the cloud. The challenge is that many of the organizations are looking to build AI models that process massive amounts of data while simultaneously reducing their IT infrastructure’s carbon footprint, he noted.

    Cisco has been able to achieve that goal by continuing to invest in proprietary ASIC processors; they are at the core of its networking portfolio to improve data operations (DataOps), said Scheibe. Many IT teams are also looking to replace legacy network routers and switches that consume more power and reduce energy costs by as much as 77%, added Scheibe. In terms of climate impact, Cisco claimed its 8111-32EH switch can now provide 25.6Tbps at 15% of the power requirement. Cisco projected that switch will save about 10,000 kg CO2e per year compared to a 12.8Tbps switch. That equates to the greenhouse gas emissions generated by an average gasoline-powered passenger vehicle driving 26,155 miles, according to Cisco.

    IT teams also have the option of deploying either Cisco’s network operating system or SONiC—Software for Open Networking in the Cloud—network operating system (NOS).

    Cisco is also providing IT organizations with the option to configure its latest routers and switches to run at either 800, 400 or 100 Gigabits with an eye toward upgrading throughput sometime in the future.

    In general, DataOps as an IT discipline is maturing as organizations realize they need to implement best practices to optimize data flows across their organization. Cisco is giving IT teams the option of employing a standard Ethernet fabric or an alternative fabric that increases throughput by predicting which packets need to be delivered to a specific location based on their attributes and the behavior of previous network traffic. As the volume of data that needs to be accessed by low-latency applications continues to expand, DataOps will become a more critical IT discipline. Historically, storage administrators tended to be responsible for data management, but as these applications continue to proliferate, a new class of DataOps engineers is emerging to optimize the flow of data across distributed computing environments.

    In the longer term, it’s not clear how much carbon dioxide emissions are factoring into IT decisions these days, but there are organizations that have begun to track it as part of an effort to lower their carbon emissions. Most cloud service providers have also committed to being carbon neutral. Cisco is betting achieving that goal will require IT infrastructure upgrades at a time when the amount of data that needs to be processed will only continue to exponentially increase.

  • Unstructured Data Management: Avoiding Insider Knowledge Gaps

    Unstructured Data Management: Avoiding Insider Knowledge Gaps

    Most IT leaders know that managing an unstructured data environment can be quite an undertaking. Before beginning an unstructured data management project, there are a lot of factors to consider including the number of storage devices, the size of a particular dataset, the number of files in the dataset and, perhaps the most intimidating, the unique environment the unstructured data inhabits.

    One-size-fits-all solutions can often inadvertently sweep important characteristics of unstructured data environments under the rug. Granted, this short-term, simple solution is easier than putting in the work to customize the best-fit solution for each organization’s data set. However, anyone undertaking an unstructured data management project should be forewarned—this methodology can create a longer-term detrimental effect known as “insider knowledge.”

    Insider knowledge is what is left when a group of people spend time setting up data management systems within an organization and then move on from their roles without passing along critical information about the data management system to those who take over. It makes things incredibly difficult for those who inherit the agglomerated system and, chances are, the organizations who currently have this problem don’t yet realize they have it.

    The Beginning of the End: Creating Insider Knowledge

    Today’s storage administrators have a lot on their plate and are perpetually spread thin. Their daily responsibilities typically include:

    • Creating shares and exports
    • Designing a file system layout
    • Assigning devices for each app or dataset
    • Hunting down unused data
    • Organizing and pairing workflows to datasets

    Often, with their busy day-to-day environment, the IT team’s decisions and actions are being made with a siloed approach. And as the project progresses, a unique language known only to the IT professionals overseeing the project is created. They may create their own tags and place files by a certain organizational system. However, most of this is not written down, meaning it is left to be deciphered like a code without an answer key. This language is the “insider knowledge.”

    Insider Knowledge and the Great Resignation

    Why is this an issue? As long as someone knows what is going on in the overall picture of the data, it’s fine, right? Well …

    Solely depending on the team of IT professionals who created the workflow means that they must stick with the company forever for their systems to make sense. After all, once the people with that knowledge leave, it is largely lost forever to the company.

    It is not a bet worth taking that the employees of a company will merely stick around to see the data management project to completion. The threat that insider knowledge brings to organizations is that only the people with an idea of how the system works can properly onboard the next hire. This is a great position to be in for the IT professionals who know everything about the system as there is a sense of job security, but it’s not great for the organization.

    For organizations, 44% of employees in the workforce are active job seekers. The Great Resignation has showcased that job seekers have the upper hand in the market today, and that good talent can be bought and poached from companies without a moment’s notice. When—not if—an organization’s IT team members move on, it is imperative to have a well-documented, organized foundation for the unstructured data management system so that the next team is able to pick up where they left off.

    Otherwise, when team members leave, they will inevitably leave knowledge gaps and questions will arise such as:

    • Are you hosting 20 TB of dead application data on your NAS?
    • Is this 20 TB of data something the storage administrator was planning on deleting, or is it important?
    • Are you going to pay to keep that data on Tier 1 disks?
    • Do you have old employee files that were archived to other storage devices but were never deleted?

    The implications of insider knowledge are important to remember before beginning to tackle a data management project. It truly is as simple as future-casting and deciding not to give a few people control over an entire enterprise’s unstructured data stores.

    Leave it to the Technology: What to Ask When Finding a Solution

    To manage unstructured data, there are now many tools on the market built with insider knowledge (and the pre-existing codes within organizations) in mind. By automating the process as much as possible, organizations can avoid a headache when their IT team moves on. Some questions to ask to aid in finding the proper tool include:

    • Location of data: How many storage devices are there? How many data centers are there? And how many countries are the data centers spread across? Do any live in the cloud? 
    • Scanning capabilities: Do the aforementioned locations of the data affect the tool’s ability to scan it? What functions can be performed, from just scanning to analyzing with big data abilities?
    • Presentation: How do you receive the information from the tool? Is it digestible and usable? Will it easily highlight the forgotten data sets? 
    • Aftermath: When you have the information at your disposal and have the insider knowledge in front of you, how can it help you move forward to promote a healthier unstructured data environment? 
    • Time: How long will it take? How much data can the engine scan? And will this affect business-as-usual?

    These are inquiries IT leaders must find answers to before deciding on the best solution for their organization’s long-term success. Regardless of how understaffed and overworked IT teams may be, in today’s modern data center, the last thing that should be done is give a few IT professionals complete control over an enterprise’s unstructured data management system.

    Staff turnover is never fun. In IT, it can severely impact an entire organization if IT teams are forced to start from scratch on projects because no one is left to translate the system’s language that was built on insider’s knowledge. 

    With the amount of data created each day, the time to clean up and automate the process as much as possible is now, as the time to decide on a solution that works for today was yesterday. IT leaders who take time to evaluate the new technologies and comprehensive data management solutions available will be poised to spend their budgets wisely to reduce the risk of setbacks, delays, cost overruns and even IT team turnover.

  • The Quest To Unify With GraphQL, The Modern Data API

    The Quest To Unify With GraphQL, The Modern Data API

    There is an ongoing trend in enterprise software: Using GraphQL as an aggregation layer to unify access to underlying data sources and REST APIs. Some call this the supergraph, and others refer to it as a meta layer. The idea is that by providing a more consumable layer to a wide range of internal sources, frontend developers are better equipped to access whatever data they need when constructing applications. But the quest to unify all data sources into a single schema doesn’t come without its share of trials.

    GraphQL adoption is steadily increasing, with significant large enterprises making up some of the early adopters. For example, Netflix engineers recently shared how one approach gaining traction is “GraphQL microservices (GQLMS) as a backend platform facilitating rapid application development.” As the use of GraphQL increases across a large enterprise, though, scalability and security hurdles present themselves. These include retaining proper authorization throughout multiple layers and mitigating the risks of an inherently open schema.

    I recently met with Hasura CEO Tanmai Gopal to discuss why GraphQL should be used as a middleware element to access and expose underlying datasets and APIs. According to Gopal, GraphQL is becoming the data API for modern applications—this could be a massive positive shift. However, the potential downside is around the time involved in integration. Thus, the key will be automating the integration of data sources so that developers and DevOps teams can focus on the apps and services they’re building instead of maintaining the data model itself.

    The Underlying Issue: Data Access

    “Data access is a bottleneck,” said Gopal. “It needs to be self-service.”

    On one side, there are data producers, such as databases, internal services and external SaaS services. On the other side, there are the clients that consume this data, such as mobile apps, web apps, microservices and APIs externalized for third-party developers.

    The problem, according to Gopal, is that the link between both sides is increasingly challenging to implement. It takes time to integrate with HTTP-based APIs and parse JSON payloads. The exchange protocols are dissimilar and you must solve for performance issues, caching and concurrency in data access. What’s more, the authorization context must be maintained for all data sources. These issues are exacerbated since not all developers are accustomed to the nuances of maintaining integrations.

    Frontend developers want to communicate with a centralized data store, but also need to mix and match various sources. At this point, Gopal explained, a data access layer is necessary to handle the communication between both sides. Such a layer requires some integration automation to truly get APIs to developers at the edge. And it also requires a simple, unified schema—this is where GraphQL comes in.

    Solution: A GraphQL-Based Data API

    “GraphQL is an absolutely delightful API to consume,” said Rajoshi Ghosh, co-founder of Hasura. “Using GraphQL as a data API can remove a lot of requirements of legacy knowledge of integration systems and how to securely fetch data.”

    The concept involves using GraphQL as an intermediary data API—this would consume all sources, whether they are RESTful APIs, GraphQL services, an Amazon Aurora database or a PostgresSQL-style database—and unify them in a more accessible, pleasing construct. To create a unified API over these disparate sources, a middleware solution must map API resources to logical models and methods and add authorization rules to logical models, explained Gopal.

    Although GraphQL enables developer agility, it’s the part in the middle that’s difficult to assemble. Building these connectors from scratch requires a lot of relearning and tedium to maintain, Ghosh said. And to truly cater to all data sources, organizations will likely need to craft custom plugins for legacy sources, he added.

    Solving Unified Graph Problems

    Data-driven initiatives are a common technology goal, but the breadth of data sources is rising. GraphQL can play an important role in uniting these sources and providing flexibility for frontend developers to call precisely what they need. A unified graph is thus a boon for developer experience but will be challenging to set up from scratch.

    Arguably, authorization is the biggest challenge when approaching GraphQL as an aggregation layer for multiple APIs. Any system that unifies multiple backend APIs must ensure that only the person with the correct permissions can access underlying data. A unified GraphQL layer must retain granular permissions that are endpoint-centric and unique to the specific caller.

    “Most enterprises don’t have a good authorization story,” Gopal said. Implementing authorization for data APIs is hard, Gopal said, as it’s unlike REST where the user ID can literally be specified within the URL request. These kinds of fine-grained attribute level entitlements must be recreated as a policy engine and applied to the graph API as well. For example, Hasura implements authorization by using a token that scopes the entire experience to the particular user.

    The Future of Unified API Strategies

    Looking to the future of this area, Ghosh noted that frontend stacks are becoming more robust and code in the backend is essentially becoming business logic deployed as serverless functions. And in this new reality of increasing abstractions, the role of modeling experts will likely become more important.

    Furthermore, GraphQL must go beyond read-only capabilities to embrace another trend—real-time. Although most GraphQL APIs retrieve persistent data, GraphQL-based data brokers must also evolve to implement streaming, said Gopal, as real-time data and event-based subscriptions are becoming more common.

    Today, you probably don’t need to unify all your sources into a unified data model all at once. This activity will likely be relegated to particular business domains or products to help them accelerate and demonstrate a proof-of-concept. But in the coming years, there will be mounting pressure to adopt a unified approach, Gopal said. “In a few years, if you don’t have a unified API strategy, you’ll have a massive problem.”

  • How Stream Processing Goes Beyond Real-Time

    How Stream Processing Goes Beyond Real-Time

    How fast is real-time? According to Robert Miller’s 1968 seminal study on human-computer interaction, we perceive a response time of 100 milliseconds or less as instantaneous. This is much quicker than the average human reaction time, which one online test clocks at 273 milliseconds.

    Nowadays, customer-facing applications need to respond in as close to real-time as possible to win our attention and meet standard requirements. For example, 100 milliseconds is a golden rule for credit card processors to meet SLAs. Especially in our new remote-first economy, other areas like media, health care, retail and manufacturing need to take full advantage of real-time data processing to enhance digital business and stay relevant. Yet, leveling up with real-time requires some technical nuances that might be challenging to implement with the current developer skills gap.

    According to Manish Devgan, chief product officer at Hazelcast, the problem with most data management techniques is that data isn’t being processed in-stream at the edge. Instead, data gets stored and thrown into a data lake where it accumulates, unfulfilled. To make matters more complicated, decentralized data management means organizations may be overseeing many different data lakes, adding to the potential for unused data and increased latency.

    I recently met with Devgan to learn how in-memory stream processing goes beyond real-time data. According to Devgan, the secret sauce is processing incoming data in-stream while tying it to persistent storage to create context. By doing so, companies could unlock more innovative real-time experiences, he said.

    Use Cases For Real-Time

    Real-time presents many opportunities for business operations and end-user customer engagements. This is especially true in the financial sector. For example, Devgan explains how one bank was able to use real-time data at the moment of customer interaction at their ATMs. The system was able to fetch a history of the user account and extend an offer for a low line of credit, increasing their total loans by 400%. Capturing value at the moment of user intent is a golden opportunity, said Devgan.

    Capital One is another example of a financial institution leveraging advanced technology to improve the product experience. “They’re one of the leading banks in terms of technology,” said Devgan. Capital One’s threat detection system calls a machine learning model in real-time to detect and prevent fraud. Only by acting on incoming data in the moment of interaction can preventative responses like these be signaled.

    Other scenarios are also beginning to incorporate increased real-time analysis. “This split-second preemptive decision-making is important for many other use cases,” said Devgan. Physical retail might loop into customer loyalty programs at point-of-sale systems to increase in-store spending. Or, automobile manufacturers may want to use real-time processing to detect a problem with their robotic arms sooner rather than later.

    Real-time vs. Stream Processing

    When an event comes in, it usually has very little information. But when you look up a customer profile and connect it to a greater context, you can direct further actions. “Correlating data in motion with data at rest is where the magic happens,” Devgan. This distinction uncovers the difference between real-time data and stream processing.

    The real-time label tends to get thrown around a lot, and it’s quantified differently for different groups. Real-time might be five minutes for some applications but 10 milliseconds for others. Real-time could simply mean that a system accesses data at rest very quickly, delivering perceivably “real-time” results.

    On the other hand, stream processing incorporates data at rest and data in motion. It computes incoming, in-stream data and correlates it with data at rest. “You don’t let the data land — you process it as it comes in,” Devgan said.

    For years, enterprises have been collecting and storing exorbitant amounts of data. A survey by Seagate and IDC reveals that 43% of data collected is largely unleveraged and other studies estimate the amount of this “dark data” is over 50%. And, not acting upon data effectively makes it just another pile on top of the growing mountain of technical debt.

    There is a significant storage expense for holding on to too much data, whether on-premises or in the cloud. But, the paradigm is shifting a bit, said Devgan. Organizations don’t want to hold onto everything. The attitude is beginning to shift toward “garbage in, garbage out.”

    Technology Behind Real-Time Processing

    So, what sort of technology does stream processing require? Processing data at the edge requires some fundamental changes to design thinking. For one, since data must be computed in-stream, there’s more emphasis on in-memory processing. Devgan explained this as “pushing compute around instead of pushing data around.” To enable this, Devgan pointed to open-source projects like Jet that provide powerful distributed batch and stream processing.

    SQL has a storied history of acting upon data at rest. Now, Devgan said, he noticed the reemergence of SQL in a stream processing context. Though the industry is still defining how the SQL dialect will conform in a streaming world, he’s bullish on the prospect of SQL guiding this area. “Streaming SQL will be the lingua franca for streaming,” he insisted. We’re also in an early era of determining how to bridge the stateful and serverless worlds, Devgan added.

    But regardless of computing location, it will come down to the net customer experience outcomes. “People want to build their business applications and are less worried about the operational management of the system,” said Devgan. Whether it’s building real-time offers for e-commerce or implementing fraud detection, it will come down to the numbers. There is also a role for stream processing in health care to optimize physical resources and enhance patient experiences with real-time data. In all these scenarios, rapid access to data and processing at the edge will determine much of their success.

  • DataOps Vs. DevOps: What’s the Difference?

    DataOps Vs. DevOps: What’s the Difference?

    There is a mindboggling amount of data today; to even measure it requires using a byte measurement called a zettabyte, which is one sextillion bytes (that’s 21 zeros). Currently, because such a ridiculous amount of data exists, there is a growing urgency to end wasteful data processes. From this environment, DataOps was born. Similar to the way that enterprises adopted DevOps to formalize and streamline wasteful development practices in the past, today, many large organizations turn to DataOps to formalize modern data management practices.

    What, Exactly, is DataOps?

    Primarily, many companies are adopting these principles because they are either trying to avoid or rectify data debt, which is the amount of money required to fix data problems due to the mismanagement of data processes. Data debt is a strong motivator for revamping outdated processes and policies, particularly when decision-makers and stakeholders require metrics before implementing change. Unpaid data debt can be detrimental to a business; the longer it remains unpaid, the more it costs to maintain a data landscape.

    By implementing DataOps principles and data governance, an organization can effectively reduce its data debt and prevent it from growing any larger. Moreover, DataOps practices and software engineering can be used to detect inefficiencies, minimize knowledge loss and capitalize on missed opportunities related to data usage.

    Similarities Between DataOps and DevOps

    Much of the processes that enable DataOps were borrowed initially from the same foundations that built DevOps. Likewise, just as companies need DevOps to provide a high-quality, consistent framework for software and feature development, data enterprises also rely on these same features to realize rapid data engineering and analytics development. For organizations that already have a DevOps framework in place, leveraging DataOps is relativity straightforward. Several important DevOps concepts adopted by DataOps include:

    • Agile development
    • Focus on delivering business value
    • Continuous integration and continuous delivery (CI/CD)
    • Automated testing and code promotion
    • Reuse and automation

    The Differences

    Despite the similarities between the underpinnings of DevOps and DataOps, there are several major differences.

    The human factor: The people using DataOps and DevOps have divergent personalities and skillsets. DataOps participants may be tech-savvy, but often their knowledge is theoretical. DataOps professionals can include data engineers, data scientists and analysts who focus on creating models and visual aids. DevOps, however, was made for software developers and engineers–coding is in their DNA.

    The process: The life cycles of DataOps and DevOps do share similar interactive properties. But, the former deviates in that it consists of a data pipeline and an analytics development process, both active and intersecting. While conceptually, the pipelines of DataOps resemble the development processes of DevOps, typically, experts note that the DataOps process is more challenging.

    Orchestration: In the DevOps process, application code does not require complex orchestration. However, for DataOps, both the data pipeline and analytics development orchestration is an essential component. Although orchestration in the DataOps pipelines occurs frequently and drives data flows, there is usually no such coordination of pipelines in application development and DevOps processes.

    Testing: Again, the two pipelines of DataOps create a significant difference from DevOps; testing in DataOps occurs during both the data pipeline and the analytic development process. These tests attempt to catch anomalies, flag abnormal data values and–unlike DevOps–validate new analytics before deployment. Likewise, these tests get embedded into a data quality framework for continual monitoring.

    Test data management: In most DevOps environments, test data management hardly takes priority; with DataOps, it’s vital to accelerate analytics development so that innovation keeps pace with agile iterations.

    Tools: DevOps is the ‘father’ of DataOps, and as such, the tools needed to support the latter are still in their infancy. While testing in DevOps is primarily automated, DataOps doesn’t have the same luxury – most users modify testing automation tools or build their own from scratch.

    Exploratory environment management: Generally, data teams use more tools than software development teams. Moreover, exploratory environments in data analytics are more challenging from a tools and data perspective; data teams also naturally depart from data islands across the enterprise.

    While foundationally, the concepts of DevOps serve as a starting point for DataOps, the latter involves additional considerations to maximize efficiency when operating data and analytical products. Nevertheless, both serve their intended audiences, reducing data debt and evolving data products or shortening systems development life cycles or providing continuous delivery. For businesses looking to make internal data-related processes more efficient, they should start by examining best practices associated with DevOps.

  • Managing Data Risk in 2021

    Managing Data Risk in 2021

    You can’t protect your data if you don’t know where it is stored. The first thing to consider when creating a risk-based approach to data protection is the ability to identify and prioritize data, and control who has access to it. But, how do we do this? How do we prioritize data?

    In this TechStrong TV episode, Ron Bennatan, senior vice president and general manager of data security at Imperva, joins Mitch Ashley to discuss privacy, compliance and managing data risk in data lake environments in 2021.

    The video is immediately below, followed by the transcript of the conversation. Enjoy!

    Transcript

    Mitch Ashley: I’m very happy to be joined by Ron Bennatan, who is GM of Imperva Data Security. We’ve talked before—Ron, it’s good to be speaking with you again.

    Ron Bennatan: Hey, Mitch. Nice to see you again.

    Ashley: As always, yes. Data and security—two of my favorite topics. [Laughter] Welcome to 2021.

    Bennatan: Yeah, 2020 finally is over. [Laughter]

    Ashley: Ooh! You know, it takes a pretty big rearview mirror to look at that, but I’m not sure I wanna look back too much. [Laughter] Well, we’re gonna be kinda talking about looking forward into this new year. First, would you introduce yourself, tell us a little bit about you, and also a little bit about Imperva?

    Bennatan: Yeah. So, Imperva is kind of the market lead for data and application security. We secure all paths to data. So, it kind of straddles between both covering how data is accessed and where data is accessed, kind of covering everything regardless of on prem and cloud, hybrid cloud.

    Ashley: Mm-hmm.

    Bennatan: And I am the GM for the data security side, and really glad to be here.

    Ashley: No small problem you’re solving. [Laughter] Data is everywhere, you know?

    Bennatan: One problem, yeah.

    Ashley: [Cross talk] data. At least not very many of ‘em. So, let’s talk about it. I mean, you know, we have the whole government breach in 2020 Russia—you know, Russia, Russia, Russia, that whole thing.

    Bennatan: Yeah.

    Ashley: It certainly brought a lot of attention to supply chain attacks. Of course, you know, that being an intelligence brief, it’s collecting a lot of data, right? You know, that could, of course, happen on a—it is, did happen on corporate networks, it could also be for financial gain; it doesn’t mean it has to be a national state attack. So, I imagine that’s gonna be a—supply chain is a word we’re gonna hear a lot about this year.

    Bennatan: Yeah, I think so. I think, you know, as a, just an individual, my fear, by the way, is—you know, there’s a really big project coming for the entire world, which is the delivery of the vaccines, and that itself is a supply chain. I just hope nothing messes up there, you know? Like, you know, knowing that whatever vaccine you’re getting is actually the valid one and not—you know, there’s a lot of…everything is in data and everything at the end sits in some kind of a, like, what’s the lineage of something? What’s the providence of something?

    Ashley: Mm-hmm.

    Bennatan: You know, it doesn’t necessarily—you know, we in the data governance area talk a lot about providence and proving the correctness of data, but you know, it goes all the way to the vaccine as well. So, I think this issue that we’re seeing, whether it’s breach—breaches get headlines, okay? And, you know, so this SolarWinds thing gets a big headline, the thing that these attacks on MySQL with ransomware, that gets headlines. But really, the bigger issue is getting control of our data and creating kind of a complete risk oriented approach, not just plugging a hole here and plugging a hole here and plugging a hole here. It’s—that’ll never work, right? It’s much easier to create a new hole than to plug all the holes.

    Ashley: The whack a mole strategy. “Here’s another mole to whack.” [Laughter]

    Bennatan: The whack a mole—yeah, the whack a mole doesn’t work. Whack a mole doesn’t work. And especially today when the complexity of the data environment is so much greater.

    Ashley: Mm-hmm.

    Bennatan: You know, 20 years ago—okay, 40 years ago, we just had some dashboard on the mainframe. Twenty years ago, we just had Oracle sitting on some Unix server. Today, it’s such a complex environment, data moves from place to place, you know, you look at what a data lake implies. It’s no longer a single stack, it’s a combination of stacks. Whack a mole doesn’t work.

    Ashley: Mm-hmm.

    Bennatan: So—and most companies understand that. Most companies are in this process of investing in a non-whack a mole approach, in a risk based approach that says, “Okay, I need to know where my data is. I need to prioritize which. You know, not all data is created equal. I need to scan, I need to understand, I need to know who has the ability to access something. I then need to know who is accessing something. I need to control what they’re accessing.”

    So, you know, I think there’s a better understanding that whack a mole doesn’t work.

    Ashley: Mm-hmm.

    Bennatan: I think 2021 is still gonna be a transition year. I don’t think we’re gonna necessarily get there, but we’re gonna transition.

    Ashley: We won’t have universal enlightenment [Laughter] on day—

    Bennatan: It just takes time. I mean, it takes time.

    Ashley: It does.

    Bennatan: There’s a lot of data.

    Ashley: I wanna ask you about—you did a nice job of kind of outlining, you know, identifying what data you have, prioritizing it, access to it. One of the things that I’ve always struggled with is that second step. How do you prioritize? What are the things to consider? You know, there’s obviously sorta crown jewels of my business—this is the customer data or secret sauce or that kind of things, but is there a good sort of stratification of how to prioritize data?

    Bennatan: Okay, there’s a technical answer and there’s a human answer.

    Ashley: Okay.

    Bennatan: Okay, technically, there are very good ways to do this. It takes some effort, okay? You can—you really do this based on a scoring of a bunch of dimensions, okay? So, one dimension could be the sensitivity level of the data, okay? Another dimension could be, what is the impact if something happens, okay? Which—but the reality is that we’re all people, and people are driven by incentive, and incentive is not necessarily, doesn’t always necessarily create the right prioritization. It creates whatever prioritization the incentive is structured around.

    So, I’ll give you an example. A lot of projects going on around privacy—a lot of projects. Because it’s a very, it’s a relatively new thing, less understood, and people don’t always equate the fact that we have been doing the work, almost the identical work, we just didn’t call it privacy, right? We called it classification.

    Ashley: Mm-hmm.

    Bennatan: But when you look at what privacy implies, it’s the same, but very new terminology—different semantics, different terminology. So many people don’t start from an investment they’ve already done in the last five years, they start from scratch, and the incentives are created because there’s a new role in the company. There may be a new—you know, a data privacy officer, okay, and that person has certain mandates and they drive things top down.

    And so, I think that the reality is, prioritization is driven by the projects and the funding that these projects have, and they’re also driven by breaches, okay? Because breaches are very dramatic.

    Ashley: [Cross talk] response, right? [Laughter]

    Bennatan: Dramatic, immediately created. So, I think that the reality is gonna end up somewhere kind of in the middle where things are gonna be driven bottom up and top down and they’re gonna meet somewhere.

    Ashley: Mm-hmm.

    Bennatan: But it’s not terribly complex to create a good prioritization strategy, but it’s not trivial, either.

    Ashley: Yeah, you definitely have to put some thought into it, and it’s not a textbook exercise, right? It’s all contextual.

    Bennatan: Yeah.

    Ashley: What’s important to the business.

    Bennatan: And I think you’ll find in—I mean, my prediction in 2021 is that there’s gonna be a lot of, you know, privacy is not gonna just drive a lot of these from a compliance perspective. Okay, until now, privacy is really—you know, when a company embarks on these privacy and classification projects, or at least what I see from our customers, is that it’s driven a lot by compliance, okay? They need to comply with a CCPA, they need to comply with GDPI.

    Ashley: Mm-hmm.

    Bennatan: I think we might end up having also, like, an attack flavor, okay, around privacy, which is not intuitive, not necessarily intuitive. And it has to do with the fact that, you know, part of the mandate is, you know—so, say that I am, you know, year one of my consumers and you store, I store some of your data. That means I—you have the right to demand some things from me. Like, you have the right to demand that I delete you from my system.

    Ashley: Mm-hmm.

    Bennatan: You have the right or portability, you have the right to ask me what data do I own or do I store about you? And you can come to me with these requests, okay? Different terms, different subject rights request or data access requests. You know, you can easily think about what is the effort that I, as a big company, have when you ask me to do this? It’s not a small effort. So, okay, if you—now, the nice thing is, most people don’t do that, okay? I’ve never called somebody up and told them, “Hey, delete me, okay?”

    Ashley: [Laughter] No, I think that’s probably very much an exception—maybe a rare exception.

    Bennatan: Yeah, but what happens is some people realize that this is the way to cripple a company.

    Ashley: Mm-hmm.

    Bennatan: What happens if they organize and they say, “Hey”—

    Ashley: Or protest, right?

    Bennatan: – protest, you know?

    Ashley: Moving of Facebook onto whatever.

    Bennatan: “We don’t like you, let’s organize 10,000 of our friends and we’ll all hit them at the same time.” What do you think that’s gonna do from a work perspective? It’s pretty serious.

    Ashley: Yeah, I’m guessing it’s not just press a button—that’s gonna consume some resource at the company, people resource, things like that.

    Bennatan: Yeah.

    Ashley: You know, one thing I was wondering, too, is, with this year, there’s discussion now about do we need a national cybersecurity strategy, things like that, just to mention politics, the Biden administration is talking about working with other countries to establish sort of norms of what we do when it comes to cyber whatever with each other and what is considered an attack.

    It seems like that is—maybe not this year, but that’s certainly potential for regulation of those kind of discussions, whether it comes out of any national strategy or not or could be another country that says, “Okay, now we’re gonna take this to the next level.” Like, what is our position around data? Not just privacy, but if it’s compromised, then what happens to you as a firm? I mean, it could get kind of interesting.

    Bennatan: Yeah, it could. You know, I mean—again, we’re delving into, like, opinions and politics.

    Ashley: [Laughter] Yeah, I know.

    Bennatan: I’m not a fan of—I’m not sure that something like this could go that route. But I do think that it goes other routes. So, for example, like, I’m reading a book on kind of the history of the Mossad, okay, the Israeli intelligence service.

    Ashley: Mm-hmm, mm-hmm.

    Bennatan: And you can see—and whenever you read any of these books, you see that all of the intelligence community, they have back channels, right? They agree on things.

    Ashley: Right.

    Bennatan: So, to me, if we wanna have something like that, it should be, like, the cyber czar of the U.S. and some other—you know, the big players, they have to agree. It doesn’t need to be regulated, it doesn’t need to be at the politician level, it has to kind of stay at the—

    Ashley: Kinda behind the scenes with the cyber czar.

    Bennatan: Behind the scenes with the experts at the—but, you know, there’s definitely an issue with state nations, okay? This does need to be addressed. Now, you know, it’s not necessarily that every—you know, we always think that, you know, we’re the good guys, they’re the bad guys, you know?

    Ashley: We would never do anything like what they did to us, right? [Laughter]

    Bennatan: Well, you know, there’s examples in history, okay, of things. So, I do think this is coming. I don’t believe it would happen at a politician level, it will happen at the actual subject matter experts. And by the way, there are good examples of things that have happened like that, especially in the financial services community. I mean, for years, maybe even decades, you know, FS-ISAC, they work together to create some level of standards across investment banks or banks. These things work, I just don’t think it’s political.

    Ashley: Interesting. Well, I wanted to kinda tap your thoughts on that. I know I threw the line pretty far out there in the water, [Laughter] just to kinda get your thoughts on that.

    Well, let’s reel that back in a little bit and talk—you know, there’s GDPR and CCAPE for California. A lot of people are still in this, I think in this kinda implementation stage, even though we’re supposed to have those things all covered, right? Maybe a little farther along with the GDPR, of course, but is that something, this year, you think it’s—we’re in this, okay, let’s now institutionalize some of those things in the organizations and really take care of our knitting there or do you think there’s more things on the horizon that are gonna come at data people?

    Bennatan: I think it’s gone into an implementation phase, for sure—which is good.

    Ashley: Mm-hmm.

    Bennatan: You know, there’s a lot of things, many things are not terribly clear, by the way, even though it’s in implementation. I’ll give you an example, right, this notion of being—my ability to request that you delete me, okay? But let’s take something else, which is a different type of compliance requirement, which is for you to keep logs for a certain period of time, activity logs. There are a whole bunch of other compliance requirements that require you to do that.

    Ashley: Data retention requirements, yeah.

    Bennatan: Yeah. Okay, so—now, what happens if part of the activity logs has my identity in it and I request you to delete it?

    Ashley: Mm-hmm.

    Bennatan: Should you delete it, or should you not delete it? One, in order to comply with one thing, you should delete it. In order to comply with another thing, you can’t delete it, maybe for three years. So, there’s even things like this, which are conflicting messages. And at the end of the day, it’s going to depend on some level of interpretation.

    Ashley: Mm-hmm.

    Bennatan: Which, at the implementation level, I can tell you, I can see two approaches. One is—okay, this interpretation needs to be given by legal or it’s okay for the business to give the interpretation to that. So—and you know how it is. If it goes to Legal, it takes a longer time; if the business can make the decision, then it goes much faster. And I’m seeing more and more of this moving to the business, taking ownership, and making the decisions, in which case, it goes much, much faster.

    Ashley: Well, and oftentimes, it seems that that always happens with laws and regulations, right? There’s what it is and there’s the interpretation of it, and there’s sorta those middle tier organizations, the consultants, the larger firms to sort of establish the frameworks that, “This is our interpretation of it,” so if you do that, you’re sort of in the norm with what the industry is doing. Who knows if that’s the right interpretation or not? It’s always subject to change, but that’s usually where that help comes from, at least on the big thing.

    Bennatan: Well, yeah. I mean, it starts from there and then usually what happens or what it evolves to, which makes it really go much faster, which is what’s happening now is that if—and by the way, it goes back to your question about this nations coming together or an industry coming together. The minute you have some critical mass within a certain peer group and you say, I don’t know, maybe I’m an insurance company, if my peers have taken upon themselves a certain interpretation, I usually don’t even care where they got it from.

    Ashley: Mm-hmm.

    Bennatan: Maybe they got it from these consultants, maybe they didn’t. But if three big players in my industry are doing the same thing, I’m gonna do the same thing.

    Ashley: Yeah.

    Bennatan: And that’s what’s happening now is that, you know, I know the insurance industry well and I know the banking industry well, and I can see that they’re now, “Oh, we’re gonna do like them” and then it’s much easier to follow, like, a track that seems to be proven than to try to invent it. And that’s happening and that’s why I think there’s an acceleration for a lot of these things.

    Ashley: Yeah, I agree with you. That makes a lot of sense. What are your other thoughts about 2021? What are we gonna be thinking about that we didn’t realize is gonna be on our radar for this year?

    Bennatan: I think we are going to kind of understand this notion of, you know, look at the data from every way that you can get at the data, not try to—like, historically, there have been a focus on pillars or silos, which were more and more, or in the past were driven really by tool specialization, okay? Like—okay, I’ve got the DLP and I’ve got this and I’ve got FAM and I’ve got databases, and I’ve got big data.

    And today, the lines are so blurred, especially on the cloud services, they’re even more blurred that there’s starting to be a realization that we really need to look at all paths to the data and all ways to secure the data and look at it instead of individual silos, looking at it as, “This is my data fabric.” Okay, so you’re starting to hear things like, you know, data lakes, data fabric, data services, and the data service could be structured by different implementation stacks. But you’re no longer trying to secure each stack by itself, you’re trying to secure the entire fabric.

    Ashley: Mm-hmm.

    Bennatan: And it’s really the right thing to do. It’s the right way to think about it. It is something that requires a transition, because even organizationally, many companies had one group doing this and one group doing this and one group doing this. And this notion of, you know, looking at a single overlay control layer, okay, which can drive policy down or can bring things up is, you know, it’s time for that. And I think 2021 is gonna be, part of this year is going to be starting to deliver things at that level of abstraction, and it’s also enabled by the fact that there is technology that can kind of bubble everything up into one level, right? You don’t need to look at it as 20 different tools, you need to look at it as kind of like a federated layer, and at least it’s doable, right? Ten years ago, it wasn’t doable to do any of it.

    Ashley: You know, it’s interesting. What it makes me think of is maybe some norms that we’ve had for a while of thinking about data owners, and that sort of puts the blinders on of, now I create that vertical view of, “This is that data for that purpose for that business unit for that application or process, whatever it might be.” And what you’re saying is looking at it more systemically, right? The entire collection of data, and securing it, but also uses of that data and sort additional intelligence and things you can mine from it.

    It’s interesting—so much, as we continue to move more and more into the digital economy, digital world for how we operate, all of those digital activities generate their own data, too. So, now, we’re adding that to the fabric of the data of, you know, we have a lot more usage information, we have a lot more telemetry that we never had. You know, it seems to be expanding exponentially. So, our ability to even just fathom what’s there and how it’s—what uses it might have and how it might be correlated in new ways sort of boggles my mind. [Laughter]

    Bennatan: Yeah, yeah. And, you know, I mean, it used to be this—one of the things people used to talk about when they started building big data platforms was that, you know, until recently, we just, we couldn’t use our data. We just threw 97% of our data away, and now we don’t wanna throw it away, but you’re right, it’s a beast that feeds itself.

    Ashley: Mm-hmm.

    Bennatan: But yeah, I think it’s, I think looking at the risk profile of the entire thing and not cutting into separate projects, which then have no tie to each other, right? I mean, if you want to reduce the risk, you asked about prioritization. Okay, prioritization needs to drive everything, right? Needs to drive reducing service area, needs to drive access control policies, needs to drive audit. If you start separating each one of them, not only is it not very effective, but it means you’re working 10 times harder, because you need to do this many, many, many times.

    So, this notion of a single kind of data security platform which kind of can have this internal feedback loop and use whatever effort you’re doing, it doesn’t matter if your effort is driven by compliance or security or privacy, whatever investment you make needs to build into this platform, not do itself over the next time you do anything. Otherwise, we’re just wasting our time. We’ll never get done. Because you said the data’s growing exponentially—if we’re chasing, if we’re not leveraging what we’re doing for one purpose when we wanna do the next purpose, we’re gonna lose. It’s gotta be a single platform.

    Ashley: Well, and to your point, it seems that once you sort of make that shift, right, to that platform view of data, now you can apply that to how you look at the data across all sources, all uses, all former kind of vertical siloed—you know, you may still want to look at it that way, but it’s not like, “Okay, now we’ve figured it out for compliance, but we’ve gotta go start over to do it for something else for operational process or whatever.”

    Bennatan: Exactly.

    Ashley: No, it’s there, you kinda take that—

    Bennatan: Exactly.

    Ashley: – those same tools and look at the data, analyze it. Now you’ve got that capability to do that for lots of things.

    Bennatan: Yeah, it’s exactly—it’s exactly right. I mean, you know, it’s like, in the business world, there was always this notion of a customer 360 where, you know, what’s the point that I have one system for quotes and one system for support and one system for my policy and I never know what’s going on, and I can’t upsell, and I can’t cross sell. It’s the same thing in security, okay?

    There’s this notion of data 360, data security 360 where, if it’s there, you know, then prioritization becomes easier, right? If you can check your configs and your vulnerabilities, okay, and you get this huge list—what are you gonna do? But if you know that, of this huge list, your consumer data is here and that, you know, the crown jewels are here—great, you don’t have to boil the ocean.

    Ashley: Mm-hmm.

    Bennatan: And if, at the same time, you can bring in and you can check your entitlements and you can say, “Well, you know, I have this vulnerability, and it is important data, but I did a really good job at narrowing it down and access to it is only by these individuals and access is Kerberized and—I don’t know, and maybe I also regulate the policy very tightly—these are all related, okay? So, if you have that single platform, you’re in much better shape than…you know, it’s like, I’m trying to think of a metaphor of, you know, when you have a single viewpoint, that viewpoint may be wrong. When you have—it’s kind of like, you know, these days, this discussion about whether news means anything any more, because it’s so tailored to what you wanna hear.

    Ashley: Mm-hmm.

    Bennatan: But it’s kind of like somebody forcing you to go and watch Fox and CNN and NPR and—you must watch all of these before you make a decision. It’s the same way. We can’t reduce risk by just looking at one thing at a time.

    Ashley: Yeah, we certainly have our own selection bias and data as we do our news programming as well, right? [Laughter]

    Bennatan: Yes, exactly.

    Ashley: Well, this has been a lot of fun talking with you. I just have a sense 2021 is already off to a good start with vaccines and the COVID situation. Hopefully—

    Bennatan: Hopefully.

    Ashley: – you know, we’ll round that hump and folks can really figure out new ways of collaborating with each other without worrying about COVID. Maybe that day’s down the path not too far, but we hope, anyway.

    Bennatan: Yes.

    Ashley: So, it’s been fascinating talking with you. So, we didn’t talk about, you know, go into product stuff, it wasn’t that conversation, but tell us a little bit about where folks can find out more about Imperva.

    Bennatan: So, Imperva.com, we are—you know, if we’re talking about data security, then there’s a whole section there, and you’ll see that we’re investing a lot in, you know, things that we believe 2021 and our customers are implementing. So, a lot of kind of complete security, complete security, whether it’s for compliance, security, privacy—a lot of effort in privacy, a lot of effort around data lakes, everything around cloud. Any cloud that you can think of or any data service, any data-centric workload on the cloud is something that we’ve worked very hard with the cloud vendors to provide support for. And yeah, I think 2021 is gonna be an exciting year because of this modernization that everybody’s embarking on.

    And my hope and what we’re trying to do with Imperva is that the path of modernization for the applications and for functionality also brings with it modernization of security, updated security. That’s really what we’re working on.

    Ashley: I would definitely suggest folks check you out. In thinking about our conversation before of the response to regulation and policies and things like that of looking at what other leaders do, Imperva works with some very impressive companies who are solving the data challenge from a platform standpoint. So, there’s a lot of good reasons to check out Imperva, because there’s a lot of problems you’re solving on the advance that could help a lot of organizations, so I hope people will do that.

    Bennatan: Yeah. Well, thanks, Mitch. Always great to talk.

    Ashley: You bet. Good to talk with you, Ron, and thanks, everybody, for listening in. It’s been a great conversation, and I wish all of you a great 2021 as well.

  • Apollo GraphQL Offers Free API Management Service

    Apollo GraphQL Offers Free API Management Service

    Apollo GraphQL has announced the general availability of a free Development Graphs cloud service to simplify management of GraphQL application programming interfaces (APIs) at scale.

    GraphQL APIs have been gaining traction as an alternative to REST APIs because they provide developers with more control over who can access data via an API and are easier to build and maintain.

    Apollo GraphQL CTO Matt DeBergalis said that approach makes it possible to build APIs using the Agile development methodology versus the waterfall processes that typically are employed today to construct REST APIs.

    Originally developed by Facebook, GraphQL APIs are being extended by Apollo GraphQL via a set of Studio tools based on a graph that makes it easier to discover those APIs and search GraphQL schemas. IT teams can also use the tools to launch queries against those APIs using an integrated development environment (IDE) dubbed Explorer. Developers are not required to have a deep knowledge of GraphQL syntax or the shape of a schema to launch a query. They can also copy links to share their queries with other users to copy a version of a query that can be run in the terminal as a CURL command.

    DevOps teams can also search for types and fields in a schema using tools that guide them through the process of querying any field in their schema. They also can find and add fields to a query without needing to know in advance where those schemas are actually running.

    The tools also make it possible to view samples of results to better gauge what data will be returned by a query, along with latency rates for each field in a query to better understand the overall cost of a query.

    DevOps teams can also view response data from their queries as JSON or as tables and create mock responses to queries to test operations, Finally, each request and view of every query can be viewed alongside the data returned.

    Connections to a private sandbox in the cloud for each local GraphQL server or any development environment are enabled using the Development Graphs tool.

    DeBergalis said he doesn’t expect most DevOps teams will rip and replace the REST APIs that are already strewn across most extended enterprise. Instead, most new applications will make use of GraphQL APIs that over time will become the de facto standard for building APIs. However, developers are clearly voting with their feet to abandon REST APIs in favor of a more agile approach, he said.

    Apollo GraphQL is looking to become the software development lifecycle (SDLC) platform for managing GraphQL APIs at scale, DeBergalis added.

    The challenge organizations face now is finding a way to integrate the management of those GraphQL APIs within the context of a larger set of DevOps workflows. Of course, that may wind up being more challenging than initially anticipated when they consider REST and GraphQL APIs could be employed alongside one another for years to come.

  • DevOps’ Data Storage Problem

    DevOps’ Data Storage Problem

    The technology sector has always been about problem-solving. When the value of big data was finally embraced, thanks to new analysis capabilities developed in the late nineties and early aughts, the industry adapted its mindset toward storage by investing in on-premises data centers to help store the data that would drive better business decisions. When this rise of data accelerated software development and deployment, creating organizational silos between developers and operations, the industry responded by creating DevOps to improve collaboration.

    Now DevOps has revolutionized the way companies do work by eliminating silos, increasing agility and creating greater visibility, resulting in faster deployments and better service overall. These new work styles have only increased the amount of data produced and the necessity to access it anytime, anywhere has made migrating to the cloud the only real viable solution to any successful DevOps team. Unfortunately, this has spawned a new challenge for the technology industry to solve: the cloud.

    Data and the Cloud Predicament

    Early DevOps adopters quickly leveraged the benefits of the cloud in increasing collaboration brought on by improved data accessibility. However, these early adopters typically entered agreements with the big three cloud vendors that offered increased storage capacity and a seemingly more flexible and accessible format. In most cases, these agreements provide instant gratification, but as DevOps expands over time new pain points are discovered as service and egress fees pile up and teams are forced to either increase budgets to accommodate storage needs or risk meeting limits and data loss.

    This situation created a cloud storage predicament: Companies either pay for more cloud storage than they’ll ever need or have to choose what data was kept and what was deleted. These unsustainable vendor agreements introduce financial pain points for scaling up or scaling down storage space, outweighing the benefits of keeping all of the valuable data being produced and collected day-to-day. As a result, most are stuck thinking about how much and what to store instead of using all of its data to drive the business forward.

    This form of vendor lock-in is a detriment to DevOps teams that rely on seemingly endless amounts of data to experiment, conduct maintenance and develop and deploy new applications.

    Accelerating DevOps With a Bottomless Mindset

    An IDC report recently indicated that the amount of data stored is now expected to hit 59 zettabytes this year alone, with the next three years of creation and consumption almost eclipsing that of the previous 30 years combined. And while DevOps has certainly contributed to this spike, the reality of our digital worlds is that data is king, and a company’s ability to seamlessly store, access and leverage it is what will set it apart from the competition.

    The problem with this growth is finding ways to sustainably store this data in the cloud, particularly when teams are handcuffed to existing storage agreements that are draining budgets. This starts with DevOps teams rethinking their approach. Instead of thinking of cloud storage and accessibility as a recurring cycle of bills and service limit notifications, cloud strategies need to focus on the ability to only pay for the space needed at any given time without the worry of additional fees for accessing archived data. Need to scale up storage capacity? Great! Add as much as you’d like. Need to scale down for a particular reason? Don’t stress over unused real estate just sitting in your vendor’s data center. Need to access archived data at a moment’s notice? Don’t hesitate out of fear of a knee-buckling egress fee.

    With this shift, DevOps teams can move away from conserving, archiving and destroying data toward gathering and utilizing all of it to drive the data-driven insights required in today’s digital economy. This change in mindset toward a “bottomless” cloud removes constraints and enables the ability to innovate at a faster and more sustainable rate.

    Unlocking DevOps Innovation

    Too often people think of DevOps and data management as being separate entities, but as software and application development evolves to include capabilities such as predictive analytics, it’s clear that a marriage between the two entities is needed. By rethinking its data strategy and embracing a bottomless approach, DevOps teams can unlock even greater efficiencies, leading to the next revolution in technology innovation.

  • How Centralized Log Management Can Save Your Company

    How Centralized Log Management Can Save Your Company

    Through centralized log management, organizations can consolidate all log data into one central, data highway and directs logs wherever they need to go

    The digital transformation wasn’t gradual. It came crashing into industries worldwide in response to safety restrictions around the pandemic.

    Nearly a third of Americans are working remotely in light of the pandemic—drastically increasing the external users—with no signs that employees will be returning to the office full-time post-COVID-19. A recent PwC survey found that 83% of employees want to work remotely at least once a week and that 55% of employers anticipate their employees will do so even after the pandemic.

    Companies have implemented new and creative solutions to deal with the spike in remote access: VPNs and SSL connections through web applications, to name a few. However, many of these solutions were only a bandage that won’t suffice for long term remote access.

    There is one thing amid all these changes that can become an increasingly large problem if left unaddressed: logs. Logs are collections of data that continue to be collected whether users are in the office or in their living room. Without centralized log management, this data can put organizations at a critical security risk. As the digital transformation takes industries by storm, it’s important for organizations to remember the importance of managing logs.

    Today’s Log Landscape

    When a company’s remote workforce jumps from 50 to 5,000 in a matter of weeks, things change. Compared to in the office, logs at these companies have jumped by 10,000%. Remote work generates logs on security access points, remote access, server logs, terminal services and application logging—all of which are fed into an organization’s security information and event management solution (SIEM).

    This jump is creating and revealing more blind spots to companies, which proves how difficult it is for administrators to increase the rates for their security monitoring tools to reach that level without drastically upping their company’s budget for its SIEM. Even Microsoft and network providers are running into issues with so many changes happening at the same time. Additionally, companies have to worry about privacy laws, making sure all personal data is encrypted and secured.

    Centralized Log Management

    Organizations should implement centralized log management (CLM) to consolidate all log data into one central, data highway that collects all logs and directs them wherever they need to go. CLM solutions reduce SIEM costs once SIEMs are no longer effective log management tools.

    Dropping all logs into a SIEM spikes costs, so oftentimes only a portion is collected, which creates fragmented or incomplete pictures and impacts security monitoring and incident response. CLMs lift the burden of having to hire staff, provide training and support for SIEMs. CLMs also reduce the costs organizations would incur with their SIEM providers, as well as the risk of endangering the SIEM infrastructure by storing unmanaged logs.

    Fragmented data collection can become a unified data collection with a data highway. Organizations can now filter unruly data and deliver only what you need. This helps overcome the age-old strategy of letting separate teams have their own sources of data, which could instead be directed to the appropriate team via your data highway. The data highway lets you collect once and use it many times, where it’s needed.

    Parsing: Cleaning up the Data

    Once logged, data needs to be parsed. To prime your data to be parsed for specific items, unnecessary and unwanted information must be filtered out. This diminishes the storage space logs take up and increases the usability of the data.

    One example of superfluous information is the timed mark that many applications add into the log of their system to show they are online. Unless a security auditor will need to see this, there is no reason why an organization should be paying to store it in their SIEM. Administrators are even able to filter out all extraneous text and add parsing for specific events. This quickly and easily reduces costs that are likely spiraling out of control during this time.

    Parsing, filtering, masking and other transformation techniques in your data highway allow security teams to overcome privacy issues and filter out personal information that shouldn’t be distributed. Resolving this problem could become crucial as more and more personal data is being collected than ever before, and as privacy laws are becoming stricter.

    The Data Highway

    Build your dream team to utilize your new data highway to its greatest potential. Not everyone reviewing logs will be SIEM experts or as skilled as a Linux or UNIX administrator (or even one at all). Build your team to be able to easily operate your new data highway, because you don’t want it to be gathering dust on the shelf.

    With your new data highway, you’ll be able to optimize your SIEM, increase your likelihood of meeting compliance requirements and be able to log from more places with easy searchability. With that encrypted data store, your compliance officer may even be able to sleep at night. Beyond the SIEM, you’ll be able to send your data anywhere: Kafka, MongoDB, any database, big data systems and more.  Don’t just optimize your SIEM; build your data highway and collect those logs once, distribute them where they need to go and cut costs with centralized log management.

    So, in this crush of new technologies spiraling into the new digital transformation age, don’t forget the importance of managing your logs. IT and security admins shouldn’t be in the position of going to company management to inform them of uncontrolled SIEM costs. They can be managed and reduced without losing their effectiveness by simply feeding to the SIEM only the data that needs to be there.

  • Collibra Employs Microservices for Data Management

    Collibra Employs Microservices for Data Management

    Collibra has launched a Collibra Data Intelligence Cloud based on a microservices architecture that promises to alter the way data is managed in enterprise IT environments.

    Jim Cushman, chief product officer for Collibra, said its approach sets the stage for leveraging microservices to create data workflows made up of reusable components. In fact, there may come a day soon when some data sets are accessed and managed as a reusable microservice, he said.

    The Collibra Data Intelligence Cloud provides a catalog for managing metadata that, in addition to making it possible to build workflows, ensures the integrity and security of data being accessed and stored. End users can then analyze that data using a wide variety of third-party tools, said Cushman.

    At the same time, Collibra enables metadata ingestion from data sources such as data lakes and data warehouses, which means IT teams don’t have to move all the data they need to manage into Collibra Data Intelligence Cloud to create a workflow.

    Collibra is also making available a beta version of its Data Matching tool for qualified customers. That tool identifies duplication within a data source and how that data might overlap with other data sources.

    Finally, Collibra has added a data scoring component to its catalog to identify the most relevant data sets; provide enhanced support for third-party business intelligence (BI) and extract, transform and load (ETL) tools; and strengthened rights management capabilities to make it easier to enforce compliance requirements.

    Cushman said the goal is to provide organizations with an easier approach to storing and organizing trusted data. As the amount of data an organization collects grows, so, too, does the opportunity for conflicting data to be incorporated in applications. The Collibra Data Intelligence Cloud creates an opportunity to make data universally available to applications without requiring them to copy it, he said, noting that approach reduces not only the amount of data that might be in conflict as data sources are updated but also the amount of infrastructure that would have been required to process and store duplicate data.

    Organizations have been struggling with data management since the dawn of IT. That’s become a more pressing issue as more organizations seek to turn data into a strategic asset that drives digital business transformation initiatives. However, those initiatives are often derailed because so much of the data is either conflicting or simply too stale to be relevant. Collibra is making the case for a single platform to bring order to what historically has been a chaotic process.

    It’s not clear to what degree organizations will be focusing more on data quality and management in the months ahead. In the wake of the economic downturn brought on by the COVID-19 pandemic, it’s clear digital business transformation has become a necessity. The challenge is that unifying how data is managed in any enterprise requires a massive amount of political capital. Each business unit within an enterprise IT environment tends to guard its data privileges jealously.

    However, as microservices make it possible to make the business a lot more agile, the day when organizations finally confront their longstanding data management issues may finally be at hand.