devopsy.kinsta.cloud

Tag: data analytics

  • CIOs See Dramatic Change Ahead

    CIOs See Dramatic Change Ahead

    Driven by heightened competition and a rapidly changing business environment, many enterprises now find themselves faced with decisions that could make or break their future.

    Consider this: Within three years, 44% of respondents to the 2019 Harvey Nash / KPMG CIO Survey expect to fundamentally change their product, service offerings or business model. It’s a finding that spans enterprises both large and small, the survey said.

    This could explain why IT leaders reported their technology budget to be increasing the most in 15 years. The report found 45% of respondents favor budget increases for money-saving technology efforts, such as task automation. Interestingly, almost two-thirds, or 63% of organizations, report that considerable amounts of their enterprise technology efforts are managed outside of the control of their IT department.

    However, the survey showed those organizations where the IT team is formally involved in business technology decision-making have better business outcomes. These enhanced outcomes include improving new product time-to-market and employee experience to be better than their competitors. Still, four in ten companies are not formally involving IT in those business-led IT decisions.

    Increased investments in automation and artificial intelligence (AI) are expected to place considerable churn on the workforce. CIOs are expecting to witness one in five jobs being replaced by AI/automation within the next five years. However, 69% of CIOs also expect enough new jobs to replace those positions lost to automation.

    “Boards are asking their CIO and technology team to prioritize automation of jobs. How organizations adapt to automation will increasingly become a priority, and many are not at all ready,” said Albert Ellis, CEO of Harvey Nash.

    Despite CIOs enjoying a rising influence within their organizations, fewer CIOs have seats on the board of directors at their organization—having fallen significantly from 71% who held board seats just two years ago to 58% today.

    The 2019 Harvey Nash/KPMG CIO Survey, conducted between December 2018 and April 2019 across 108 countries, is based on the responses of 3,645 CIOs and technology leaders.

    While automation and AI may, or may not, displace workers in the future enterprises are counting on them today to help close their perceived skills shortage. According to respondent’s, enterprises continue to find it a challenge to obtain the talent they need. In fact, respondents reported skills shortages at their highest level since 2008.

    The employees most difficult to find are those in data analytics (44%), cybersecurity (39%) and artificial intelligence (39%).

    Finally, respondents surveyed who consider their organizations to be digital leaders (those who see themselves as very effective or extremely effective at using digital technologies to advance their business strategies, performed better than their competitors on every aspect surveyed.

    According to the survey, digital leaders outperformed in time to market (53% versus 34%), customer experience (65% versus 49%), revenue growth (55% versus 43%) and profitability in the previous year (50% versus 37%). Digital leaders, the survey said, are also more likely to introduce ‘major new changes to products and services’ in the next three years (55% versus 39%) and focus on profitability.

    — George V. Hulme

  • Cloud Complexities Hinder Effective Enterprise Data Management

    Cloud Complexities Hinder Effective Enterprise Data Management

    Enterprises everywhere are rushing to embrace automation and AI to improve existing business processes and innovate. But they’re learning that if APIs are the glue they use to build their systems, data is the fuel that drives these efforts forward.

    According to data management provider Actian, however, enterprises are experiencing significant challenges when it comes to obtaining fresh data, effectively wrestling with the complexity of data and getting to the point of driving innovative initiatives.

    The Actian Datacast 2019: Hybrid Data Trends Snapshot surveyed 300 IT decision-makers and professionals in businesses with 259 or more employees. Ninety-four percent of respondents said it’s important for them to have a system in place that makes certain users keep getting data, yet 58% of respondents admitted it is only somewhat likely, or not likely, they are actually getting fresh data to their users. So much for smart, real-time decision making.

    That could explain why just 34% of respondents who said they are using data for decision-making are using it to push innovation forward, instead, the majority are using it for traditional business operations and reporting.

    “Business as usual operations keep enterprises running from day-to-day, but limiting data to operational reporting tasks means missing a key piece of the data puzzle–new insights that lead to awareness of products, markets, consumer trends, strategy and more,” said Actian in a statement.

    The costs of data management are also taking their toll. The survey found 84% of enterprises would deploy more data if they could do so more cost-effectively.

    Those costs could be driven, in part, by the complexity created with data being stored within application siloes and across multiple cloud environments.

    “Over 50% of these businesses say data complexity issues, due to siloed applications, are a top barrier to entry for accessing data and gaining effective real-time insights,” said Actian.

    Actian concludes, unsurprisingly, that enterprises must get to the point that they can exploit their data in real-time and better manage data across cloud platforms.

    “Slower decision-making is only one consequence of having to wait for data to become available for analysis,” said Jack Mardack, vice president at Actian. “There are exciting new analytics use cases, like customer-360 and hyper-personalized real-time offers, that simply don’t work with stale data”

    “Enterprises are struggling to keep up with the growing demand for business-as-usual reporting that it leaves no viable bandwidth for generating new insights. This requires that enterprises provide not only the necessary data management infrastructure to make data from all their sources available to analytics discovery, but also that they invest in and empower data teams that can pursue new insights from new data spaces,” stated Actian.

    — George V. Hulme

  • The Essential Toolbox for DevOps Analytics

    The Essential Toolbox for DevOps Analytics

    With DevOps teams under constant pressure to improve software delivery cycles while maintaining—and improving—quality, many have turned to continuous testing (CT), implementing it as part of their overall development strategy. CT offers a number of benefits to DevOps teams, but it also introduces new challenges, specifically infinite amounts of data that require analysis. As more teams integrate CT into their strategy, it is clear that analyzing, understanding and filtering the results data quickly is critical to preventing bottlenecks in the DevOps process. The problem is, teams simply don’t have the time needed to analyze data.

    Current research indicates that teams spend anywhere between 50 and 72 hours per regression cycle analyzing test results. This includes filtering out noise and assessing failures that may impact their software releases. However, the amount of time DevOps teams have to review and qualify test results has shrunk dramatically in recent years—from days to hours or even minutes, in some cases—leaving teams scrambling.

    With pressure to accelerate velocity and deliver consistent, high-quality experiences, DevOps team require new analytics solutions to help them manage large test-results datasets in a CI/DevOps environment.

    Let’s explore five essential tools that enable DevOps teams to quickly and efficiently analyze data, triage issues and act upon failures with the best possible insights.

    Executive Dashboards

    Dashboards have evolved immensely over the years, allowing dev managers and QA managers to easily examine the pipeline, see CI trends related to time, build health and more. When looking for the best solution, make sure it includes quality heat maps and CI dashboards, which make it easy to spot an anomaly in the CI pipeline and allow managers, DevOps teams and practitioners to quickly drill down into the single test report and the issue.

    Single Test Report Visibility With Advanced Reporting Artifacts

    With a single test report, practitioners have access to the entire flow of test steps, video, logs, screenshots and Jira–even the ability to drill down into the source code. This makes it quick and easy to locate an issue. For teams, these means more time fixing and less time searching.

    Cross-Platform Reports

    These analytics reports show the UI/UX simultaneously across multiple screen sizes, resolutions and platforms. Since the boundaries between the digital platforms need to be seamless, whether on desktop browsers or mobile devices, having the ability to see the UI/UX on multiple form factors (and layouts) at the same time is essential to assess overall quality.

    DevOps Noise Reduction Tools Powered by AI and Analytics 

    Today, finding which error classifications are true application bugs is critical to efficiency. Noise reduction tools that are powered by AI allow teams to filter out problems caused by various issues that are not software defects—for example, device connectivity issues and wrong scripting. Having a tool with the ability to locate each failed test execution, the root cause and the classification category is a huge productivity boost for both the test engineer and the developer.

    DevOps and Actionable Insights

    Analyzing data is one thing, knowing what to do it with it is another. Today, there are a number of analytics solutions on the market that provide teams with both capabilities. Once an issue is detected, classified and reported, having deep insight brings practitioners halfway toward the resolution of the issue.

    To succeed in software delivery, teams not only need better automation as part of their processes, test flows and CI/CD workflows, but also an analytics platform to manage all of their test results data. Since DevOps involves a collection of team members from all parts of the software delivery lifecycle (SDLC) process, the central platform needs to meet the needs of all team members. As you work to build your next test analysis toolbox, consider the five features above to efficiently evaluate the data, act upon it and deliver iterations and features with confidence.

    — Eran Kinsbruner

  • DevOps Chat: We Are Devo, With Walter Scott

    DevOps Chat: We Are Devo, With Walter Scott

    If you have massive amounts of data, Devo wants to help. The Madrid-based company specializes in the handling and processing of massive amounts of data.

    In this DevOps Chat we spoke with Devo CEO Walter Scott about the use cases for Devo in a data-crazy world. Walter is a seasoned CEO in the tech sector. He knows a problem when he sees it and he thinks Devo is the answer. Have a listen to why and check out Devo at devo.com.

    As usual, the streaming audio is immediately below, followed by the transcript of our conversation. Enjoy!

    Transcript

    Alan Shimel: Hey everyone, this is Alan Shimel, devopsy.kinsta.cloud. You’re listening to another DevOps Chat. Really happy to have a new company, a new friend, joining us on DevOps Chat today. I have Walter Scott, CEO of Devo. Walter, welcome to DevOps Chat.

    Walter Scott: Thanks, Alan.

    Shimel: So, Walter, you and I met recently in Iceland, of all places, at a Insight IGNITE event that really kinda focused on DevOps, cloud, cybersecurity, if you will, the topics, but it was my first exposure to Devo. I don’t know if our audience is familiar with Devo. Why don’t you–give us the foundation, man. “We are Devo.” What are you guys? What do you do?

    Scott: Well, Devo is a data analytics platform that’s the next gen of technology, that provides a data fabric to allow our customers to take all of the noise that exists in the machines and petabytes, and petabytes of data coming in, for them to really start getting signals so that they can start their automation or their AI or their machine learning process, to make their companies more efficient; to be able to have them act more quickly; as well as to be able to get the people who use our platform, to get them the time back that they spend in managing the underlying data platform and allowing them to add the value to drive IT, to drive that value back to the business.

    Shimel: Excellent. And, you know, Walter, one of the things that impressed me with Devo was this wasn’t your sorta typical Silicon Valley story—a couple of wiz kids come up with a PowerPoint business plan and get funded via VCs, pre-revenue, pre-product even. Devo—the underlying technology here, the company’s been around a while and it’s not a Silicon Valley company, right?

    Scott: That is correct.

    Shimel: Give us a little background on that.

    Scott: Well, you know, over my six stints of being a CEO, I’ve never really run an American-based company. There’s great IT being built all over the world. I’ve done them from Malta to Scotland, Germany to Armenia to now with us here in Madrid, is where the company started. And we had a not Silicon Valley startup, but it was five guys—or four guys and a girl or woman—and started off the product in a research key, to really build something special and change the market because the bank that they worked for had to completely change the way they do business.

    And over the course of three years in research and we finally launched a product in 2014 and have been three-figure type growth every single year, since the date of the company being founded, ’cause it’s such a large market. And it’s really accelerating because of what we’re seeing happening with businesses and trying to get ahead of everything from getting arbitrage out of supply chain to solving their security problems, just to start with basic log management and monitoring to consolidate tools and increase agility.

    Shimel: Sure. So, Walter, you were—as you alluded to, this is your sixth or seventh startup where you’ve come in as CEO and had some tremendous success. And, look, not everyone is a success, right? That’s the mantra in business, but you keep plugging. When we look at Devo—and you and I spoke a little bit about this in Iceland, but so much of what we see as, quote unquote, “AI” today is really machine learning. And so much of machine learning is based, really, on data, right? You’re only as good as your data. In order to find patterns, you have to have enough data to analyze it and make actionable kinda calls out of it.

    But this is not a new problem and there’ve been other solutions. What makes Devo’s solution—I don’t know if you use the term “next-gen” or “new-gen” or something like that, but what makes Devo’s solution different?

    Scott: Well, you know, this has been going around a long, long time and you’ve heard me say, in the absence of data, the machine can’t learn. And I go back, all the way back, to the Walmart days in the ’80s, when they built—I would argue … them as the first person to really take the competitive advantage, coming from their data warehouse, right? And —

    Shimel: I agree.

    Scott: —They actually had the line, they could tell your daughter’s pregnant before you could by looking at the data. Well, today, that’s become necessary, but, over the last 10 years, it’s really—what’s happened with the cloud is being able to get that data manageable. I don’t like the term “AI.” I like more “augmented intelligence” because we’re not at the place—or even narrow AI ’cause I think it can solve very specific business problems that people might have. We have one customer that needed to actually build some ML and machine learning to detect what a bot was, were hitting their site, because, when they launched a new product, the bots would come in and sell it all in the secondary market. They wanted that brand loyalty to go to the end users.

    Or we have another customer that has a bunch of IoT devices integrated into their sporting equipment, where they’re trying to figure out if somebody’s gonna get hurt and using machine learning to be able to define what is that case, in terms of repetitive hits, but they need that data over long scales. And just the data science is actually making that possible. It could be done in the past; the problem was it just wasn’t cost-efficient enough to do it.

    We have customers that are taking literally over a trillion events a day to try to—and building machine learning models to understand which customers may leave their environment, based upon dropped calls or poor support integration. And it’s not just having that one set of data; it’s about having hundreds of different sets of data to help prove those hypotheses, to get those more and more efficient. And the data scientists are spending 70% to 80% of their time getting their arms around the data. We take that problem away from them and allow them to have—to solve that other 80% now, they can put in training their models and getting data to make them more and more efficient.

    Shimel: Excellent. Great answer. Hey, you know, Walter, you and I, this is neither one of our first rodeos. And, when I look at your job, your mission over there, at Devo, one of the things that I guess it’s a double-edged sword ’cause it presents so many opportunities, but, at the same time, there’s an axiom in business—“Put all your wood behind one arrow,” right? There are so many use cases for what you can do here, right? There’s the cybersecurity case, right? Where, “Look, man, we need this data. We need to be logging everything. We need to find if we’re under attack, where the attack’s coming from, what’s happening?,” right? So there’s that. And one can spin up an entire company around just using this kind of technology in cyber.

    You alluded to a few other use cases, right? That had nothing to do with cybersecurity. Better business intelligence. Understanding your customers better. Understanding your product better. There’s only so many targets, though, that you can go after, right? Without a tremendous—you know, you wanna build an IBM kind of force or something or a Dell force, that’s a whole different story. So how do you—Walter, as the CEO, how do you prioritize? What are the best use cases for this technology? What are the right markets for us to pursue?

    Scott: Well, what you say no to is just as important as what you say yes to, in a business.

    Shimel: Yep.

    Scott: And, right now, we are very, very focused in the Americas, as well as in western Europe, right? So we’ve been able to say no to Asia for another year or two. Our value proposition is really about the biggest of the big—the telcos, the banks, the large retailers, people who’re looking at real scale that they’re trying to solve problems with. But the fact is, is where I see the market going over the next three years is more about consolidating the DevOps, the ITOps, the Network Ops, the SecOps. All of those are coming into one. And I’ve heard a couple of people start using the term “fusion center,” which is really helping them drive the business and stop pointing fingers and reducing that mean time to response.

    Well, every security product starts with the logs, right? And every IT monitoring product starts with the logs and the data. And the advantage of us being able to load in all of that relational data, which is about 20% of our data that’s in our systems, across our customer base, is things that is coming out of Xdata or data warehouses or relational or customer application type or weather data or threat hunting data that comes in, and being able to combine all of that. So, for us, it’s the larger customers or companies that have data at scales that we’d see in the global 1,000 and really looking at the people who understand that what Google, and Apple, and Facebook, and Netflix have done is taken a data-first approach to their business.

    And, when we look at customers that are forward seeing—that see forward into the market, they realize there are a whole new set of competitors and being able to ask what their real competitive advantage is in their business and the data that it takes for them to be able to do that, whether that’s a telco to figure out how to launch a 5G network because they have all the 4G, 3G and 2G data, whether it’s in their customer retention rates or net retention of “Can they get more dollars out of their customers each year?” to whether it be their manufacturing process and figuring out that arbitrage in the supply chain.

    So I think we are—because of the ease of use of our product, because it’s so easy to get data in, because we keep data in raw format. So you can have one set of data that’s very easy to get in and use it 100 times, not having to index it or reduce that data because it just makes it more efficient.

    But I see us, in the next couple of years, really focused on the security side of the equation. Looking at people who’re doing the transformation and the weight that it’s gonna take on their DevOps and CloudOps teams and being able to work closely with the network and the DevOps teams to be able to grow their business because the cost of becoming so great on having so many different tools, at different layers of your network, I say that the best and the brightest companies will take control of their own data, consolidate those tools so they have a better competitive advantage.

    Shimel: Good. Good. Good. Walter, I’m gonna throw a term at you and I want your opinion on it, “AIOps.” What do you—this is a term that we, here at devopsy.kinsta.cloud anyway, we see it rising with a bullet. And, like you—to me, there’s AIOps with a capital “A” and a capital “I” and it’s like Robby the Robot stuff that’s gonna make us live like The Jetsons, finally. And then there’s AI with a small “a” and a small “i,” like the real, functional, kinda down in the dirt–maybe it’s not as sexy, but it provides real change for us. AIOps is—man, everyone—what’s your story on AIOps? It reminds me of when cloud came out in 2005. What do you think?

    Scott: So let’s be very clear. AIOps is a term that’s been coined by Gartner. Out of the last 100 conversations I’ve had with CIOs, I’ve only heard it referenced once or twice. I think it’s more the vendors that are getting on it. And, when you look at it, log management is a big part of AIOps. You know, user behavior analytics is part of AIOps. The anomaly detection is part of AIOps, right? There’s seven or eight different components to that. I don’t think any vendor right now, in the world, really has defined themselves as an AIOps vendor. I think we’ve probably got three years, through help of M&A and deciding what that is, ’cause incident response or the service management is also there, so I think it’s gotta evolve.

    But, of the seven or eight different components, the monitoring, the dealing with events, the user behavior, the anomaly detection and the log management is there, which is where we focus. And I think the enterprise log management, I thought, five, six years ago, was solved, when we solved who all those vendors are, but now, with the complexity of the cloud, it isn’t solved at the scale customers need to do with it. We had solved router management; now, with SD-WAN, it’s a whole new market.

    Shimel: Yeah. You’re right. I mean, and it’s funny that things that–it’s kinda like the measles, right? We thought we cured measles and here we are back with measles. Walter, we’re both CEOs of companies and I don’t know how many people listening out there are CEOs—many of our listeners are kinda getting their fingernails dirty, practitioners as well as managers at large enterprise—what keeps you up at night about Devo? What do you worry most about, Walter?

    Scott: Well, I think part is we have to manage tens or hundreds of petabytes of data for our customers, right? And I think we have a very large team internally for security, to make sure that the data that we have is protected and we work very closely with Amazon and Google and Azure to make sure that we address that. And that’s something that you’re never gonna sleep well ’cause you have to be passionate about making sure that you protect your customers.

    We were on the phone literally an hour ago with a manufacturer that makes toys for children, and that was the question that’s keeping him up–he says, “I have 10-year-old boys and girls logging into my website. I have to protect that data, making sure in the things that we do with being able to obfuscate data at the row, in the column level, without having to duplicate it and our algorithms that are in there.”

    But my job, as a CEO, is to help people build careers. I’ve been very fortunate in my career and have had some very large exits in part of the business, but it’s making sure that the team around us continues to make sure that we handle all of the things, in dealing with our business and our main offices in Madrid. We also have a big office in Cambridge, Massachusetts, because I love the guys and women coming out of MIT, where we do a lot of hiring. But it’s mostly about the team and then making sure that we continue to have what I call a industry-abnormal or an anomaly in terms of net retention. And our goal is to help our customers continue to grow because it benefits us and wanting to make sure that we make it easier for them to be able to get more value because our mission is about making IT so that they can provide data to the business that is actionable.

    And my belief is is the CISO or the CIO that has control of that data, if they push it out to the business and say, “Hey, did you know that, on this particular case or when this happens, we’re losing customers or we’re gaining customers?” That data is 80% more likely to be leveraged than it is if the customer asks the business. So—

    Shimel: Nice.

    Scott:–I think it’s empowering the IT to accelerate value to the business and most of my nights are, well, we see this new customer is bringing in this type of a data, bringing in our data science teams to help think about use cases that we can drive more value from.

    Shimel: Excellent. So, Walter, as I mentioned to you before we started recording today, the good news and bad news about this audio format is it goes so darn quickly. And we blew past our time. So what we’ll do, though, is we’ll—secret’s out: you live down in Florida, too. We’re gonna need to get you down here and into our studio and let’s do a whole video on this. We’ll continue our conversation in a video format. But, for now, we’re gonna call a—have to pull the plug on this one. Walter Scott, CEO of Devo, thanks for being our guest today on DevOps Chat.

    Scott: Great, Alan. Thanks.

    Shimel: All right. Hey, this is Alan Shimel, devopsy.kinsta.cloud, and you’ve just listened to another DevOps Chat.

    — Alan Shimel

  • The Biggest News from Google Cloud Next ’19? Better Capabilities for Big Data

    The Biggest News from Google Cloud Next ’19? Better Capabilities for Big Data

    Data analytics is about to become a whole lot easier. Google’s annual Google Cloud Next conference brought plenty of new announcements and innovations. But the search giant’s focus on data has stolen the show. Partners of the tech titan such as Fivetran, DataStax and Hammerspace are bringing unprecedented data capabilities to the Google Cloud Platform (GCP).

    Accelerated Data Analytics = Automated Data Infrastructure

    When Google BigQuery came onto the scene, it changed data analytics forever. Acting as a serverless and scalable data warehouse, the service unlocked unparalleled price-performance and productivity for enterprises around the world.

    Now, Google aims to simplify things even more with BigQuery Data Transfer Service, a new offering that lets organizations automate data movement from SaaS applications to Google BigQuery on a scheduled basis. Basically, you can build a robust and dynamic data warehouse without having to write any code.

    Fivetran, a startup that lets you automate data infrastructure into your warehouse for advanced analytics, is one of Google DTS’ launch partners. The company has built a solid reputation around its prebuilt, maintenance-free data connectors; you can deploy them in five minutes and completely replicate applications, databases and more.

    Without leaving the Google Cloud environment, users can automate setup and product integration between Google Cloud and Fivetran. This allows them to drastically reduce overhead maintenance when centralizing disparate data sources. In turn, this accelerates analytics and reduces time to insight.

    Currently, Fivetran and Google Cloud offer connectors for more than 100 sources, including NetSuite, Salesforce, Marketo and more. Fivetran is currently developing new data connectors, so you can expect this list to expand soon.

    A New Database-as-a-Service Offering

    DataStax is a company and GCP partner that delivers “Active Everywhere,” distributed cloud databases to support always-on, web-scale applications. It has roughly 400 customers across 50 countries.

    The company’s CEO, Billy Bosworth, realized that its clients needed more flexibility in moving data among public and private clouds. So, for the first time ever, it’s now providing a database-as-a-service (DBaaS) offering.

    Bosworth explains the demand: “Recently, enterprise customers have said they want those database attributes available to them in a variety of on-premises and cloud options—or in other words, for their hybrid cloud and multi-cloud realities.”

    The new agreement with Google Cloud to offer DataStax Enterprise as a DBaaS will do just that—and more. Not only is the new service the first major step in making this wish a reality, it also will provide users with optimal flexibility in how and when data gets moved to public and private clouds.

    Hammerspace DaaS Arrives on GCP

    Fivetran and DataStax aren’t the only ones bringing new data management capabilities to GCP. Hammerspace, a Los Altos, California-based company, recently announced its new Data-as-a-Service (DaaS) platform has arrived on the GCP Marketplace.

    Hammerspace is dedicated to simplifying data consumption for hybrid cloud and container users. The company provides a global data workspace for organizations that is free from the usual lock-in that comes with storage silos.

    Since its cloud data control plane is both storage and protocol-agnostic, Hammerspace can abstract data from the infrastructure to provide self-service hybrid cloud data management. When combined with metadata management and machine learning, it allows the company to deliver its DaaS in which enterprises can make file data cloud-native. In turn, this allows them to automate only once and deploy anywhere.

    Hammerspace DaaS will let users protect and manage their data across hybrid cloud environments with global enterprise file data services. And with support for both GCP storage classes and on-premises storage, it will be easier than ever before for analytics teams, application owners and data operations to leverage their new service.

    What Will You Do With Your Data?

    With new data offerings from Fivetran, DataStax and Hammerspace, GCP is becoming the place to be for all data endeavors. But this is just the beginning; you can expect more announcements soon as Google continues to double down on more data tools to offer.

    For now, the only question left is this: What will your company do with these new data capabilities?

    — Ben Bloch

  • AI Delivering on the Business Analytics Promise

    AI Delivering on the Business Analytics Promise

    There’s a revolution underway in data analytics that could create a competitive advantage for companies that jump on it. C-suites, take note: AI-powered business analytics is finally delivering future-facing data points that CEOs need to guide their businesses to the sweet spot.

    The artificial intelligence (AI) technologies machine learning and deep learning are being used today to power through mountains of big data, much of which is previously inaccessible unstructured data such as text, video, customer reviews and customer feedback. Data scientists analyze the trends the machine learning uncovers, resulting in human-vetted, AI-derived, predictive analytics that provide strong decision support.

    Companies using AI-enabled business analytics are seeing positive results, yet most believe we’re just at the tip of the iceberg in utilizing AI’s capabilities in decision-making. Indeed, almost 70 percent of the CIOs who responded to a recent survey said they believe decisions made by machines will soon be more accurate than those made by humans.

    “Machine learning and deep learning have fundamentally changed enterprise data analytics,” said Tom Wilde, CEO of machine learning solution provider indico. He pointed out that a large majority of data is unstructured; in fact, according to some measures, as much as 80 percent of enterprise data today is unstructured. “In many cases, the unstructured data is where the real signal can be found,” and, as such, AI identification and categorization techniques can be used to develop analytics that could be used to improve a company’s product, predict future sales, identify cross-selling opportunities and reach new markets.

    “Machine learning supercharges business analytics and data science,” said Scott Zoldi, chief analytics officer at FICO. Machine learning algorithms explore relationships and data that even the best data domain expert may not think to explore, but, properly used, “Machine learning augments data scientists to make their discoveries seem superhuman. Data scientists that use machine learning will ultimately drive differentiated business value for their organizations.” In other words, AI is like using newly invented night-vision glasses for deep dark data—it should give a company a tactical advantage.

    What C-Suites Should Know about AI and Analytics

    AI and business analytics may seem like a panacea for what ails a business. But it’s not a plug-and-play technology—AI requires constant care and feeding to churn out the trends behind the data. Here are three things the C-suite must understand before adopting AI as a business driver:

    1. AI in its current form is unlikely to replace a human being. It has transformative power, yes, “but AI is going to further a lot of progress that’s already in place, not necessarily upend everything,” said AJ Abdallat, CEO of AI firm Beyond Limits, an AI company. AI will be used to create great things in years to come, but it’s not a magic bullet, it’s not infallible and it requires humans in great measure today. People should not be concerned about their jobs, nor should there be any expectation that AI might save money on personnel. These cultural concerns, along with a not uncommon resistance to change, are one of the barriers to entry.
    2. If your company punted on digital transformation, you’ll probably need an outside vendor to help you foray modestly into artificial intelligence. “The sooner you can modernize your digital infrastructure, the sooner you can bring AI in to address your [larger] business issues,” said Abdallat. In many ways, the ability to employ machine learning and AI may wind up being looked at as one of the main positive outcomes of digital transformation. Working in concert, these technologies are probably your best protection against sudden disruption by a digital-native startup with good ideas.
    3. “Too many CEOs delegate AI to their innovation teams, who conduct small-scale experiments. It’s important for CEOs to bring their own AI vision to their organizations,” said Gurjeet Singh, executive chairman and co-founder of Ayasdi. CEOs need to apply AI to the most pressing problems facing their businesses. “The world is going to change far faster than they can grasp. They need to set course immediately.”

    Adopt AI Soon

    One thing that has become clear as more companies join the AI fray, either as purveyors or consumers of the technology: The time for digital infrastructure improvements and adoption of AI is now. That said, the scope of enterprises that have already adopted AI is comparatively narrow. Companies currently using artificial intelligence to explore big data include the obvious: Amazon, Apple, Citi, DoCoMo, Facebook, Google, HSBC, IBM, Lockheed, Microsoft, Mt. Sinai, Tesla and Twitter. Many other companies are experimenting with AI in one facet of their business.

    How significant a trend will the use of AI by enterprises be? Is it at the scale of networking, the internet and cloud computing? “AI will have the same or greater impact on businesses for the next 20 years,” Wilde predicted.

    “Part of me wants to quash as much of the AI speculation as possible. The hyperbole,” Abdallat added. “But frankly, I do think that AI is potentially more transformative [than the internet, cloud etc.] … I know I’m a little biased in this, but we might be entering the era of the cognitive corporation, when man and machine at all levels of the enterprise will [cooperate] to solve problems, each learning from the other.”

    — Scot Finnie

  • Study: CIOs Want a Chief Data Officer

    Study: CIOs Want a Chief Data Officer

    Quick, what does CDO stand for? Before you say chief digital officer, think again. A new study points to the chief data officer as the more important of the two titles, finding 82 percent of CIOs believe there is a compelling case to hire a CDO in most organizations. The survey, commissioned by Experian Data Quality, was completed by 200 CIOs and 50 chief data officers in June and released in August.

    But while the report stresses the importance of the CDO to companies that want to become data-driven, it also reveals severe challenges for some CDOs, including very high expectations, insufficient resources and, despite the C-level title, a lack of authority to get the job done.

    The survey states that the core responsibility of the CDO is “to define a data-management strategy and translate it into tactical, [achievable] steps for the business.” To make data actionable, it needs to be accessible, accurate and standardized. Here’s a simplistic example of why that is important: Let’s say a data analyst needs to create a report that isolates customers based in New York City. If some of them are listed as being headquartered in NY, NY and others are listed as based New York, NY and still others are in the database as New York City, NY, that’s not standardized. The data analyst would need to realize the data is splintered and needs to be stitched together to get accurate results.

    Making your business data-driven is part of the potential promise of digital transformation. CDOs and their ilk face a daunting task. Transforming data into business intelligence is the alchemy of this age.

    Chart courtesy of Experian Data Quality

    The study identifies three main drivers for hiring a CDO: capitalizing on big data opportunities, creating a competitive advantage and reducing risk on data-driven projects. These roles go far beyond the data management and data governance toward enabling business innovation. Over the next few years, the CDOs surveyed predict that their top business challenges will concern data security, keeping pace with technology advances and managing increasing user expectations.

    Perhaps one of the biggest challenges some CDOs face is the inconsistency across companies of the nature of their role and their stature within the organization. Bottom line: They may not have the authority they need to succeed. The survey reveals, for example, that the average tenure of a CDO is only two years. Many CDOs report that despite their C-level title, they are not considered to be a part of the C-suite. Although 37 percent report directly to the CEO, a larger percentage reports to the CIO, CTO or some other part of IT. That can create some friction because it’s important that CDOs are able to bridge the gap between IT and lines of business—and empower the business.

    Tips for Structuring the CDO’s role

    In creating a new CDO position, the C-suite should be advised of these common challenges faced by CDOs identified by the survey:

    • Staying within budget
    • Keeping up with evolving technologies
    • Knowing how to interpret information
    • Finding/hiring people with data expertise
    • Managing ever-increasing user expectations
    • Balancing the needs of business users and IT
    • Having a lot of responsibility with little room for error
    • Getting pulled into the weeds
    • Maintaining data security

    You should consider extending these responsibilities to your CDO:

    • Having significant control of the management and use of company data
    • Playing an active role in digital transformation
    • The responsibility for enterprise-wide governance and utilization of information as an asset
    • Manager of a team of experienced data employees
    • The purview to implement different approaches to managing data as an asset
    • The purview to generate business intelligence that unlocks and supports business innovation
    • The authority to be recognized by the entire company is the person overseeing data strategy and process
    • Recommend: At least a dotted-line report to the CEO.

    — Scot Finnie

  • DataOps: Time to Bring DevOps Ideas to Big Data

    DataOps: Time to Bring DevOps Ideas to Big Data

    What can data scientists learn from DevOps? A lot, actually. By applying DevOps principles to the world of big data, you get DataOps, which streamlines data analytics.

    If you’re reading this site, you’re probably already familiar with DevOps. The DevOps philosophy emphasizes continuous collaboration between software developers (the “dev” people) and system admins (the operations, or “ops,” folks)—as well as everyone in between, such as software testers.

    It’s possible to apply similar ideas to the world of big data and data analytics. There, as in the world of software development and delivery, different teams traditionally have worked in isolation from one another. Those teams include storage admins, who keep data storage systems running; data analysts, who write and execute programs for interpreting large volumes of data; data quality specialists, who ensure data consistency; and data security teams, which protect data from attackers.

    As DevOps professionals know, keeping different teams isolated from one another is not efficient. It limits collaboration and maximizes confusion when problems occur.

    The data world, therefore, has much to gain by ending this isolation between teams and adopting a DataOps workflow. This will keep data operations flowing smoothly and provide continuous insight into the data that organizations collect.

    DataOps is not a brand new concept. People like Andy Palmer have been writing about it for a little while now. But the term and the idea have yet to gain nearly as wide a following as DevOps. (There is not yet even a Wikipedia entry for DataOps—and you know an idea is still in its infancy when no one has taken the time to write about it on Wikipedia.)

    Making DevOps Bigger

    Meanwhile, from the DevOps perspective, the growth of interest in DataOps is significant because it exemplifies the broad applicability of DevOps principles. So far, the DevOps conversation has focused on the relatively narrow context of software development and delivery. But it can extend much further, too. DataOps is a start.

    So, if you’re a data analyst, storage admin, data security professional or someone who works with data in any other capacity, it will be worth your while to borrow some ideas from your DevOps brethren. And if you work in software development, testing or delivery, it may be worth your time to educate your data friends on DevOps principles. It will make their workflows more efficient, while introducing DevOps to a wider audience.

    — Chris Tozzi

  • Things to Know When Planning for Cassandra Backup

    Things to Know When Planning for Cassandra Backup

    NoSQL databases such as Apache Cassandra, MongoDB, Amazon DynamoDB, Azure DocumentDB and others are being adopted by enterprises to meet the data requirements of next-generation applications. Specifically, Apache Cassandra database provides linear scalability, a flexible consistency model and shared-nothing architecture that enables enterprises to scale their database footprint according to performance needs.

    As mission-critical applications are being deployed on Cassandra databases, there are enterprise data management requirements also, such as backup and recovery. Today, there are various options available for backup and recovery of Cassandra. These options have pros and cons, which I will elaborate on in this blog post. It is important for a Cassandra database administrator (DBA) to understand the pros and cons of these options before choosing any option and validating products and solutions available to achieve enterprise goals for backup and recovery.

    Let me first point out the important features that DBAs should look for while planning the backup:

    1. Secondary Storage footprint: Efficient storage utilization is key for any backup and recovery design and plan. The backup data set may be much larger than primary setup depending on the retention period of the backup data. Hence, if storage management is not efficient, customer storage capital expenditure could increase multifold, defeating the purpose of data protection.
    2. RPO and RTO: Recovery point objective (RPO) is the maximum targeted period in which data might be lost from an IT service due to a major incident. Recovery time objective (RTO) is the targeted duration of time and a service level within which a business process must be restored after a disaster. Typically, enterprise requirements for these next-generation applications dictate very small RPOs and RTOs.
    3. Backup performance: This is the time spent in backing up the desired data set. Typically, the backup performance should be scalable enough to match the data change rate at the primary cluster. In addition, the time spent in taking the initial full backup of the primary cluster also matters, so periodical backup is not affected.

    Cassandra Backup Options

    Now let’s take a look at four options available for performing Cassandra backup:

    Snapshot-based Backup

    Cassandra provides native node-level snapshot option via the nodetool command. Snapshot creates a hard link to existing Cassandra data files and expects users to delete it after backing up the files. This way, despite changes in Cassandra data, the backup is performed from snapshot files.

    Pros:

    • Simpler to manage: It requires periodical snapshot commands on each of the nodes, triggering backup on snapshot files and removal of snapshot files after backup.

    Cons:

    • Not a true backup: Node-level snapshot doesn’t mean cluster-consistent backup and such a simplistic strategy will result in heavy repairs upon recovery.
    • Potential large RPO: The snapshot operation is not trivial, as it requires flushing of all in-memory data on disk and, hence, frequent invocation of snapshots will impact the production cluster’s performance dearly.
    • Large storage footprint due to data compaction at primary cluster: As part of data compaction, Cassandra merges multiple data files on a node into a new file and, hence, reduces multiple versions of a record to a single and latest version. Compaction not only helps in reduction of data at the primary cluster, it also improves the read performance as it reduces the number of files to look up for a record. As a side effect of compaction, it’s possible that same records will appear in multiple versions, as they are merged into a new compacted file. It isn’t a trivial task to filter out already captured records from a compacted file. Depending on various aspects such as workload type, compaction strategy or versioning interval, compaction may cause multifold more data to be backed up, causing a serious impact on CapEx planning.
    • Snapshot storage management at primary cluster: Cassandra expects users or admins to clean up snapshot files after they are no longer required. It requires a reliable job to clean up the snapshot folders; otherwise, they may affect the primary cluster’s availability due to lack of storage.
    • Siloed solution: Enterprises require multiplatform backup and recovery product and native tools are built specifically for a particular database and don’t scale for multiple databases.

    Incremental Backup

    Cassandra supports incremental backup functionality. If enabled, every time Cassandra flushes memory content as a new file, a hard link for the new file is created in the incremental backup folder. Once the backup of these files is done, the administrator is supposed to delete these hard links. The best part of incremental backup is that the hard link is created only for newly created file due to the flushing of in-memory data contents. Files created due to compaction are not hard-linked.

    Pros:

    • Better storage utilization: There are no duplicate records in the backup, as the compacted files are not backed up.
    • Point-in-time backup: Companies can achieve better RPO, as backing up from the incremental backup folder is a continuous process and, hence, versioning can be done more granularly.

    Cons:

    • Space management at primary cluster: With this option, companies need a reliable space management process on the primary cluster. The incremental backup folder must be emptied after being backed up. Failure to do so may cause serious space issue on the primary cluster.
    • Creates lots of small size file in backup: Since incremental backup facilitates a backup of new files created due to the flush of in-memory data from Cassandra, typically the file size is small. This leads to many small files, making file management and recovery not a trivial task (and fairly expensive compared to the RTO guarantees customers want and expect).

    Incremental Backup in Combination with the Snapshot

    Incremental backup in combination with snapshots is another rudimentary way to take Cassandra point-in-time backups. Periodically, data is backed up from the snapshot, while incremental backup-based files are used for point-in-time needs. This means periodically the incremental backup-based files, which are already covered via snapshot-based backup, must be deleted.

    Pros:

    • Fairly large-size backup files: Only the data between periodical snapshots are from the incremental backup.
    • Point in time: It provides a point-in-time backup and restores.

    Cons:

    • Space management in backup: After every snapshot, the incremental backed-up data needs to be cleaned up. Also, for backed-up files from the snapshot, a large backup storage footprint due to compaction at primary always is a concern.
    • Operationally very heavy: This method requires DBA admins to script solutions; this is not scalable for enterprise scale.

    Commit-Log Backup in Combination with the Snapshot

    This method is similar to incremental backup. Rather than backing up the newly added SSTables, DBA admins archive the commit-log. Periodical snapshot-based backup provides the bulk of backup data, while the archived commit-log is used for the point-in-time backup.

    Pros:

    • Point in time: Snapshot-based backups in the combination of commit-log archival provide a point-in-time backup option.

    Cons:

    • Space management: Again, the commit-log archival space management is admin’s responsibility; this method results in operational heavy solution (OpEx).
    • Restore complexity: The restore is a lot more complex, as part of restore will happen from the commit-log replay.
    • Storage overhead: Snapshot-based backup will have a storage overuse issue because of duplication of data due to compaction. This results in a very large use of secondary storage and, thus, a very high CapEx expenditure.

    Summary

    Overall, Cassandra provides multiple options for backup. Typically, snapshot-based backup in the combination of commit-log archiving/incremental backup is used if the point-in-time restore is a requirement. If point-in-time restore is not a requirement, simple snapshot-based backup is much easier to implement, though it is not a true backup and will result in heavy repairs.

    Most of these options require significant manual effort and are error prone. They work for staging or preproduction and test/dev environments that are smaller in scale and can sustain data loss and high RTO. However, for large enterprise customers and production deployments, there are key requirements such as data reduction via novel compression/de-duplication techniques, various failure handling and faster recovery with desired consistency.

    For mission-critical, always-on applications where minimizing data loss and operational resiliency are important factors,  administrators should consider leveraging next-generation data protection products from companies that specialize in this area.

  • 4 Tech Imperatives in 2016 to Prepare for Tomorrow

    4 Tech Imperatives in 2016 to Prepare for Tomorrow

    Around the start of every new year come a bevy of predictions about what’s coming (or what someone, or some vendor, or some analyst hopes is coming) in the year ahead. Rather than do that, based on recent conversations with many CIOs, I’ve collected what I think are imperative to-do’s for CIOs and IT leaders to ensure their enterprises can innovate in the years ahead.

    If you don’t already have these in your crosshairs, here are four tech imperatives for 2016:

    Action Item 1: While IoT gets the buzz, groom those mobile app backlogs
    The world has gone mobile. That’s no secret. Mobile analytics firm Flurry found that business productivity apps grew 125 percent in 2015, because most every employee today is armed with a mobile device and more comfortable than ever working on mobile devices. And mobile enterprise investments will continue.

    Currently, however, most enterprises still have considerable mobile backlogs. In the year ahead, it’s imperative they address those backlogs. This investment in mobile apps will continue, so enterprises must clear through the clutter and groom their mobile app backlogs. This year you’ll hear more and more about the Internet of Things, or IoT. (I know, I know, how can we expect to hear more about IoT in ’16 than we did in ’15? Trust me, we will.) Companies first must focus on enterprise and productivity-enhancing apps, while planning and preparing systems and plans for the expansion to IoT.

    In the next few years, IoT will include access to wearables and their data, enterprise office equipment, office buildings, cars and virtually anything else that can be tethered to the Internet. But, for now, the focus for most enterprises must be on clearing their mobile app backlog and grooming their mobile app pipelines, and then setting the stage for the architecture that will be required to support their eventual IoT efforts.

    Action Item 2: Get a tight focus on micromoments
    These days, employees are making every second of the business day count, even when they’re away from their desk. Most employees aren’t sitting, working in front of computer monitors; today, most everyone is on the move and spending more time working on small-screen tablets and smartphones. This has changed how we interact with each other—and our applications.

    Today, workers have short exchanges in text or on social media, short discussions on the phone, and short interactions with their apps. Rather than settling in and computing at a desktop PC, we are computing on our mobile devices while we walk office to office, or are waiting in a conference room for a meeting to start. We are computing, more than ever, in “micromoments.” Guiding developers to develop strategically for mobile is still an imperative for CIOs this year.

    What do I mean? Development teams must ensure their apps are built so users can engage in the context of their moment, based on their location, the task they are undertaking or perhaps the time of the day. Thus, apps must be developed so that data can be entered quickly and easily and consumed readily. And the micromoments app experience must be congruent with those on notebooks and tablets, when people typically are settling in for longer stretches of work and concentration.

    Action Item 3: If enterprises haven’t already, they need to develop their IoT brokers and management platform strategy
    All of those IoT devices we mentioned above will need to be managed. This will require cloud-based platforms that will:

    • look at the data,
    • protect the traffic from being attacked,
    • accept legitimate traffic,
    • dispatch it to where it needs to go,
    • save the appropriate data for long-term analytics,
    • send other data for real-time analytics and response, and
    • enforce security and privacy policies.

    Managing the security, storage and traffic will be a strategic effort in the years ahead, and the groundwork needs to be set sooner rather than later.

    This year is the time to build the strategy for your enterprise’s IoT platform strategy, so the IoT data can be managed properly and management can scale as the use of IoT grows. Think this won’t apply to you? Think again. Gartner predicts that there will be more than 25 billion IoT devices by 2020, up from 4.9 billion today—and a good portion of them will be enterprise IoT devices.

    Action Item 4: Create the data-driven culture
    Big data analytics has touched nearly every aspect of business, from health care, with data-driven medicine saving hundreds of millions annually, to electric utilities managing the grid more intelligently, to helping marketing teams understand and pinpoint their target audiences with more intelligence and precision than ever before possible.

    What’s often forgotten about big data analytics is that success here is about much more than databases, storage, dashboards and data scientists. It’s as much about creating a data-driven culture in which decision-makers in the organization are motivated to make the most of the data and analytics to move the organization forward in the best way possible. Strategically, CIOs want to create this culture, make as much data as possible available to employees, and deliver the data to them in the context of their roles. They want to develop platforms that democratize the data to as many people as possible so they can seek the insight they need.

    Enterprises today already have a lot of the critical data in their software, ERP systems, employees and customers. In the years ahead, IoT devices will create additional tsunamis of data to be analyzed and acted upon for strategic advantage in the earlier years and to simply maintain the status quo in the years beyond. That’s why it’s so important to get this on the strategic radar today.