devopsy.kinsta.cloud

Tag: big data

  • AIOps: How to Solve Performance Issues in Complex Data Pipelines

    AIOps: How to Solve Performance Issues in Complex Data Pipelines

    AIOps takes center stage to topple barriers as modern data application growth accelerates

    There seems to be a chasm between DevOps and big data teams in the enterprise, and the bridge over that chasm is long and treacherous.

    In fact, data analysts who are charged with managing and making sense of the growing volumes of data in most organizations traditionally have been excluded from the strategy-making and operations functions of DevOps personnel. Yet, data analysts are critical to the success of software development and at every stage of the software delivery pipeline.

    DevOps Under Siege as Modern Data Applications Proliferate

    Let’s take a deeper look. We know that DevOps personnel are on the front lines in ensuring that all systems, both new and existing, perform as expected in a production environment. But, even today, DevOps teams struggle in dealing with technical challenges when faced with complex data pipelines and the underlying big data stack that supports them, including:

    • Root cause analysis — determining if an application issue is due to application code, config errors or resource contention.
    • Correcting inefficiency in code.
    • Improperly configured auto-scale clusters.
    • Lack of actionable insight into how clusters were tuned at both a cluster and job level.
    • Difficult access to support engineers and promotion of DevOps collaboration.

    And these are just the tip of the iceberg.

    The pressure is on. On average, companies with data-driven environments garner up to 50 percent higher market value than their competitors, according to IDC, and big data is a contributing factor. Thus, big data implementations have become a top priority for businesses of all sizes, and on multiple fronts: revenue acceleration, creation of new products and contributing to better, quicker decision-making. But the unfortunate truth is that, as big data projects move from pilot to production, they invariably encounter serious performance problems that frustrate both data analysts and IT operations.

    What’s more, DevOps and big data personnel face a mutual problem: lack of centralized, top-down visibility into their end to end data pipelines and applications. And, as a result, they face challenges in pinpointing the source of problems—namely, that problems can become visible anywhere in the stack, but their root cause could be elsewhere in in the stack. And, these problems result from root causes as diverse as bad code, inefficient data partitioning, mismatched system configuration settings, best-guess resource allocation and infrastructure issues. Furthermore, a big data stack running multiple applications faces resource contention and prioritization issues that can reduce both individual application performance and overall system performance.  And streaming applications create their own challenges;: for example, if a streaming application starts to lag in processing data in real-time, diagnosing the root cause takes considerable time and effort.

    The industry is faced with a reality where the problems are outrunning the solutions. In the case at hand, organizations have relied on brute force and expensive custom solutions for making analytics-based recommendations more user-friendly to improve individual application and full-stack performance.

    AIOps Takes Aim

    AIOps, or artificial intelligence for IT operations, is gaining steam in the enterprise, with many new use cases appearing as data volumes grow and systems become more complicated. Gartner expects AI to pervade every aspect of managing IT, including application performance management (APM), IT infrastructure monitoring, cloud platform monitoring, performance management, capacity management, the delivery tool chain (CI/CD) and IT service management.

    In fact, AIOps is on a tear: Just three years ago, it was only a concept. Today, Gartner predicts that by 2019, 25 percent of global enterprises will have implemented an AIOps platform to support two or more major IT operations.

    But even as it continues to evolve, AIOps today can deliver automated recommendations to solve expensive, challenging and time-consuming problems with big data deployments and across entire complex stacks. Without AIOps, there would be simply too many variables to correlate manually.

    AIOps makes APM for modern data application possible. It does so by helping to eliminate painstaking, trial-and-error configuration, troubleshooting and remediation workflows that have long plagued big data operations teams. And, for the first time ever, dataOps and DevOps teams can troubleshoot, support SLAs, remediate issues and manage large-scale deployments in an agile and dependable manner.

    It’s a new environment for the modern data center. As part of an APM solution, AIOps is hard at work helping automate and guarantee the performance of big data applications and addressing the primary challenges that DevOps faces with the unrestrained growth of big data.

    It could still be a bumpy ride—organizations of all kinds face unprecedented increases in data volumes. But now, working together, DevOps and AIOps are better-equipped to contend with the challenges.

    — Bala Venkatrao

  • DevOps Chat: AIOps and APM with Unravel’s Kunal Agarwal

    DevOps Chat: AIOps and APM with Unravel’s Kunal Agarwal

    Unravel is making a name in the AIOps, APM and big data markets. In this DevOps Chat, we speak with Unravel CEO Kunal Agarwal about what makes Unravel a must-have for its customers. We also take a sneak peek ahead to what the near and not so near future holds for big data, AIOps and the tech scene in general.

    As usual, the streaming audio is immediately below, followed by the transcript of our conversation.

    Transcript

    Alan Shimel: Hey, everyone, this is Alan Shimel, devopsy.kinsta.cloud, and you’re listening to another DevOps Chat. Today’s DevOps Chat features Kunal Agarwal, CEO of Unravel or Unravel Data. Kunal, welcome to DevOps Chat.

    Kunal Agarwal: Thank you, Alan. Thank you so much for having me here.

    Shimel: Excellent. It’s my pleasure to have you here. So, Kunal, let’s get right to it. “Unravel” is the name of the company, correct? Not “Unravel Data.”

    Agarwal: “Unravel,” yes.

    Shimel: But yet Unravel is about data to a certain degree. I don’t know if everyone in our audience is familiar with Unravel. Why don’t we start there?

    Agarwal: Sure. We are unraveling the mysteries of big data, Alan. That’s why the name of the company.

    Shimel: Excellent.

    Agarwal: Everybody’s jumping on to big data apps, and what I mean by that is you may be doing something with machine learning, artificial intelligence, IoT, even big-scale data warehousing. And there’s just so many problems that can happen, running these applications at scale, that people just have not seen before. And this is a problem because of all these new systems and technologies that are coming about, such as Spark, Kafka, Hadoop, Cassandra, HBase, many, many more. And the complexity’s all about “Hey, how do I figure out why’s my application not performing well or if it’s failing?” And it’s usually a multi-tenant environment that these big data applications run on, which means there may be hundreds of users, thousands of applications, that you are creating every day and running them.

    And figuring it all out and making sure that they run well is a black art, so there’s a lot of trial-and-error that goes on into fixing these problems, finding the root cause of these problems. And that’s what we are automating, meaning Unravel, after it’s installed in your big data cluster, will be constantly monitoring and managing everything. And if there is an issue, Unravel will be proactive about it, tell you why that issue happened, what caused it, and then even take a step further where it would resolve those issues for you so that you can spend time creating all these amazing big data applications, rather than focusing on fixing all these systems- and application-level errors.

    Shimel: Mm-hmm. So, you know, I mean, there’s a lot there that we’re gonna dive into, but before we do, Kunal, just give our audience – ’cause I always find this fascinating. People wanna know “How do I become a CEO? How does one get to be the CEO?” Give us a little insight into your own personal journey on how you’ve come to be CEO at Unravel.

    Agarwal: CEO is “chief everything officer,” when you first start off. You gotta sweep the floors and you gotta raise the funds and you’ve gotta grow a team. There’s so many things a CEO’s role requires to do, especially in a young company, as you’re growing it. Today, now that we’ve grown to about 75, 80 people in total, across 3 different countries – we actually have offices here in the United States, one in London, and one in India – my job is becoming more about recruiting, more about getting the team together.

    And I think that’s, personally, for me, if that is taken care of, the entire company just hums and does well. Everything else is secondary for me at this time, but I do spend a lot of time doing marketing and sales and finance and product. So it really depends on the day, Alan, of what my job requires me to do.

    Shimel: Amen. You know, I’ve founded or cofounded three or four companies myself and even just running MediaOps, which is devopsy.kinsta.cloud and Security Boulevard, you are chief cook and bottle washer when you’re – you’re chief everything officer, as you said.

    Agarwal: Right.

    Shimel: You know, looking in your background, you’re a double-major graduate – electrical engineering, computer science master’s degree from Duke. I mean, you’re well positioned to be a CEO, people would say, looking at the background, but it’s – you know, you never know what’s coming out next as the CEO. If I had to ask you, Kunal, what’s kind of been the biggest surprise or biggest kind of curveball that you’ve seen in your time at Unravel that you think, “Geez, I didn’t think I’d be doing this or I didn’t think this was part of the job,” what would you say?

    Agarwal: Alan, actually, going to school and getting all those double-degrees does not help you at all becoming CEO.

    Shimel: You get it. I mean, you’re talking to someone who went to law school. I know all about it.

    Agarwal: Right?

    Shimel: Yeah.

    Agarwal: I wish there was a school for CEO, but you know what? The only school for being CEO and leading a company is actually leading the company because there’s so many things that you learn along the way. One thing, because we deal in enterprise software, there’s so many complexities to making a deal happen with a big company.

    Shimel: Mm-hmm.

    Agarwal: And I think that was one of the things that took me a while to grasp and make sure that we’re doing it right. And I mean all the boring stuff, like the contractual paperwork, all the legalities, going through info-sec requirements. All the operational stuff around making a deal happen. You know, when you’re in college and you’re starting to think about a startup, you think about the cool products you’ll make. You’ll think about the impact your product will have in the market and how so easily everybody will flock and come to you and get your product. [Chuckles] But it’s not that simple.

    Shimel: I wish it was.

    Agarwal: and it takes 9 months to 12 months to close some of these deals, so I think, more than the “surprise” factor, preparing yourself and making sure that you can drive efficiencies in all these different processes. I think that’s been one of the key challenges for us.

    Shimel: Absolutely. Absolutely. So, Kunal, if I’m a Silicon Valley VC, I’m looking at a company like Unravel, man, you’re playing in all the right places. Big data is certainly such a huge – I think, when we talk about things like machine learning and AIOps, really, they’re almost the children of big data – you couldn’t have those things without the big data, first of all – so you’re in the right place, obviously. The right time. This is a very happening kind of space. What is Unravel doing that is so – what’s the special sauce? What’s unique about it?

    Agarwal: Yeah, Alan. So, when we created this company – and by “we,” I mean myself and my cofounder, Dr. Shivnath Babu, who’s actually a professor of computer science at Duke University – we both met at Duke – and the problem we were trying to resolve is one of the most fundamental but also probably the most complex and challenging problem to solve, which is “How does one ensure that all of these big data technologies actually work and deliver on their promise?” And, you know, our approach to the problem was always “Hey, there will be a shortage of experts in the market. Even if there are experts, these problems will be requiring a machine to solve them on time and efficiently.”

    So we took the long path of actually creating not just a monitoring tool but a AIOps tool, if you may, which can decipher the problem, connect the dots, and resolve it for you. So it actually took us about three-and-a-half years to build this product or at least the first version of this product. And we moved west to start working with some companies – I lived in New York; he was in North Carolina – and we gave up our cushy jobs, moved to Silicon Valley, because you wanted to work with companies who were using big data tech at scale.

    And some of these companies, we still have advisors in, from Zynga, Rocket Fuel, Twitter, LinkedIn, and they really helped us understand the day-to-day issues that come up when you’re running these technologies at scale. And we took a lot of those learnings into our first product. So we were always about solving a core need that teams in big data tech would definitely need. And, you know, they always say, “Hindsight is 20/20,” right?

    Shimel: Yeah.

    Agarwal: But we always put a big bet that, “Hey, these technologies – Hadoop, Spark, Kafka, et cetera – will become mainstream and companies will start using this a lot more.” So what LinkedIn was doing five years ago, the biggest banks would start doing it now, and that’s where we are in the journey today.

    Shimel: Absolutely. And, you know, it’s funny, full circle, talking about that journey and what you’re doing. That’s how you become a CEO. It’s about being willing to take the risk, giving up those cushy jobs. Right? To do something that you really believe in. And I think that’s what separates CEOs from people who kinda look and say, “Boy, I’d like to do that,” but they don’t actually do it.

    Let’s talk about the market. We’re coming to the end of the year. You’re participating in our Predict 2019 panel, which will be December 18th event. You’re gonna be on the AIOps panel, looking ahead. Why don’t you give us a sneak peek? What’s kind of your take on what 2019 holds for AIOps and big data and so forth, Kunal?

    Agarwal: Well, yeah, Alan. So AIOps is an evolution of IT Ops or APM or DevOps, where you’re using machine learning to, again, understand all of the data – and this could be monitoring data, instrumentation data – that all these systems and applications are generating to draw insights from.

    Now, funnily enough, this is what our customers do with their data set, right? So with some of the biggest banks, some of the biggest insurance companies, some of the biggest health care companies, all Fortune 500. And our methodology of going about monitoring for big data stack was always the same, which is “Why aren’t we using big data technology on our data set, which is machine-generated data, to glean those insights and drive some prediction and actually make things simpler and more accurate?”

    So AIOps is also one of the use cases of artificial intelligence and machine learning. Now, just as I’ve said the industry’s evolving, what we predict to see is the evolution of AIOps, so Unravel, for example, is already on one of the end-of-the-spectrum places, where we actually make decision on behalf of the operations and the development teams, whether it is re-evaluating an application, whether it is making sure that your SLAs for your applications are being met, by moving low-priority jobs to a sub-optimized queue and high-priority jobs to a priority queue or killing bad applications in the cluster, all on your behalf, to make it more proactive.

    But what I predict is there’s gonna be an evolution in just the incumbent APM companies as well, where it’s not just about giving people a monitoring screen or even a native monitoring screen and saying, “Hey, check out all the information we know about an application,” but then I leave it up to you to decide and understand what’s amiss, what’s wrong, and how do you fix it. But I feel we’re already now in the correlation and the insights phase, which is now I can start to connect the dots, with these new software technologies that are out there. But, in 2019, we’ll start to also see some of these companies take the leap to be able to do automated root cause analysis, as well as guided remedies or automatic solutions, to get out of these problems as well.

    This industry’s evolving at a very fast pace, so I’m super excited. And it’s all about making sure that some of the biggest applications in the world, which millions of people actually rely on, all work like a well-oiled machine. And it’s all about guaranteeing reliability, which, at the end of the day, improves confidence of the customers using those products, which further fuels more innovation and more growth. So it’s an amazing cycle and AIOps is gonna be a very, very key part of it.

    Shimel: No doubt about it. No doubt about it. You know, in many ways, I look back on 30 years in IT and it really is a golden time to be alive, if you’re a developer or ops person or just in IT in general. Right? There’s the possibilities and the tools, and what’s in front of you is so exhilarating, right? I mean, I think back, if I was 20 years old now, 22 years old, coming into the –

    Agarwal: So you mean like 10 years ago, Alan?

    Shimel: Yeah, something like that. About 10.

    Agarwal: Okay.

    Shimel: About 10. And, I mean, just unlimited opportunity. I mean, that’s kinda almost the way I look at it. And, of course, it’s companies like Unravel, right, that are kinda letting these opportunities take shape and packaging them for people to run with. So great stuff. Specifically, Unravel – we only have a couple moments here left, but – so, Kunal, anything on the horizon with Unravel you wanna share with our audience?

    Agarwal: Yeah, Alan, absolutely. So, even in big data technologies, there’s so many to cover. We started our life off supporting Spark, Kafka, the entire Hadoop stack, some NoSQL systems, like HBase and Cassandra, and some newer NPP platforms. In 2019, we’re gonna further expand some of those technologies and systems that we cover, especially as we start to see more and more big data workloads move to the cloud.

    We’re also seeing a lot of our companies demanding a hybrid real estate management from the cliched “one pane of glass,” that “Can I see everything that’s in my big data environment, whether that’s on premise and on the cloud, together?” So Unravel is launching that support as well.

    And, last but not least, we are also creating a solution, from our product, which is directed towards helping companies move from their on-premise environments to cloud environment. So actually using Unravel as a migration project management tool, to help you assess, migrate, and then validate your migration. And, on the assess side, figuring out “What system should I get? Which instance type is best for my workloads? How can I keep under cost? What is the cost of running my big data workloads on the cloud?” to then validating and making sure that your SLAs are met, that you’re under cost and under time, and everything is flowing like a well-oiled machine.

    The way we look at it, Alan, again, the way to look at our road map and anything on the horizon is we are here to help our customers in the entire journey of big data tech, so, whatever they’re deploying today and they wanna use tomorrow, Unravel will be supporting that. And this is a natural evolution where companies are going from trying something out to getting something in production, first starting on premise and then going to the cloud, and those are all the shifts and transition points that we wanna hold the hands of the customers that we work with.

    Shimel: Excellent. Excellent. Well, Kunal, thanks for spending a few moments with us. As I told you, the time goes quickly.

    Agarwal: Yeah.

    Shimel: We’re way over time, but it’s okay. Hopefully, people have stayed on till now to listen. I wanna wish you lots of continued success with Unravel. We’ll be checking in on it. Again, a reminder: Kunal will be on our December 18th Predict 2019 virtual event and Kunal will be participating in the CEO panel, taking a look ahead at AIOps in 2019. So join us. You can register for that at predict2019.com.

    Agarwal: Thanks, Alan.

    Shimel: Thank you. Hello to all of our friends at Unravel. And that’s gonna make a wrap on this episode of DevOps Chat, everyone. This is Alan Shimel for DevOps Chat, devopsy.kinsta.cloud. Have a great day.

    — Alan Shimel

  • 5 Ways KPI Analysis Helps DevOps

    5 Ways KPI Analysis Helps DevOps

    Four Vs drive big data solutions: volume, velocity, variety and veracity. Volume and velocity are technical considerations usually receiving a healthy dose of attention among architects and coders. However, variety and veracity often determine success or failure, and they tend to sneak up on teams who haven’t fully considered them. Key performance indicator, or KPI, analysis can head off variety and veracity problems before they escalate.

    When some folks say “KPI analysis,” they mean KPI monitoring and evaluation—the business intelligence perspective. I’m talking about a DevOps analysis discovering which KPIs should be monitored and what data is needed. One step in this analysis traces each desired KPI back through necessary transforms to its required data sources, often termed as facts. Such an analysis clarifies variety and quickly establishes a path to veracity.

    From a strategy perspective, KPI analysis can also mitigate the symptoms of “big bad data” that devour an organization, consuming people and resources rapidly. Here are five ways KPI analysis helps DevOps teams get a handle on big data.

    1. It prevents over-measuring. If you don’t know what you’re looking for, more data probably won’t help you find it. Teams experiencing FOMO (fear of missing out) expand their data gathering, hoping more variety uncovers something interesting. Once your KPIs have been discovered and fully analyzed, stop exploring and start exploiting. Stick to monitoring KPIs directly driving the most impact for your business.
    2. It minimizes bloating. In any computational system, the most expensive operation is moving data. Simply moving stuff around eats processor, network and storage bandwidth. Even with cost-effective storage available, unnecessary data means a bloated infrastructure and less efficiency. KPI analysis identifies a minimum spanning set of data sources. A workload-optimized solution then concentrates on efficient algorithm implementations, yielding results aligned with your KPIs.
    3. It stops wallowing. Allowing people too close to raw big data is extremely dangerous. Off-the-shelf analytics tools often carry a lot of non-KPI variables with them. Cloud-based martech tools are some of the biggest offenders; they are essentially showing off how many ways they can expose their underlying data. Avoiding the temptation to wallow through intriguing but ultimately irrelevant data saves precious brain cells. Customize reports and optimize dashboards around your KPIs.
    4. It de-obfuscates conversations. Nope, not talking about JavaScript here. We want everyone in the organization talking about the same KPIs and only those KPIs. You’ve seen those presentations where someone tries to dazzle an audience with mostly meaningless charts and graphs. Obfuscating your KPIs underneath a pile of other observations wastes time and risks defeating the purpose. Trust your KPIs and focus all organizational conversation on those.
    5. It thwarts non-causality. This is one place where data scientists and analysts really earn their money. Part of a solid KPI analysis determines causality—if your data sources change by some amount, the KPI needle responds by a quantified amount. Understanding those relationships is critical in avoiding false conclusions from data. If your KPI isn’t moving despite changes at the data sources, or your KPI is changing and the obvious data sources aren’t, step back and look at causality and transforms again.

    If you’re seeing these symptoms developing in your organization, it might be a sign automation was rushed in before KPIs were completely analyzed. At that point the toughest symptom to deal with is bloating. You may be stuck with a sized and funded infrastructure, and you really shouldn’t throw out good data already captured. Ongoing KPI analysis can lead your team to optimizations moving forward.

    Ideally, KPI analysis is performed upfront and these symptoms never develop. Realistically, complex systems have cycles of learning before their behavior is fully understood. Even a good upfront analysis may benefit from a revisit as trends start becoming apparent. When KPIs are rolled up periodically, you’ll need a minimum of six periods of results before expected causality is confirmed, maybe more. If that period is monthly, you’ll need ample patience.

    You’re spending too much on a big data solution to afford a big bad data outcome. KPI analysis is an important step, one DevOps teams with both technical and business expertise can excel at with some practice.

    — Don Dingee

  • DevOps Chat: Instrumentation with Influx Data’s Evan Kaplan

    DevOps Chat: Instrumentation with Influx Data’s Evan Kaplan

    Evan Kaplan, CEO of Influx Data, is someone who knows his way around the technology world. In a lifetime of experience that has seen him found and manage several successful companies from startups to public entities, Evan has had a great track record of success.

    Influx Data is yet another success for Kaplan. Building on the open source InfluxDB, the company boasts more than 200,000 users of Influx today.

    In this DevOps Chat, Evan and I discuss systems instrumentation today and what big data and velocity means for developers and everyone up the stack. It is a very different world from just 15 or 20 years ago.

    As usual, the streaming audio is immediately below, followed by the transcript of our conversation.

    Transcript

    Alan Shimel: Hey, everyone, it’s Alan Shimel, devopsy.kinsta.cloud, and you’re listening to another DevOps Chat. Happy to be joined in this episode of DevOps Chat by, actually, someone who grew up in my neighborhood. We knew each other – well, we didn’t know each other way back when, but we’ve come from a very small area of concentrated – lot of people, you know, we know from that area, but, anyway, Evan Kaplan, CEO of InfluxData and of North Woodmere. Evan, welcome.

    Evan Kaplan: [Laughs] Thanks, Alan. Thanks for having me. Look forward to _____ –

    Shimel: My pleasure. So, Evan, it’s been a long, strange trip from North Woodmere. Why don’t you give our audience a little bit of your background and your journey? And  then we’ll jump into –

    Kaplan: Yeah. Sure. Sure. Very long for me. If those of you don’t know, that’s a suburb of New York City, right near Kennedy Airport, that we both grew up. My strange trip took me to Seattle, where I finished up college and worked as a climbing guide for years. And then worked in aerospace and started a company, the first company to develop SSL-based VPNs, called “Aventail,” back in the mid-90s. _____ through the bubble. Ran a public company called “iPass,” and, about two-and-a-half years ago, met Paul Dix, the founder of Influx. And Paul and I really connected about his vision and what he was doing, and we partnered together to start building Influx from 25 employees back then to north of 100 now. So growing fast.

    Shimel: Good stuff. Good stuff. And, Evan, I’m gonna assume most of our audience has at least heard of InfluxData, but, for those who have or even those who haven’t, just to set the level-set, tell us about what InfluxData does.

    Kaplan: Yeah, so InfluxData is an open-source time-series-based data platform. It’s the leading time-series platform out there. Time-series is the fastest growing market in the database, data category. And so Paul, our founder, spun up Influx back in 20 – I think it was late 2013. It received a lot of accolades on GitHub. It was a very popular project and it has grown dramatically since then, from roughly – I don’t know – 1,000 sites running it to, today, we’re approaching 200,000 sites running it.

    Shimel: Wow.

    Kaplan: And, you know, most of the large Global 2000 is running it somewhere, so.

    Shimel: That’s really impressive, man. That’s something to be proud of. So, Evan, we could probably spend our whole 20 minutes talking generally about Influx DB and what’s happening, but I’m wanted to kinda dive in a little bit and talk specifically today around instrumentation in modern software installs, in the modern software factory. This whole term “modern” has become sort of a catchall buzzword, dog whistle, if you will, for when we talk about  things like Agile and DevOps and serverless and microservices and all of these ways that we’re seeing software being written, managed, and deployed today. So talk to us a little bit about it, if you will, Evan. Where’s your take on this?

    Kaplan: Yeah, so I think we’re – we grew up in the industry, going back a couple of decades ago, the common myth for monitoring or instrumentation of systems was you would buy a package from HP OpenView or CA or a mainframe management package, and you’d buy a monitoring-for-a-service package. They were expensive and they were largely horizontal across the stack, and you’d implement those. And what’s happened over – certainly over the last ten years, during the DevOps revolution, is that stuff has fractured and you’ve seen companies, like Splunk and Elastic and a variety of other folks, arise to handle the modern workloads in an open-source sort of fashion. And now, if you take a look a look at DevOps world, most of solutions are dominated by open-source kinds of models.

    If you look at state of the art now, that people are really paying attention to, the revolution that brought you to logs and log views of stuff and the consolidation of logs has moved much closer to metrics and events. And what I mean by “metrics and events,” the instrumentation of real time of these complex systems. And they’re not just software systems that are DevOps things, so – those are obvious, but they’re also hardware systems; they’re IoT things. And so what’s happening is people wanna see these complex systems and they wanna interact with them in real time because the only way to characterize these systems is to watch them behave in real time. And “observability,” as you know, Alan, has become sort of a keyword here. And observability generally means real time and it means metrics and events; it doesn’t mean verbose logs.

    And so what we’re trying to do is help developers, specifically and, up the chain, architects and CTOs, build instrumentation or telemetry natively into these complex systems, so they can view these things in real time. As you can guess, the data loads with this are absurdly high. You’re talking about hundreds of millions of points a second in some of these systems and you’re talking about to visualize those in near real time, you know, control loops in near real time. And you’re talking about this sort of next generation of, lack of a better way to say it, system instrumentation and management.

    Shimel: Sure. Evan, if I could stop you there for a sec, you know what I find interesting? We’re talking about building this instrumentation and to give people insight and who are we giving insight? Well, you mentioned developers and managers and architects and CTOs, et cetera, but you didn’t mention ops. And is that –

    Kaplan: Yeah. Yeah.

    Shimel: Isn’t that a little bit of, you know – I don’t know if it’s Freudian or if it’s just insight into what today’s world looks like for these things.

    Kaplan: You know, that discrete ops, right? That discrete ops role is changing, right, into what you obviously cover, which is this DevOps. And so our view of how our – uptake of our platform into large corporations, companies like IBM and Cisco and SAP and Tesla and PayPal, has been from developer or DevOps people up. So we tend to think of “Make it super easy for developers, right? Get their time to value down to really, really minimal and then it will rise up and be taken as a complete operational system.” So we tend to think of it from developers up. Not to say the operations guys aren’t relevant – they’re super relevant – it’s just, when we enter into a large opportunity or business or that sort of stuff, it’s usually coming from a developer who figures out he can instrument his code, his service, his offering relatively quickly in Influx.

    Shimel: Got it. And I’ll tell you something I – and it goes back to what you mentioned earlier – the sheer volume and the acceleration of data, right, the – so, no, it’s how much data and the speed that it comes at us. With automation and everything that that brings, it easily outstrips the ability of a human, a lot of times, to stay on top of it, so you almost need to build automation, machine learning, AI into these types of instrumentations, that you bring what the human really needs to see and touch or make a decision on to his attention ’cause so much of it is just whizzing by right now.

    In the last week or two, Evan, I’ve probably had a couple of discussions – trying to think of a friend of mine who – CEO at ScienceLogic and another company called “Ravel Systems”; some friends of mine in security just launched something called “DisruptOps.” And they’re all talking about this term, “AIOps.”

    Kaplan: Yeah. Yeah.

    Shimel: And I’m wondering is this something relevant to what you’re talking about, with instrumentation? Where does that fit in? How do we account for just the big data nature of this?

    Kaplan: Yeah, I think, first of all, absolutely. So, if you think about – so the journey. And let’s call it the end state of all of this is true autonomy, which is very rare and, in fact, almost never the case, right? But, if end goal state is true autonomy, what’s that journey look like? It looks like, first and always and most importantly – and, by the way, the stuff that we skipped a lot, when we were in the early stages of our career – it is, in fact, instrumentation, is telemetry. Telemetry always was an afterthought before whether shipping a product or a service or that sort of stuff. Now the ability to drive that cost of instrumentation down so low that the ability to see what’s going on and create the data – not just see as in “human” see but create the data – has changed dramatically.

    So, if you follow that journey, you go from instrumentation; you go to observability; you go to automation in the form of “I’ve observed these sorts of things, whether it’s through AI capability or not. I built control loops because I’ve observed them, so I’ve built control loops to keep it in a relative stasis. Right? To autonomy.” All of that – the last two steps, right, even the last three steps – involves some form of machine processing of learning, whether it’s machine learning or AI or call it, but all of that gets increasingly sophisticated. But it means taking in, as you sort of articulated, you know, millions, hundreds of millions – in some case, billions – of points per second, right?

    Shimel: Yep.

    Kaplan: Think about it. Now that’s just completely – you know, what a different world, than doing server management with your CA Unicenter product 20 years ago.

    Shimel: [Laughs] To say the least.

    Kaplan: Right?

    Shimel: Yeah, I hear ya. I mean, it really is. If we transported – you know, if we could do that – if we could transport someone from not that long ago, Evan, an hour or _____ _____ –

    Kaplan: No. No.

    Shimel: From 1997, ’98, right? I was helping an early ASP, Interliant. We were building – opening data centers, managing Lotus Notes and PeopleSoft and stuff like that.

    Kaplan: Right.

    Shimel: If I could take a NOC guy from there and show him today’s velocity and the fact that we actually do have telemetry and we actually are building tools that allow us to manage at that level, it would be like taking someone from colonial times and showing them highways with cars. I mean, the supports –

    Kaplan: But, Alan, these solutions are still sitting out there. They’re a huge _____ _____ _____ –

    Shimel: No – well, that’s the scary thing. Yeah, I mentioned my friend, ScienceLogic, my friend Dave Link. He was saying that’s his – you know, and they’ve done very, very well, and his thing was just replacing the Tivolis and the Unicenters and that generation, if you will, of network management, of systems management.

    Kaplan: If I could tell you what, back in the late ’90s, is I was building an ASP also, based on the Aventail, the SSL VPNs, what we were standing on Netcool, back in the day, and things like that, _____ _____ –

    Shimel: And Netcool was a new program, right?

    Kaplan: Yeah.

    Shimel: Before IBM bought it, right? It was one of the cool, new kids at the time.

    Kaplan: Exactly. If I could tell you what somebody could take with Influx today and do, relative to what Netcool did back in the day, and the difference, right – first of all, they could do it for nothing. Forget paid licenses. And the amount of scale, the amount of data collection, and the ease of sort of visualization has dramatically – and all those _____ _____ –

    Shimel: It’s phenomenal. Phenomenal, man. It’s just crazy. So, Evan, let’s bring this back to Influx. How does all of this now manifest itself with your business today?

    Kaplan: Yeah. So that’s right. So we look at our businesses in two segments, right? We look at it as the instrumentation of and the collection of data around software, being the platform of record for this next generation of cloud software. And I don’t have to describe to you is is that, if, in the old days, it was an X Windows terminal talking to a Unix server app, today, there is no “there” there. There’s a virtual network; there’s VMs, containers, and microservices that are growing at exponential rate. The whole infrastructure is ephemeral, so the monitoring and the instrumentation workload has dramatically changed. So one side of our business is really focused on that, that opportunity, and we’d call that the “DevOps chain” of the business, right? Because it comes up into organizations through DevOps folks who bring in Influx and instrument.

    The other side is IoT, which is completely different. So, while DevOps is largely metrics-driven, IoT is largely event _____ driven. And that’s the instrumentation of the physical world, right? So our watches, our clothing, our health care, our cars, our homes, are all being instrumented with sensors. The amount of data pouring in through that is driving whole other systems into business. And, while most of our time was spent on the management of software systems back, now it’s this deep, deep integration of software and hardware. Deep. And so our business breaks out into those two categories, largely.

    Shimel: Got it. Got it. Evan, anything – any new news coming out of Influx you wanna share with our audience or –

    Kaplan: You know, where – we have – you know, we’re almost constantly releasing stuff into the community on a regular basis, so no news that I’d share at this moment. We can touch base in three or four weeks. We have our big InfluxDays event. We’ll make some announcements there, but you can look for us to build increasingly sophisticated solutions that allow people to do their machine-learning workloads, their application tracing, their log and metrics, their asset management, on top of Influx. What we want is people to use us as the platform of record for all of these metrics and events pouring in.

    Shimel: You know, we live in interesting times and I can’t wait to see where this all goes as well, so we’re about out of time, though, here. As I mentioned to you, the 15, 20 minutes goes so quickly.

    Kaplan: Yeah, it really does.

    Shimel: And you barely ever – you know, it is what it is. But, Evan, thanks for joining with us. Evan Kaplan, CEO, InfluxData. You know, so, in three, four weeks, you have your event coming up; maybe we’ll have you back. We can talk a little more about that. We’re doing a big – I didn’t even announce it yet, but I’m gonna announce it right here on the show with you – we’re gonna be doing a big virtual event, December 18th, called “Predict 2019,” and we’re gonna –

    Kaplan: [Laughs]

    Shimel: Yeah. We’re gonna be opening up the crystal ball and asking a lot of pundits and experts and so-called experts what their predictions are for what 2019 has in store. Maybe we can have you back then and we can do some crystal-ball gazing.

    Kaplan: That’d be great, Alan. Always a pleasure. Always a pleasure.

    Shimel: All right, my friend.

    Kaplan: Have a great day. All right.

    Shimel: Well, Evan Kaplan of North Woodmere and InfluxData, thanks for being our guest on this episode of DevOps Chat. This is Alan Shimel for devopsy.kinsta.cloud, and you’ve just listened to another DevOps Chat. Have a great day.

    Kaplan: Take care.

    — Alan Shimel

  • DataOps: DevOps Plus Big Data

    DataOps: DevOps Plus Big Data

    In traditional DevOps, there are the complimentary forms of development operations (that I call DEVops), and development operations (that I like to call devOPS). Between them they automate the toolchain and bring the people working on getting an application out to users onto the same team—not necessarily the same organizational team, but using their strengths in tandem to meet the needs of the business.

    DataOps is an interesting extension of DevOps. You still need all of the coding side to get data into the system and queries consistently maintained. You still need all of the operations side to get your database/noSQL/whatever up and running. In fact, while the development side in a DataOps environment might be (but isn’t necessarily) lighter, the operations side is almost always more complex. No matter the big data engine in use, it is a complex system in addition to other supported systems in a normal environment. My first installation of a big data environment (Cloudera, as it happens) was a weeks-long learning voyage. Only after I’d completed it did I use an automation tool (which is no longer available) to make it easy. My second round was hours … But it assumed the knowledge I had gained in the first attempt.

    The other bit of DevOps—ongoing monitoring and management—is also more complex in a big data environment, but we’ll come back to that in a moment.

    Once the normal DevOps systems are in place, ETL/data import tools will need to be supported also. The volume that these tools crunch through in a day makes their inclusion in DevOps in a data-heavy environment critical. If the data uptake is slow or the data itself inaccurate, there is an impact on the organization. This step also requires inclusion of data scientists, people who traditionally have not been pulled into the DevOps model. But they are the ones that can gauge the accuracy of data, and normally are responsible for data acquisition anyway.

    Which brings us to the monitoring and management stage of DataOps. Normally when we talk monitoring and management in DevOps, we are talking about tools that ultimately help with things like availability, responsiveness, auto-scaling and recovery. DataOps needs all of these things, like any other application does. And we’ve already mentioned that it has a complex environment embedded in your complex environment. But it also (some would argue more importantly) needs data monitoring. A knowledge of the state of the data is imperative. Whether from a piece of code gone wrong or data poisoning, monitoring the health of big data archives is essential to keep results from getting out of whack. This monitoring will again involve the people who know the data—data scientists by whatever title. This iterative process will need to run relatively constantly, checking on whatever makes the data valuable, normally using cross-field checks to verify that data hasn’t somehow been corrupted, or comparing trending averages to current average to monitor for sudden swings in results.

    Of course, for a system (and in research there are many) that spins up, finds results and spins down, the monitoring part is less essential, but for anything that keeps accessing and adding to the data pool, it is the key to a healthy application or application portfolio based upon the dataset.

    If all that sounds a bit intimidating, you’re not alone. The phrase “DataOps” has been around for a few years, and while data scientists use it, it hasn’t yet gained serious traction in the enterprise. Here’s how I would approach it as CIO: Given that DevOps is running on other projects, that is the low-hanging fruit. Given the rate of change for servers (as queries—particularly long-running ones—increase and decrease, systems needs change is one example), traditional DevOps is a greater benefit to big data than many other corporate systems. So roll out DevOps first without serious concern for DataOps beyond how DevOps impacts ETL—not ETL data quality, but ETL data availability/performance. Only when the system and the team are in place and the software/hardware/code part of the overall system is fully in a DevOps environment, start looking at ways to move data into it. Data monitoring will be the big one, but ETL under the initial round of DevOps implementation will have shown places it could be improved and ETL data integrity can be added, so improving those areas also. A measured, step-by-step approach that builds off of organizational experience is key here. Data monitoring and management will require different tools than normal DevOps monitoring and management, so having the data scientists look into available toolsets and how they mesh with existing tools early on will help with eventual implementation.

    Make no mistake, it is a big job, moving large data driven environments to DevOps. But the benefits are increased awareness of data quality and data quality issues, along with more responsive systems and teams that DevOps offers every business application. The amount of data available to any give organization is currently on a massive increase as IoT, open datasets, partnerships and ML drive greater pools. It is worth getting the automation and cooperation of DataOps into the mix as early as possible.

    — Don Macvittie

  • DevOps Chat: CHAOSSEARCH New Spin on Big Data Analysis

    DevOps Chat: CHAOSSEARCH New Spin on Big Data Analysis

    Analysis is the dark cloud overhanging big data — it’s difficult without the right tools. Elasticsearch by Elastic has become ubiquitous in helping organizations, but it can get unwieldy, as storage combining with compute is tricky. CHAOSSEARCH is an early stage startup designed to  make searching massive amounts of data using elasticsearch and Amazon S3 easy.

    Pete Cheslock, VP of Products at CHAOSSEARCH and a good friend of devopsy.kinsta.cloud, joined me in this DevOps Chat to discuss his latest company’s technology and how it might make big data searches less daunting. The key for big data search and analysis is separating storage from compute, using Amazon S3 and Elasticsearch.

    As usual, the streaming audio is immediately below, followed by the transcript of our conversation.

    Audio

    Transcript

    Alan Shimel: Hey everyone, it’s Alan Shimel, devopsy.kinsta.cloud, and you’re listening to another DevOps Chat. Today’s DevOps Chat features a returning guest to DevOps Chat. He’s actually probably been on DevOps Chat a number of times, right from I think the very beginning of when we first launched devopsy.kinsta.cloud. He’s pretty well-known in the DevOp community and really needs no introduction from me. It’s Mr. Pete Cheslock. Pete, welcome to DevOps Chat.

    Pete Cheslock: Hey, thanks a lot. Thanks for having me.

    Shimel: Yep. Pete, always a pleasure to have you, man. You know, I remember speaking with you – actually it was probably before I even launched devopsy.kinsta.cloud, up in Boston. And you kind of gave me your views on what was then a very kind of just starting to blossom DevOps community and DevOps movement. And since that time obviously a lot of water under the bridge and many different hats that you’ve worn, Pete. Here to introduce today though with your latest position in helping a company grow, and that company is called CHAOSSEARCH, correct?

    Cheslock: Yeah, that’s correct.   

    Shimel: And Pete, you’re VP of Products?

    Cheslock: Yeah, yeah, VP Products, Products, VP of Products? I don’t know, it’s a working title. We’re still trying to figure out what the best way to –

    Shimel: As long as they don’t call you a DevOps engineer, it’s all good. [Laughs]

    Cheslock: Exactly.

    Shimel: I hear ya. So Pete, well let’s start off with this. I’m going to assume a lot of people listening to this today don’t really know or are not familiar with CHAOSSEARCH, so why don’t we start with that? What’s CHAOSSEARCH?

    Cheslock: Yeah, absolutely. And yeah, I think that’s definitely a good assumption. You know, we’re an early-stage startup. We’re just going through seed funding right now, so we’re just going through an A round of fundraising, so then we’re going to be coming out to the market in a few months here. But basically what CHAOSSEARCH is is they’re trying to make it easier to access and make query-able your data as it exists on object stores. So if you think of an object store like Amazon’s S3 service. And so they developed basically a technology that allows you to store data. They basically process your data from S3 and store it back into your bucket. So what’s really interesting is they don’t hold any of your date. They’re simply – you know, you give them like a read-only to your S3 data. They’ll read in that data and shove the compressed, processed data back into your bucket. And then from there they expose APIs to allow you to access it.

    So for example, they’ll extend the S3 API to give you some really interesting capabilities of slicing and dicing your data. But the thing that they’ve exposed that’s most interesting, which got me excited about the company, is that they’ve exposed the elasticsearch APIs on top of your data on S3. And so it’s really compelling, because I spent a very long time as a really early user of elasticsearch, and it’s a great technology; it’s a great open source database. But it gets unwieldy over time. And so what really blew me away when I saw the product and what they’d built is that I can point open source log tools like Kibana to the CHAOSSEARCH APIs, accessing my own data on S3. No elasticsearch required. And so the kind of short pitch, we’re trying to make that long tail of data available to people, whether it’s log data or event data of some kind, where customers can run much smaller elasticsearch clusters of their log data or event data, and then they can essentially push the data up to S3. We’ll process it, and now they can get access to that data for weeks, months or even years.

    Shimel: Yeah. And Pete, that really highlights sort of a soft-white underbelly or a dark corner around the whole big data issue. It’s great to have – I mean, big data and – big data is only as good as the analysis we can do with it, right? That we can perform on it. And with big data we could certainly get some really keen insights to show us relationships, trends, facts that we might otherwise miss. But the dirty little secret was, for big data, to really get the value out of it, generally you needed big money kind of search to perform that kind of analysis. Because the bigger your data got, the harder it was to actually manage it, search it, analyze it. And so CHAOSSEARCH seems like a real solution for those people who are literally drowning in data.

    Cheslock: You’re exactly right. I worked at a company eight or nine years ago called Sonian, which – not a lot of people have heard of Sonian. They were one of the earliest users of Amazon’s S3 and EBS. At the time we were probably the top-three user of S3 and maybe the top-five user of EBS. What’s amazing about Sonian is the number of companies that actually came out of that place. Companies like CloudHealth, which just got acquired by VMware. Sensu was a technology that was built inside of there. Stackdriver was another one. And the reason all these startups came out of Sonian is because they were so early in the cloud space, they had to build out the technology to support it.

    One of the other ones that kind of came out on the side, not really related to us, was elasticsearch. We were trying to index customer e-mail data, like an e-mail archiving product. And so we got connected with Shay from elasticsearch before he really started the company because we were so blown away by what he had built. And elasticsearch at the time made sense, right? Using Lucene to store indexes of customer e-mail data allowed the indexes to be much smaller than the sourced e-mail files. But then over time, over the last say eight years or so, elastic’s been around, there’s been this explosion of using elasticsearch for log data. And in many cases, like structured JSON that we’re shoving into elasticsearch. And due to the inverted index nature of Lucene, the data blow up inside of Lucene where you might actually store four or five times more in disc storage on elasticsearch.

    So if you shove in a terabyte of JSON logs into your elasticsearch cluster, that could be three or four terabytes of Lucene indexes. And then you have to think about like, “Well, I want durability and reliability, so now I need to like run a replicant index.” So now you’re talking for every terabyte into elasticsearch you need like eight terabytes of physical disk. What was really amazing about what the founders and the early team here has created is the format, kind of solving for the Lucene problem. They’ve actually been able to reduce the size, yet still making the data query-able, using kind of the same APIs, which is what really blew me away, in that it’s now – you know, so many people have to make the choice. Do I keep my data for analysis, at a really high price of elasticsearch? Or, do I throw it away? And I think a lot of people just throw that data away, because they just can’t make it worthwhile in any sort of cost effective way. 

    Shimel: Right, not in an economic way. Pete, we’re far from a big data solution here at MediaOps, between our seven or eight websites, but I will tell you, just pure search, like an index search on devopsy.kinsta.cloud has become a struggle. Because a lot of the native WordPress search – you know, especially if it’s on server, is killing resources on your web server and so forth. So you want to do it sort of off-server search and it’s just a pain in the butt. Take my word for it. So this is great. You said the company’s launching in a few months though? Not that it’s launching – it’s looking for seed money now. Product will be in market. Where exactly is it now in terms of – is it in beta? Is it alpha?

    Cheslock: So yeah, we’re probably going to be closing our A fundraising round over the next 30 days or something like that. The company actually started about two and a half years ago, and so they’ve been essentially building this technology out over the past couple of years and really trying to refine it as they were kind of were coming up with this like product market fit. Like what’s the market we’re going into? And as the founders really looked at the space, they realized that log and event management is where a lot of pain is right now. What’s interesting is when people start using the product, they realize that they can process any kind of data with this and query it. And that’s where there’s a lot of interest, where maybe it’s not just log and event data. Maybe it’s like some long tail of data that you want to get access to or to monetize for your customers.

    But where we’re at right now, and what was exciting about me coming onboard is not only did my role change completely – I used to run Technical Operations for Threat Stack, which is a pretty successful cloud security company that’s still just crushing it. I changed my role over to become the Head of Product. So working with customers to understand their use cases and how they’re using tools like elasticsearch. So where we’re at right now is I’m basically looking for what we’re calling design partners, where – the best way to describe the product is in that beta phase, where we’re looking for people with really interesting datasets and maybe people who are on Amazon and have data on S3 or are using elasticsearch, and getting them into our proof of concept setup right now, where we can have them run queries, test the analysis, test how they might use it and really help us refine the final bits as we kind of _____ things up towards GA.

    And then, once we go to GA, towards – I would say October timeframe I think is what we’re aiming for, it’s really to continue to refine these really specific use cases like, “I’ve got data on S3 and I want to process this data.” We’ve had some customers that want to push their elasticsearch indices up through the service, which is a really interesting idea, because those loosing indexes already have a defined schema. And so that way, instead of storing seven days of data in elasticsearch, maybe they just store one day and that’s in their hot cluster in elasticsearch on EC2 or something. And then days two through 365 are up on S3 and are available through the CHAOSSEARCH service. So yeah, that’s kind of what the next month or two looks like for me. You know, really trying to work with customers, people with interesting datasets. You know, people who have just way too much elasticsearch and they want less of it. You know, they want to kind of run less servers, spend less money – which I feel like is kind of everyone out there right now.

    Shimel: Excellent. Very cool stuff. I mean, Pete, look, for you to get up and leave Threat Stack and come over here, obviously it had to be something really appealing and compelling, to bring you over there. Pete, it’s interesting. My background prior to devopsy.kinsta.cloud was founding and building companies, and one of the things I learned is it’s great to have these grand – come up with the idea of Docker for containers and change the way – you know, the basic platform that we use, right? For our apps and for IT. But those revolutionary kinds of things are really kind of few and far between, especially the ones that succeed. Really IT moves along in evolutionary kind of ways, where it’s a thing like this. It’s thing like with CHAOSSEARCH is doing, that makes it easier, better, faster, more economical to really get a hold of your big data and do the analysis and get a handle on it that you need to make it valuable. Right?

    Cheslock: Yeah. Exactly right.

    Shimel: And so, you know, this just feels that way to me – that this is a lot – some people may say, “Well, it’s not earthshattering.” But no, but it’s the logical step. It’s the glue that you need to – you know, big data is a big concept and it’s great and it’s going to revolutionize things when we get it. But you need the CHAOSSEARCH or sort of the glue to make these things happen, right?

    Cheslock: Yeah, what’s amazing is, as I’ve talked to people running a lot of elasticsearch, a lot of data on S3, it’s a really easy sell, because I can go to them and get them away from running these huge clusters that – they may not even take a lot of queries too. That’s what’s interesting is maybe they’re only querying their data in an automated way the last hour or 24 hours. But then you start looking out seven days, a few weeks, maybe a few months; those queries are significantly less. And so you have to either build out like a tiered storage. But at the end of the day the storage and computer are coupled so tightly that your scalable units on Amazon or any cloud provider is a lot of disk and a lot of CPA all together. And what CHAOSSEARCH has done, it uncoupled storage and compute, where the storage is on S3, specifically your S3 account, and the compute is the CHAOSSEARCH service.

    And so it’s a totally different way of thinking about, “How do I access my data? How do I make it query-able?” And now, because it costs so little – and we’re talking like go-to-market pricing a tenth of the price of running elasticsearch yourselves, and significantly less than a lot of the other kind of hosted management, log manager vendors. You know, it’s like now I can keep my data for month and a year and I can run machine learning models cost-effectively, because I can keep so much data. That’s the part that’s super interesting, is, how does this change data science and those companies that really want to have the long tail of data for ML/AI testing, and how does this change how they work their models, how they build it in? And we’re even trying to build in like tenser-flow integration into the application so that you can just access it from within without needing to have all this crazy data science experience.

    Shimel: Excellent. It really is. Pete, you know, 15 minutes goes quick. We’re coming up on our thing here. You’re VP of Product. Where do you see this going six months, a year, two years out? It’s a long way to look, but where do you see yourself taking the product direction here?

    Cheslock: Yeah, no, I always like to joke that the next six months are clear and in focus. Things get really fuzzy really quick after that. But the nice thing is I think we have a pretty clear plan going out of it. Through the end of the year, reaching GA, bringing some customers onboard that we can solve a problem for them. But really, early next year we’re going to start beta-ing a self-service kind of model, where people can come in and try it out. And probably doing even a freemium model, because since the data is in your S3 bucket and we’re just processing that data, cost is very low. So being able to provide someone a couple of gigs a day of data processing, it’s pretty easy for us. And then that way people can kind of get their feet wet and try it out and we can really start building a community around it.

    And then from there, I think the sky’s the limit. You know, we’re looking at some time end of next year, the ability to run it within your own data centers. Right now we want to be kind of pure SaaS so that we can continue and build it out and improve it in our kind of very quick, iterative development cycle. But really it’s a data platform we’re building. I mean, we’re going after that log and event management space because it’s so painful for so many people and it costs a lot of money. But the long-term for us is, what other APIs could we expose on top of your data on S3? Elasticsearch seemed to make a lot of sense, given the market. We’ve extended S3 with some interesting features. But like what else is there? What are different ways people are trying to access different data? And so I think that will be a lot of fun to kind of chat with people, understand their pain and maybe actually bring them a nice solution to that pain.

    Shimel: Yeah, absolutely. And I’m sure you’ll have a ball doing it, Pete. Anyway, we’re over time here. Pete Cheslock, newly minted VP Products, CHAOSSEARCH. Thanks for joining us on this episode of DevOps Chat, and we look forward to hearing more about CHAOSSEARCH and about your adventures, Pete.

    Cheslock: Yeah, thanks a lot. Thanks for having me. This was a lot of fun.

    Shimel: Always a pleasure. Pete Cheslock, CHAOSSEARCH. This is Alan Shimel for devopsy.kinsta.cloud. You’ve just listened to another DevOps Chat. 

    — Alan Shimel

  • Moogsoft Extends AI Reach into IT Operations

    Moogsoft Extends AI Reach into IT Operations

    Moogsoft, at a Moogsoft 2018 AIOps Symposium last week, extended the reach of its IT operations analytics software out to where applications are deployed, rather than requiring data to be moved into a Big Data platform.

    Will Cappelli, CTO and vice president of product strategy for Moogsoft, said Moogsoft Observe, available in beta, discovers anomalies by applying algorithms where data is generated, versus requiring IT organizations go to the trouble and expense of moving massive amounts of data into a central data lake.

    Moogsoft Observe ingests time-series and metrics data in real-time and then applies artificial intelligence (AI) to detect incidents at the source of the problem. The single-agent technology can deployed anywhere, including cloud instances and on-premises systems and applications running containers and virtual machines.

    Cappelli noted Moogsoft Observe doesn’t require data scientists to configure it. IT generalists can employ Moogsoft Observe on their own without having to rely on a centrally managed IT operations analytics platform managed by someone else. Moogsoft claims this approach reduces the time to detect a problem by up to 60 percent while cutting nearly 90 percent of the storage and communication costs.

    Analytics generated by Moogsoft Observe can then be stored in the Moogsoft AIOps platform to enable IT organizations to run additional analysis against data collected from multiple data sources.

    Moogsoft also announced that version 7.0 of the Moogsoft AIOps platform will be available this October. This update adds Vertex Entropy and Situation Topology Visualization, algorithms that enable IT teams to visualize the root cause of an issue via a system topology map. That capability makes it possible to apply algorithms capable of generating contextual analysis that pinpoints the source of issues rather than generating a series of probabilities based on when a set of events occurred, said Cappelli.

    Advances in how AI can be applied to automate IT operations are now coming at a rapid rate. Those advances increasingly are becoming affordable for the average IT organization to employ. Most IT teams spend more time trying to determine the source of problem than they do fixing it once the root cause is discovered. AI technologies promise to eliminate many of the rote tasks that IT administrators are required to manually perform time and again, which will make it simpler to manage IT at scale. It also means more IT teams can expect to not have work as many odd hours of the day because fewer things are likely to go wrong. There may never be a perfect IT environment. But IT environments should be a lot more predictably reliable.

    Of course, the rise of AI undoubtedly will have a significant impact on the IT staff. Many IT administrators will need to be retrained as more processes become automated. It’s not likely the need for IT administrators will go away anytime soon. But as some tasks become automated, many entry-level IT roles will change dramatically or, in some cases, be eliminated. At this point, those changes are now all but inevitable.

    — Mike Vizard

  • Creating a Better Plan with Agile Data

    Creating a Better Plan with Agile Data

    Agile methods help companies respond to change, beat the competition and build high-quality products that customers want by aligning development work to business priorities. But what happens when an organization is unable to align strategy to execution or align its departments or teams due to internal silos? As organizations advance in agile maturity, so do their data analytic needs. Compiling and synchronizing the data of a few agile teams is significantly different than rolling up data across many teams or an entire portfolio.

    Without a clear picture of how teams are delivering their work, organizations may be flying blind when it comes to understanding the delivery of their products or services.

    See People, Time and Work in a Predictable Way

    Every team, department, and organization relies on data to make important decisions that steer the business. What many organizations lack is a way to clearly understand, prioritize and utilize the data they have. From an agile practice perspective, that level of visibility should be established early on in a data reporting structure—project, initiative and work hierarchies—and with the proper agile software solution.

    Setting data reporting standards helps teams understand how they are going to refer to, read out and analyze the data within their company. This type of data structuring promotes the standardization of data across teams, projects and work. But agreeing on a data infrastructure is only one part of data analysis. All of your organization, team and work-task information still needs a centralized system of record for proper utilization. Without a centralized database or agile software solution, data management becomes an extremely manual task. By establishing a data schema for your team-level, day-to-day work within a centralized system of record, all teams can start down a path of easier data accessibility, rollups, comprehension and insight.

    But, this type of reporting is not the end all, be all for most organizations. To truly have a full and comprehensive read-out of agile work, teams need to be able to slice their data any way they want to accurately reflect the work of the organization as a whole.

    How Does Data Discipline Lend Itself to Predictive Insights and Planning?

    Formatting data in an organized fashion has to be the first step in understanding the work that is happening across an organization. Without this first step, agile metrics become a tedious task of manual data rollups and missed opportunities. The hodge-podge of data becomes nearly unusable outside of each individual team, as the organization does not know how to compare health, status, risks, dependencies, cadence or velocity team to team.

    Getting an organization grounded on common data reporting practices has many benefits, such as:

     Everyone is Aligned to the Priorities

    With all teams seeing the work produced across an organization, teams keep focused on the most important tasks, align on dependencies and ensure they meet important delivery timelines. It also prevents unsanctioned “work for the sake of work” from happening, as all eyes can see the work in the system and how it impacts the overall business goals.

    Highly Prioritized Backlog

    Not only are teams able to align on the priorities, but they align on the prioritization of the work needed to meet and deliver on the business goals.

    Better Time Management

    With a common data management structure built into your agile software, time management, velocity and cadence planning become much easier. Leadership can see which teams are successfully delivering all their work during each sprint, as well as how much work they can deliver over time. This sense of time tracking and management gives executives a glimpse at predictive delivery. With predictive delivery comes more predicative revenue.

    Consistent KPIs

    If you ask several individuals what their idea of success is, you will get a different answer from each person. Looking at agile metrics is very much the same. If teams are given free rein to set up their own data structure and reporting, all reports and dashboards will be different. This makes rolling up the progress, status and health of initiatives time consuming and painful. With a governed data structure prebuilt in an agile software solution, all teams and departments within the organization can see the same type of reports and metrics all in one place. These teams then have the ability to analyze their data using standard reporting metrics from standard reports or dashboards.

    Transparency

    While no one wants to think that a project can go awry, it does happen. And it is much better to make course corrections as early as you know there is an issue, as opposed to waiting until it is too late. Giving all parts of an organization a centralized place to review progress in a standardized way promotes overall success.

    The Ultimate Goal: Build a Better Plan

    Standardized agile performance metrics that stem from an ordered data-schema empower organizational productivity, predictability, quality and responsiveness. With the insight that is gained from this level of data consistency—historical, real-time and health details—the pump is primed for deeper insights and advanced data manipulation.

    The Future is Predictive Planning and Big Data

    The old adage, “You only get out what you put in,” also holds true for how an organization uses its data. This is why that critical first step of organizing and housing your agile work in a similar structure in a centralized system is key. Without this simple alignment, advanced metrics and analysis are nearly impossible. And it is those advanced metrics that are the key to truly predictive planning. The future of planning lies in customized data reporting and manipulation with big data analysis. With big data, disparate pieces of information can be analyzed, effectively slicing and dicing the metrics any way an organization needs. Reports can be created to fit the shape of the organization, roles and teams that need them–how they need them. To do this, organizations need to input the data into a centralized system first! Analysis is only as good as the data inputting allows.

    — Brook Appelbaum

  • Strap in, CIOs: Analysts Predict Wild Ride for 2018-2020

    Strap in, CIOs: Analysts Predict Wild Ride for 2018-2020

    Digital transformation is the underlying theme of many of the 2018 and beyond IT predictions made recently by analyst firms Forrester, Gartner and IDC. Technologies such as cloud, AI, data analytics, big data, blockchain, IoT, conversational interfaces, DevOps and cybersecurity will also force CIOs to perform a juggling act. The rapid evolution of these technologies combined with the business opportunities they unlock will create a pressure cooker for CIOs, become an everyday topic of discussion for C-suites and possibly even a preoccupation of boards of directors.

    We’ve analyzed the predictions by the three research firms and have gathered the key themes as the seven most pressing issues for the C-suite next year and beyond:

    1. CIOs are at a Crossroads, Representing Both Challenge and Opportunity

    Forrester believes that CEOs will make digital business a top priority in 2018 and CIOs will lead the digital transformation at most companies. In a November report, “CIOs Make the Chief Digital Officer Obsolete,” Forrester summed up: “In 2018, boardroom discussions will focus on digital. Key questions will revolve around platform-based business models, new partnership ecosystems, and new ways to organize for success. CIOs will build new skills and competencies to help drive this change.”

    The challenges for CIOs during this period will demand that they step up, expanding their digital skill sets and employing technologies with both eyes focused on business value. Forrester predicts that CIOs will need to orchestrate the underlying technologies and supplier relationships for companies to manage their platform-based business approach. CIOs would also be smart to leverage cloud computing and open source software, and build on containers, APIs and microservices.

    CIOs also may need to develop new leadership skills to direct their company’s digital transformation. That includes displaying a laser-like customer focus, attracting the necessary talent, restructuring their organizations and developing the culture required to deliver tangible business results. Doing so, Forrester said, “will present CIOs with the opportunity of a lifetime, allowing them to demonstrate the level of trust and performance that their senior peers and boards require, and positioning them as leaders of the digital transformation — even the enterprise.”

    IDC’s November “Worldwide CIO Agenda 2018 Predictions” offers predictions that dovetail with Forrester’s view:

    • By 2020, 60 percent of CIOs will implement an IT business model and culture that shifts focus from IT projects to digitally oriented products.
    • By 2020, 60 percent of the CIOs who have crossed the digital divide will prevail in C-suite turmoil in competition to become digital business leaders for their enterprises.

    Advice to CIOs: Find a leadership coach. Choose one or two lieutenants you trust to delegate important roles to. Find a senior product manager-type person in your company to help you become more knowledgeable about generating business value. Make a point to talk to your CEO about the advantages of digital transformation. Develop a plan for adopting a new digital governance model. What should your digital platform look like?

    2. Digital Transformation Becomes Real for Most Companies in 2019-2020

    We’ve been hearing for a couple of years that companies are digitally transforming their businesses. The truth is that those companies represent the leading edge and they are a small minority. Most CIOs are still stuck doing more with less, the hangover from the last recession. CEOs: Your CIO needs more budget to position your company for digital success over the next few years. IDC predicted that “through 2019, 75 percent of CEOs and their enterprises will fail to meet all their digital objectives, dragged down by conflicting digital transformation imperatives, ineffective technology innovation, cloud infrastructure transition and underfunded end-of-life core systems.”

    That’s a lot of failure. CEOs and CIOs are going to need to work together to make digital transformation a reality. It’s not just about updating your technology, it’s about business innovation.

    Key points on digital transformation from IDC’s CIO 2018 predictions:

    • By 2019, 60 percent of CIOs will complete infrastructure and application replatforming using cloud, mobile and DevOps, clearing the deck for accelerated enterprise digital transformation.
    • By 2019, 60 percent of IT organizations will have deployed digital transformation platforms supporting new customer- and ecosystem-facing business models.
    • By the end of 2019, digital transformation spending is expected to reach $1.7 trillion worldwide.

    3. AI is No Panacea

    Forrester named one of its 2018 prediction reports “The Honeymoon for AI Is Over.” Conventional analytics are undergoing full disruption, largely because AI is perceived to be the solution to both programming analytics and solving the big data riddle. And while it’s likely that AI/machine learning will be able to help with both problems, you can’t just snap your fingers and make it so. Harnessing AI successfully requires specialized engineering talent, quality data, objective planning and some trial and error. According to a Forrester survey, 70 percent of respondents say their firms will have implemented AI in the next 12 months. They chose decision management as their most important use. One of the key points to remember is that the decision-making abilities of it machine learning tool can’t over calm a lack of quality data. Forrester warns: “Don’t be fooled by vendor claims that more intuitive human-to-machine interfaces are [a magic potion] that reduces the need for data and metadata management and curation.”

    Quick AI predictions from IDC:

    • By 2019, 40 percent of digital transformation initiatives will be supported by cognitive/AI capabilities, providing timely critical insights for new operating and monetization models.
    • By 2020, 85 percent of new operation-based technical hires will be screened for analytical and artificial intelligence skills, enabling the development of data-centric digital transformation projects without hiring new data-centric talent.

    4. One-Third of Enterprises will Abandon Big Data Repositories in 2018

    Has your company built a large data lake that hasn’t delivered any significant business value? Well, 2018 may be the year when you rethink that decision, according to Forrester.

    On the other hand, Forrester believes that AI will be able to resolve the longstanding issue of the boundary between structured and unstructured data because of vendors including Digital Reasoning, Expert Systems and Stratifyd.

    As you make your move to digital transformation, having more actionable data based on your pool of big data, especially unstructured text data, could deliver book valuable business insights. This might not be the best time to give up on big data. Especially if you’re actively engaged with AI.

    5. Blockchain: Confusing, Skills Gap, Steeped in Hype—but Full of Potential

    Blockchain will begin to show next year just how difficult it will be to employ but, at the same time, that it’s worth exploring.

    “By the year 2020, the banking industry will derive $1 billion of business value from the use of blockchain-based crypto-currencies,” states Gartner’s October “Top Strategic Predictions for 2018 and Beyond.” “When compared to the estimated $7.6 trillion that is the industry gross output of the world’s banking industry, $1 billion may not seem like much. However, the value is more about the tacit endorsement of crypto-currency as a legitimate option by the banking industry.” Cryptocurrency is just one application of blockchain.

    Advice: Build your blockchain ecosystem early, Forrester recommends in its 2018 prediction report, “Be Ready to Face the Realities Behind the Blockchain Hype.” “The blockchain terminology will remain confusing,” Forrester noted. “The term ‘blockchain-inspired’ is already a common descriptor for software that has some of the characteristics of blockchain or reflects some of the principles of the blockchain network, but isn’t a blockchain in the true sense.” There is a growing skills gap pertaining to software engineers and security professionals who have experience with blockchain. It’s a nascent technology; make sure your team sets realistic goals.

    6. IoT Hits Prime Time, but it Might Not Be Ready

    By 2020, IoT technology will be in 95 percent of new product electronics designs, Gartner said. IoT’s growth is very rapid, even for the computer industry. But that growth comes at a cost. The technology has not overcome what many believe to be rampant security vulnerabilities. “Through 2022, half of all IoT security budgets will be spent on fault remediation, recalls and safety failures rather than protection,” Gartner predicted.

    Forrester is far more positive about the business advantages of IoT in its report, “IoT Moves from Experimentation to Business Scale.” Its predictions are also more detailed in the several areas where IoT might gain traction in 2018. But the security issues are not lost on Forrester, either. The research firm predicted: “IoT will be at the center of broader and more damaging cyberattacks. The Mirai botnet was a harbinger of future IoT security-related attacks and breaches. … Most firms still haven’t shown a consistent commitment to mitigating IoT-specific threat vectors.”

    Advice: IoT is an accident waiting to happen. For B2C companies, the consumer backlash could be fierce if there is a major incident. Make 2018 be the year that you start plugging up those IoT security vulnerabilities and protection gaps.

    7. Most Companies Won’t Comply with GDPR in 2018

    The European Union’s General Data Protection Regulation (GDPR) takes effect May 25, 2018. Forrester predicted in its November report, “A Year of Reckoning,” that 80 percent of companies will fail to comply with the GDPR. Perhaps more to the point, Forrester additionally predicts that 50 percent of those non-compliant firms will do so intentionally, “meaning they have weighed the cost and risk, and are taking the path that presents the best position for their companies.” The other half of the 80 percent will try to comply but will fail.

    Advice: It seems likely that some large company will be made an example of. You don’t want that to be your company.

    Takeaways

    Those are the standout IT-related issues for the C-suite in 2018 and beyond. If you weren’t tuned in to some of them, knowing is half the battle right? Well, probably not. These are tough challenges. Each organization has its own strengths and weaknesses. But generally speaking, if you parse this list by investing time and budget on the first four while minimizing the potential downsides of last three, you’ll be doing very well.

    — Scot Finnie

  • Is AWS Cloud Per-Second Billing Boon or Bane for DevOps Engineers?

    Is AWS Cloud Per-Second Billing Boon or Bane for DevOps Engineers?

    Amazon’s introduction of its AWS services in 2006 with a per-hour billing model for compute resources was a game-changer for the IT world. Since then, AWS has taken over the cloud computing market as a leader, while broadening the range of cloud services offered every year. However, the per-hour billing business model was one of the major issues for many customers—in particular, those who had applications using EC2 for only minutes at a time.

    Recently, AWS announced support for per-second billing of its EC2 service, addressing the aforementioned concerns. Of course, this also created controversy among the per-hour supporters. While I won’t get into the pros and cons across the whole enterprise, one group in particular is affected from the new per-second model: DevOps engineers.

    Here are the top five implications for DevOps teams as a result of per-second billing:

    Autoscale without worrying about cost

    Most engineers were vary of autoscaling at will, mostly because of EC2’s hourly charges even though they might need additional compute capacity for just 15 minutes. With per-second billing, DevOps engineers don’t have to worry—it’s as simple as scaling your EC2 instance as you need to, and when you need to.

    No more custom logic for hourly batches

    Many customers use EC2 for jobs and batch processing purposes. Due to the hourly pricing, engineers have written custom logic to benchmark and constantly tune the number of nodes in their batch clusters to ensure they are cost-optimized. Per-second billing frees up headaches for DevOps engineers that are using EC2 for batch processing and job clusters.

    The need to speed up EC2 launches

    With per-second billing, it’s important to take advantage of prebaked Amazon Machine Images (AMI) for launching your EC2 instances, instead of complete configuration setup on every instance launch. This will reduce the costs for you, by saving anywhere from five to 15 minutes. It’s time to create prebaked AMIs for faster instance launches, ultimately saving costs.

    Dilemma of Lambda or EC2

    Be prepared to compare whether to use Lambda or EC2 for new workloads and applications more often than you are used to. Your benchmarking between Lambda and EC2 has become a little more dicey. The serverless lambda just got competition from EC2 again.

    Your big data just got easier

    With EMR also supporting per-second billing, your job has become a lot easier. As a , you no longer need to constantly benchmark your EMR clusters for number of nodes and job processing times to be optimized for per-hour billing.

    DevOps engineers have a lot to benefit from per-hour billing, but also much to consider when it comes to time-to-deployment and choosing a compute service. Now is the the time for DevOps engineers engineers to bring these concerns to their CIOs to make a smooth transition to the new AWS business model.

    About the Author / Vijay Rayapati

    Vijay Rayapati, CEO of cloud management platform company Botmetric, identifies himself as a technology entrepreneur, passionate about creating products, building high-performance teams, and solutions that solve real-world business issues. Vijay is a popular speaker in India and the US, and has co-authored a popular book, “Learning Big Data.” Connect with him on LinkedIn and Twitter.