devopsy.kinsta.cloud

Tag: analytics

7 things 2016 has in store for the future of DevOps and IT
With only a couple weeks left in 2015 it’s time to start looking ahead to what next year has in store. Let’s be honest—most of us have already mentally checked out for the holidays anyway, so it’s already 2016 for all intents and purposes. I spoke with a handful of industry experts to get some thoughts on what we have to look forward to in 2016.

Let’s just jump right in:
1. Log Management will play a larger role
According to Christian Beedgen, CTO of Sumo Logic, log management will emerge as the most underrated opportunity for IT. “Thinking about log data may seem too far in the weeds for most tech industry professionals. However, using analytics to monitor, manage and gather insights from logs will be the only near-perfect way to make sense of the increasingly complex and cloud-based architectures.” Log management isn’t exactly new or cutting edge, but Beedgen believes that 2016 will see more vendors trying to move into the log management space, and more organizations trying to use log management to make sense of their big data mess.
1. Business leaders will demand better insight
On a similar note, Rick Fitz, senior VP or IT Markets for Splunk, believes that analyzing data and metrics will also play a role from a business strategy perspective. “In 2016 we will see data science demystified, moving from Ph.D. to MBA, with easier ways emerging to consume, analyze and correlate machine data. As business leaders increasingly go digital, this will result in greater “digital insight” into engagement, satisfaction, competition, effectiveness, response, revenue and other business goals.”
1. Sharing economy will transform IT
Companies like Uber and AirBnB have transformed entire market segments. The sharing economy has changed the way businesses engage customers and deliver services, and that includes IT through aspects such as cloud services, microservices, APIs and more. Fitz predicts, “Next year this ‘Uberfication of IT’ will turn into a ‘Balkanization of IT,’ driving stakeholders to demand better insight, governance and control of federated technologies to integrate adoption, usage, monitoring, security, cost control and more across shared services.”
1. Enterprise DevOps will grow rapidly
The DevOps revolution has taken IT and app development by storm in recent years, but most of the traction has been in startups or smaller businesses that have more freedom to shift gears and embrace new concepts. Enterprise adoption has been growing, but at a much slower rate. Andi Mann, chief technology advocate for Splunk, said, “In 2016 we will see rapid acceleration of DevOps with 50 percent or more of large businesses finally adopting ‘Enterprise DevOps’ — a pragmatic, scalable approach to adopting the culture, process and technology changes at the heart of DevOps.”
1. The singularity approaches
OK. That may be an ominous over-statement. In a nutshell, Sumo Logic’s Beedgen claims that the line between machines and humans will continue to fade. The foundation of this is automation—leveraging technology to perform mundane or routine tasks and sift through mountains of data exponentially faster than a human can. As artificial intelligence evolves, though, the scope of what can be automated expands as well. “In automating processes, a human can today wield the power of a legion of operators with a single hand,” stated Beedgen. “Likewise, machine learning will increasingly be adopted for its usefulness in augmenting human understanding of complex interaction and large data sets by uncovering the unknown unknowns.”
1. Automation and analytics will improve IT service health
DevOps and microservices have transformed many areas of development and IT infrastructure, but many organizations still rely on decades-old monitoring tools, arcane metrics, and specialized admins dedicated to managing the cumbersome process. Splunk’s Fitz thinks that is getting ready to change. “In 2016, IT analytics will be liberated as easy-to-use dashboards will correlate all sources of service health—legacy tools, APIs, mobile apps, Web logs, wire data, containers and more—while predictive analytics will automatically respond to prevent quality issues.”
1. DevOps is here to stay
From the point of view of DevOps practitioners or companies that have adopted DevOps tools and practices, this may seem both painfully obvious, and yet still be good news as well. Christian Beedgen predicts not just that DevOps is here to stay, but that the importance of DevOps and the sphere of influence affected by DevOps will expand. “Increasingly, DevOps will permeate all manner of businesses that rely on information technology to compete. Bimodal IT (the practice of managing two separate, coherent modes of IT delivery) will become the equivalent to “private cloud”—the last forced resistance of the old guard. Both will be forgotten in a short number of years, as public cloud and lean and agile processes dominate the businesses that still compete.”

There you have it. Now you can enjoy your holiday break in peace and come back in 2016 with an idea of what you have to look forward to in the year ahead.
December 15, 2015
How IT Operations Analytics is Delivering on the Promise of DevOps

DevOps is one of the most significant movements in IT of the past several years, and the same trends that are making DevOps so important – speed, automation, and iteration – are also opening up new opportunities for IT infrastructure management. DevOps, at heart, is a matter of adopting new philosophies of working as an IT team, while letting go of legacy processes and technologies that no longer deliver an adequate level of efficiency. DevOps is all about doing things in a more efficient, smart and cost-effective way by adopting the latest innovations, working across siloes to combine the strengths of different people on the team, and bringing different functions together to make a more responsive and effective DevOps organization.

In the same way, today’s IT operations analytics solutions are becoming more friendly to this spirit of collaboration, as infrastructure is becoming more complex and heterogeneous. DevOps teams have to work together in new ways, with different processes and different parts of the organization interacting in unique ways that might not have been expected just a few years ago. Similarly, today’s IT operations analytics world is evolving to run IT more smartly by taking advantage of new opportunities to manage the stack more efficiently.

Ultimately, the role of DevOps and the role of IT operations analytics are about developing competitive advantage. If your organization can use DevOps principles to build packages or release code faster or more accurately, you’re going to be ahead of the game compared to your competitors. If your organization can harness the power of the latest IT operations analytics innovations, you will similarly have a competitive edge because you will have a higher degree of control and visibility into what’s happening within your infrastructure. IT operations analytics can serve as an ideal complement to DevOps – because it helps to reinforce and protect the competitive advantage gains that are realized by DevOps.

Here’s what I mean by that statement: DevOps does a great job of combining speed, automation, and iteration. But as a practical issue, creating competitive advantage is more than new features, the apps you deliver must be reliable and high performance or customer expectations will not be met. IT operations analytics serves as a complement to the speed and agility of DevOps by identifying new innovation opportunities and providing “wow!” moments for the customer. With better IT operations analytics, you can exert better control over your IT infrastructure – and better control leads to bigger innovation opportunities, because you can keep the control in-house without having to rely on an outsourcer for identifying your innovation opportunities.

There are several market segments where the link between IT, the customer experience, and revenue are quite obvious. Optimizing online game play using real-time analytics is a great example, but the trend is much broader than that. With so much of the customer experience determined by IT systems, we are seeing more and more large customers moving to: (a) Continuous Integration and Continuous Delivery to accelerate delivery and (b) IT operations analytics as the way to ensure service delivery. IT operations analytics helps create the stable foundation for the high-speed and dynamic innovations of CI/CD.

With DevOps opening up an unprecedented level of speed, it’s important to have the right tools in place to process significant volumes of data. This is another area where IT operations analytics serves as an ideal complement to DevOps – because today’s IT operations analytics tools are powered by Big Data and machine learning, making it possible to achieve new levels of scalability. Unlike just a few years ago when getting a Big Data system up and running required significantly more resources, companies can get a lot more Big Data bang for the buck today than was possible even 3 years ago.

Now, companies are focusing on the analytic applications on top of Big Data, which is how businesses will find the optimization and innovation opportunities. Companies like Netflix are leading the charge in how Big Data can be used, not only to ensure service levels but also to create great customer experiences. While customer data may be the “crown jewel” of a corporation, IT operational data is the “diamond in the rough” that companies will be mining for the next big wins. We are just barely scratching the surface of what is possible with Big Data analytics of IT operational data – by drawing upon a fuller picture of what’s happening within the stack (various applications, servers, shared services, etc.), today’s IT operations analytics tools are going to help IT managers identify new insights and innovation opportunities that would not have been possible in the pre-DevOps era.

Another lesson of today’s reality of DevOps and IT operations analytics is that the divide that used to exist between customers and IT is almost gone. The future of IT is represented by the Internet of Things (IoT), where every system and behavior is monitored and analyzed in real-time, triggering operational processes from turning off a light to delighting a customer. Everyday objects, from household appliances to cars, are going to get exponentially “smarter,” and our IT processes need to evolve along with them to continually analyze, evaluate and respond to the ever-shifting ocean of data that is silently speaking to us. While DevOps and CI/CD define the trend to more rapid innovation, the collection and analysis of the reams of data generated by systems and customers will be the key to finding and maintaining competitive advantage.

This is where IT operations analytics will have an ever-more-important role as a complement to DevOps – with Big Data analytics and machine learning tools becoming more responsive and integrated into the life of overall IT operations, our IT admins are going to have more powerful insights at their fingertips each day. The convergence of IT operations analytics and DevOps will be a virtuous cycle of enhanced speed, smarter automation, and better control of IT operations for more reliable service and more delightful customer experiences.

About the Author/Omer Trajman

Omer Trajman, Co-Founder and CEO of Rocana. Omer’s fifteen years of front-line business and technical experience guide his vision for Rocana. Leading teams responsible for some of today’s largest modern database management system deployments, Omer has worked with customers and partners to identify where Big Data technology solutions address business needs. Omer’s experience in Big Data includes responsibilities as Vice President of Technology Solutions at Cloudera, focusing on Cloudera’s technology strategy and communication. Prior to this role, Omer served as Vice President of Customer Solutions at Cloudera, which included responsibility for Cloudera University, Cloudera’s Architectural Services and Cloudera’s Partner Engineering team. Before joining Cloudera, Omer was responsible for the Cloud Computing, Hadoop and Virtualization initiatives at Vertica. Omer received his Bachelor’s degree in Computer Engineering from Tufts University and was a visiting scholar at Oxford University reading in Computation and Electrical Engineering with a focus on large scale distributed systems.

August 19, 2015
Jut Operations Data Hub for DevOps

“You have to be able to ask questions of your data in real-time and then respond to the answers,” says Apurva Dave, VP of Marketing, Jut. Jut leverages live streaming, batch analytics, and visualization to enable enterprises to ask important queries of their operational data using the Jut operations data hub, which Jut created with DevOps goals (such as Continuous Deployment with Quality) in mind.

Jut supports real-time and historical data queries across all log, event, and metrics data. “Our platform is built on dataflow, dealing with large scale streaming data,” says Dave. Jut is an option for organizations that don’t want to go out and build their own operations data hub software.

Core Data Types

Jut intends to join data such as user activity (on websites and software), event data (triggered operationally such as support ticket filings, customer transactions, and software upgrades or migrations), and unstructured data (application and system logs) together for analysis, says Dave. Jut works with structured data such as metrics data (such as application level data like the user numbers and key action numbers or system level data like CPU utilization, memory utilization, and network I/O data), explains Dave.

Jut attempts to analyze the whole of your data to answer questions you may have like: “What resulted when I deployed a new version of my application into production?” Jut might answer based on an analysis correlating system resource utilization (“Was the app too resource intensive this time? More so than we expected?”).

Or you may ask Jut this question: “How did my code respond to user activity / user demand? How did it perform?” Jut could ask this question based on its assimilation of both unstructured and structured data. “It’s common for enterprises to A/B test two separate versions of a site or software feature. Using Jut, you can ingest those test results and analyze them in real-time to determine what version produces the desired effect,” says Dave.

Big Data Analytics & a Huddle about Juttle

“We have users ingesting tens of billions of data points per month. That number is growing fast,” says Dave. Jut targets big data management for easier analysis using data infrastructure techniques that simplify / streamline, integrate, and manage big data backend infrastructure. “We also use our own dataflow language to quickly iterate and visualize lots of data,” says Dave. All this is designed to ease DevOps teams’ work in accessing and assessing operational data to discern how the software and the business that it serves are performing.

Jut designed its Juttle dataflow programming language for data analysis and data visualization. And Juttle is dedicated to those tasks alone. “While you could try to cobble together the right packages and libraries in a general language like Python or Java to do these analytics, it would be relatively messy, time-consuming, and error prone,” says Dave.

Juttle is an extensible language enabling analytics and visualization. “Jut’s data scientist built an anomaly detection algorithm entirely in Juttle and then published it on GitHub for anyone to use inside Jut,” says Dave.

Juttle uses a structured approach to querying your data. “The approach looks like a series of piped UNIX commands, so anyone familiar with scripting languages such as JavaScript or query languages such as SQL should feel at home with Juttle,” says Dave. Jut responds to questions in seconds to tens of seconds (for larger data sets) based on iterative, interactive analysis.

Investigate, Question, Consider (Try?)

Outsourcing to Jut puts the enterprise in a position to focus on existing coding duties while proving whether Jut is the better operational analytics move for analytics performance and cost-savings (Jut offers a free trial). But as with the analytics themselves, ask a lot of questions of Jut. Do your homework before committing even to trial software (Jut has a demo and an online “playground” as well).

Tip: Jut made sparkling comments from its Beta-development-phase customers such as NPM and Mylio available. Rather than include those for your consideration, I suggest you reach out through your connections to these and other enterprises that are experimenting with this data hub software.

July 23, 2015
Optimize app performance with the help of real-time monitoring and analytics

Back in 1980 there was a movie called Nine to Five—referring to the commonly held perception of a standard work day. Of course, that work day evolved to be more like eight to six (or seven), and now it seems like the concept of a work day—or work hours at all—is a distant memory. In order to keep up with the instantaneous, always-on, connected 24/7 pace of today, companies need to be able to understand how their apps are used, and proactively address issues in real-time.

That’s where companies like MixPanel come in. MixPanel offers an advanced analytics platform for Web and mobile apps so that developers can keep tabs on when, where, and how apps are used, and any problems that might arise.

I spoke with Tim Trefren, co-founder of MixPanel. He proclaimed, “It’s easy to build a product. What’s hard is building a good product.” I assume that wisdom applies both to MixPanel itself, as well as to the customers that employ MixPanel to monitor and improve their own products.

Trefren told me that MixPanel provides a very flexible system that enables customers to track and monitor the metrics that matter most to them. It allows for custom metrics, so companies can choose to monitor elements of apps and app functionality that are completely unique. Customers can also implement A/B alternatives and compare metrics between the two to determine which is most effective.

Monitoring performance metrics, and analyzing that data to fix or improve apps fits into the broader concept of agile development, and rapid iteration that define DevOps. Trefren explained that some customers also employ MixPanel in an innovative way that also applies to DevOps—to actually monitor operational data.

Trefren says MixPanel may not be the ideal tool for monitoring operations, but it does enable admins to monitor everything from a single display, and it can be used to alert IT personnel when issues arise. For example, if MixPanel starts logging metrics that show an app is not being used at all, that could indicate that a server is offline.

MixPanel also recognized that it was in a position to do a little more. It offers a service that analyzes captured metrics data, and enables MixPanel to help companies understand who their customers are. Understanding the audience, and the actions they take (or don’t take) with an app provides valuable demographic data that companies can use to target marketing efforts with greater precision.

Trefren also told me that he is exceptionally proud of the engineering team at MixPanel. While the volume of data being analyzed has tripled in each of the last two years, the engineering team has only doubled over those same periods. That means that the MixPanel engineering team is working more efficiently, and getting exponentially more done in order to keep up with demand, but that’s another story for another day.

“The combination of power, flexibility, and performance set MixPanel apart from competitors. We’re able to do more, and do it more quickly,” said Trefren.

That may be true, but MixPanel isn’t alone. I wrote a while back about similar services from AppDynamics, including its DevOps War Room feature. Clearly, the rapid pace of development, and the sheer volume of apps have created a market for being able to monitor metrics, and measure the performance of those apps in real-time.

MixPanel offers its analytics as a freemium service. Paying customers are billed for metrics per data point, so the overall cost ends up being a reflection of the volume of data being analyzed.

January 8, 2015
New Relic Announces Groundbreaking Real-Time Analytics Platform
Cloud-based New Relic Insights collects, stores and processes trillions of events from web and mobile software to allow customers to easily and quickly answer the most important business questions
SAN FRANCISCO AND NEW YORK – March 19, 2014 – Software analytics company New Relic today announced the arrival of Insights, a real-time analytics platform that collects, stores and presents valuable data directly from modern software, and transforms the data into insights about customers, applications and the business. Delivered as a cloud-based software-as-a-service (SaaS) offering and using New Relic’s lightning-fast and custom-built Big Data database platform, New Relic Insights empowers application developers and business users alike to make ad hoc and iterative queries across trillions of events and metrics and get answers in seconds.

The Big Data Challenge

Traditionally, there hasn’t been a fast, easy or cost-effective way to collect, store and interact with real-time data to gain business or IT insights. With today’s fast-paced economy, businesses can no longer afford to wait hours or days for a static report that may not provide the data required for data-driven, software-powered businesses. Frequently, the data is out of date and ad-hoc queries take hours to complete, limiting what and how many questions decision makers can ask – making data-driven, iterative decision making impossible.

New Relic Insights to the Rescue
- Collect data about your software, customers and business directly from your software using New Relic’s intelligent agents
- Store volumes of data in New Relic’s super-cluster and take advantage of a custom-built database to query billions of metrics in seconds
- Present visualizations and dashboards immediately with the New Relic Query Language (NRQL), a simple SQL-like language that automatically generates data visualizations, so even novice users get answers fast
New Momentum for the Category of Software Analytics

New Relic Insights is only one offering from New Relic within the much broader Software Analytics category. Every modern business is software-driven and interacts with its customers digitally whether via mobile or the web. As a result, software holds answers about your customers’ behaviors and actions, which reflect on the state of your business and your relationship with customers. There are five components that make up Software Analytics:
- Real-time Analytics Platforms – Collect trillions of events from applications and allow users to query the data easily to answer tough business questions
- Application Performance Management – Developers and IT operations groups monitor and troubleshoot the performance of their web and mobile software
- Operational Intelligence – IT operations groups analyze and identify trends in operational systems
- Infrastructure Management –IT Operations groups monitor and troubleshoot the performance of IT infrastructure
- Business Optimization and Analytics Apps – Business user groups optimize their job functions
Ways to Gain From New Relic Insights
- Product Management – Make a query about the adoption of new product features that were recently launched, determine which customers are actually using the features to enable actions and next-steps.
- Marketing – Find out if the new marketing campaign is a homerun or a strikeout in real-time. If it’s a homerun – spend more. If it’s a strikeout, kill the campaign before the budget is blown.
- Sales – Track customers’ experiences and product usage during a free trial. Immediate insight into key stakeholders and issues that they might have with the product.
- Customer Loyalty and Support – Manage customer engagement and happiness through each click of their mouse. Understand why a customer makes a call to support before they make the call.
- Application Developers and IT Operations – Track customer interactions with a site over the previous 10 minutes to find the root cause of a support issue that cannot be reproduced.
The New Relic Insights Beta Program Highlights
- This new Real-Time Analytics Platform (previously code-named Project Rubicon) is now in public beta. More than 275 customers have participated in the private beta to-date. All paying New Relic customers get immediate access to the new product for free for a limited time.
- Each account gets 1 billion metrics stored for up to seven days for free during the beta period. More than 1 trillion events have already been stored in the new product over the last month.
Supporting Quotes

“New Relic Insights is like a data Swiss army knife, a tool you can use for a multitude of analytics needs and opportunities. For any modern publisher, analytics is omnipresent with millions of data points – think March Madness and World Cup 2014 just for starters. Bleacher Report sees New Relic Insights adding significant engagement opportunities with our sports fans around the globe.” – Sam Parnell, CTO, Bleacher Report

“The New Relic API and the software analytics now available through New Relic Insights are allowing us to analyze and iterate feature changes to our website and donor experience in sophisticated A/B testing that wouldn’t be available any other way with any other tool. Software Analytics is helping us save the lives of children in countries like Zambia and Uganda, all made possible by New Relic.” – Joel Worrall, CTO, CURE

“New Relic Insights is a spotlight that rapidly uncovers major insights about our properties — and is an invaluable tool for both our tech and marketing teams. The custom dashboards and ability to add custom attributes allow us rapidly to ask better questions and quickly learn from those, so we can dig further and make better business decisions — both for the platforms as well as the user experiences they support. The monitor-test-get results cycle has been reduced from two weeks to daily (or even intra-day) — the speed with which you can build a query and get a response has created a near real-time intelligence capability. Finally, the instrument-launch-monitor cycle now fits in our agile development and marketing processes.” – Paul Ryan, CTO, Yellow Pages Group, Canada

“I am so excited that New Relic Insights is now a reality, a completely new type of product that signals New Relic’s second act. It builds upon our ability to collect, store and make sense of billions of metrics and offers the ability to instantly query live production software, iterate on the fly and act upon the results. I truly think New Insights will transform the way businesses use data from their software to make better decisions.” – Lew Cirne, founder and CEO, New Relic

Online Resources
- Learn more about New Relic Insights by clicking here
- Discover more about the Software Analytics category with the paper, “Learn Why Your Software Should Rely on Software Analytics,” by The Big Data Group, an independent market research firm
- Enterprise customers can access the independent market research firm Enterprise Management Associates paper, “New Relic Unveils New Relic Insights: Software Analytics for Business Insights” here
- See how New Relic Insights helps developers in this infographic here
- Read CEO and Founder Lew Cirne’s blog post here and get lots more details about how New Relic Insights works as well as a technical perspective on New Relic Insights here
About New Relic

New Relic is a software analytics company that makes sense of billions of metrics about millions of applications in real time. Our comprehensive SaaS-based solution provides one powerful interface for web and native mobile applications and consolidates the performance monitoring data for any chosen technology in your environment. Our 90,000 customers use our cloud solution every day to optimize more than 200 billion metrics for 3 million applications. When your brand and customer experience depend on the performance of modern software, New Relic provides insight into your overall environment. Learn more at newrelic.com.
March 19, 2014