devopsy.kinsta.cloud

Tag: big data

  • Building Your Own AIOps Platform Is a Bad Idea

    Building Your Own AIOps Platform Is a Bad Idea

    It takes considerable expertise to not only build, integrate and maintain an AIOps platform

    In the wake of the economic downturn brought on by the COVID-19 pandemic, investments in digital business transformation have accelerated. The applications that drive those processes are not only highly distributed, but they also operate at a level of scale that no IT team can manage using legacy approaches to managing IT. It can take weeks to discover the root cause of any issue.

    Enter AIOps. Machine learning algorithms make it possible to reduce the time it takes to resolve any issue and enable IT teams to continuously optimize IT environments at any scale. Many aspects of AIOps, however, are still largely unexplored. Rather than opting for proven platforms, some IT teams are building custom solutions in-house.

    I have experienced firsthand several large enterprises that have embarked on this risky journey, including one Fortune 500 company that asked a partner to help develop a solution and ultimately deployed a commercial product. Enterprise IT teams go down this road to solve a specific tactical problem, such as alert noise reduction. In many cases, all the time and effort put into a do-it-yourself (DIY) project simply winds up being wasted.

    Why DIY Platforms Usually Fail

    Given the prevalence of open source AI tools and frameworks such as TensorFlow, Theano or the Microsoft Cognitive Toolkit (CNTK), it can be tempting to build your own custom AIOps platform. It takes considerable expertise, however, to not only build an AIOps platform but also integrate it into an enterprise and maintain it.

    Here are the leading reasons why in-house developed AI projects are risky:

    • You’ll need a properly constructed data lake: AIOps platforms require access to data residing in multiple technology silos in real-time. IT teams that build their own AIOps platforms need to make sure they are gathering all the right log data, metrics and traces alongside data collected from IT service and incident management platforms. These comprehensive data sets are needed to train whatever machine learning framework is in place, which is often selected at random. Invariably, that means building or buying a costly big data platform to create a data lake to store all that data. A poorly constructed AIOps platform will be worse than the proverbial disease it’s meant to cure because the insights don’t accurately reflect what’s actually occurring in the IT environment. Do you have the funds for this and experienced data science experts onboard to get this right?
    • Designing AI-enhanced workflows is unlike other workflows: Getting the data is just the beginning. Determining how the system behaves and affects existing workflows is the next step. IT teams must decide to what degree they merely want the AIOps platform to passively surface recommendations based on what’s observed versus automatically resolving issues based on defined parameters.
    • Deployment is complex: After developing a few AIOps algorithms to produce meaningful results, the next step is to determine how to deploy it in a resilient and performant architecture. What other systems does it need to integrate with and how will results be monitored and viewed?
    • Monitoring user impact is critical: How will end users interact with the algorithm and what is the ideal UI/UX and workflow? How will feedback be provided by end users for improvement and adoption success?
    • AIOps support and maintenance is not a project, but a team. Ultimately, an internal IT team would need to build the equivalent of a product which needs ongoing maintenance and support. The total cost of the custom platform starts to rise as the bulk of the IT team could wind up spending most of their time managing the AIOps platform instead of making continual improvements. Even if the IT team has the expertise required to build an AIOps platform, there’s no guarantee those individuals will always be available to maintain and update it. Very few IT professionals spend their entire career at one organization.
    • Keeping pace with marketplace innovation: Finally, AIOps as a field is still relatively nascent and the startup community has hundreds of millions of dollars in VC backing to support R&D. Advances are being made at a rate most internal IT teams can’t keep up with, let alone evaluate and vet on their own.

    How to Safely Encourage AIOps Exploration

    There’s no substitute for knowing where an organization needs to go and how to get there. A commercial AIOps platform incorporates all the best practices that have been defined by legions of IT experts, along with these benefits:

    • Faster time to value: You can embark on the AIOps journey much sooner. A commercial AIOps platform will begin surfacing insights in a matter of weeks. It will take an internal IT team months to build an equivalent platform with no guarantee of success. Time is better spent on user adoption and adding and refining use cases for business benefit.
    • Seasoned experts: A commercial platform provides immediate access to not only a proven framework but also, AIOps experts who can troubleshoot and optimize issues quickly. There’s almost no AIOps challenge they haven’t seen before.

    How to Measure ROI From AIOps

    Savvy organizations that invest in AIOps are primarily betting on a better way to manage IT that will enable them to accomplish more as a business. The real value proposition of any AIOps platform is that it enables an existing IT team to do more not by just eliminating rote tasks but also making it possible to deploy more applications reliably without adding IT staff. It’s worth remembering that the cost of labor continues to be the single biggest IT expense.

    The return on investment from an AIOps platform can be easily calculated by measuring:

    • The number of incidents resolved in a given period of time.
    • The size of the IT operations/incident management staff before and after an AIOps platform is deployed.

    Arguably, however, those savings pale in comparison to the opportunity costs of using advanced technologies which can increase operational intelligence and deliver new business value in the form of excellent user experiences and high-performing digital services. IT organizations that don’t embrace AIOps will be unable to compete with faster, more nimble rivals that have modernized their IT processes. The easier, more affordable and least risky way to go about this is by deploying a solution already proven and in market.

  • Logz.io Launches COVID-19 Dashboard

    Logz.io Launches COVID-19 Dashboard

    Logz.io this week launched a Community COVID-19 Dashboard Project that seeks to provide software developers and data scientists a reliable source of data to assess the global impact of the pandemic.

    Dotan Horovits, product evangelist for Logz.io, said the goal is to help organizations make sense of often conflicting data by making it accessible using a publicly available dashboard based on open source Kibana software. The data itself is stored in an ElasticSearch repository, he said.

    Logz.io also plans to make the data accessible via a dashboard based on open source Grafana software to make it easier to navigate time-series data, he added.

    Horovits acknowledges that the data concerning the COVID-19 pandemic is fluid at best. However, over time it will become easier to identify suspect data by applying analytics. Much of the COVID-19 data being made available is well-formatted, he said; the challenge is simply finding a way to aggregate that data in a way that is accessible to a broad range of potential end users.

    In the short term, Loz.io envisions organizations such as first responders using the dashboard to help optimize the deployment of limited resources. Longer-term, that data can also be employed to assess the impact the pandemic has had on everything from the environment and the economy to the mental health of individuals living in different regions. At the very least, access could go a long way to reducing the amount of COVID-19 misinformation being spread, Horovits added.

    Logz.io started with the COVID-19 data available from Johns Hopkins University. On top of that core set, the company is moving to aggregate data made available from additional public sources including media, governments and the World Health Organization (WHO). The company is also inviting any organization that has aggregated COVID-19 data to contribute to the project.

    Many organizations will, of course, use the data provided by Logz.io and similar initiatives to build a wide range of analytics applications. For example, it may be possible to avert future pandemics by identifying areas where individuals are experiencing high fevers sooner. Given the importance of those applications, it’s probable most of them will need to be built using best DevOps practices to ensure timely delivery.

    It’s not clear how the pandemic may advance data science. Organizations have been leveraging big data analytics applications for years. However, the pandemic may lead to more ambitious efforts to aggregate and analyze medical data on a global basis. Arguably, it’s been the absence of reliable data that has enabled the virus to spread so widely. Otherwise, initiatives such as social distancing might have been enforced earlier and medical facilities would have been outfitted with more equipment.

    In the meantime, there may not be much most IT professionals can do to combat the current pandemic other than working from home much like everyone else. However, in terms of preventing the next one, the level of data science expertise being applied to virus research may soon become a matter of life and death for millions of people.

    — Mike Vizard

  • DataOps and Beyond: How DevOps Methodology Transformed Our Approach to Data Science

    DataOps and Beyond: How DevOps Methodology Transformed Our Approach to Data Science

    The DevOps methodology has become synonymous with forward technical thinking–a workplace culture that reinforces best cultural practice and promotes more, better quality output by synchronizing the functions of development and operations teams. As a testament to its popularity, the term has expanded in reach to refer to a broader set of transformational software-related practices. The list continues to grow: DevTestOps, DevSecOps, GitOps, DataOps. 

    In 2020, organizations are drawing upon thousands—if not tens of thousands—of data sources, which exist in hundreds of different formats and fall under unique sets of restrictions and compliance regulations. Figuring out where all this data comes from, what it means and how to ensure that the right teams can access it when they need to, are common data management challenges. To address these challenges, some solutions were borrowed from effective practices in software development; automating analytics for improved quality, and allowing data scientists to shift away from time-consuming tasks to focus on value-adding projects instead. 

    What happens when software development and deployment methods are applied to processing and integrating data? Does that mean that DataOps is exactly like DevOps, but for data? And what benefits do organizations expect from adopting this approach to data management?

    The xOps Mindset

    At their core, the ops practices are linked by fundamental characteristics that shape their goals and practices. Let’s explore some of these commonalities:

    • xOps ties together workflows from create to run: In an ideal environment, production and distribution don’t function as separate entities, but are synchronized to meet target metrics that can include hundreds, or thousands, of daily deployments. Culture changes have rewritten rules around internal communication and feedback loops in a development environment where the conditions or requirements for success are constantly changing.
    • xOps automates the mundane (non-creative) tasks: Automation not only increases an organization’s agility, but when executed correctly can eliminate communication gaps between teams—for instance, automating traceability and issue tracking within code changes. Within larger, more complex projects, automation can save on time and cost by executing and managing CI/CD—ensuring that software builds and changes reach their designated target environment with little to no human involvement.
    • xOps builds in best practice: Equally as important to the functions they perform, xOps tooling plays a key role in building in best practice from the top down and bottom up. Best practice might not look identical between organizations, but the common goal is delivering value for a range of stakeholders through continuous production, in an environment where it’s encouraged to learn through experimentation and to respond to problems as they occur.

    In short, DataOps loosely applies the DevOps process to data pipelines—using automation and Agile methodology to reduce the amount of time spent fixing issues in pipelines, and to get data science models into production quicker. There are also some obvious differences between DataOps and DevOps. While DevOps refers to the collaborative process between two technical teams, DataOps facilitates cooperation between data analysts, engineers, scientists and any members of an organization who use data. This makes DataOps a much more multifaceted process than its DevOps cousin. 

    xOps for the Big Data Age: Benefits of Running DataOps 

    We know that for many organizations, converting raw data into usable, accessible and relevant (that is, profitable) business intelligence has given rise to a plethora of data management issues. The sheer volume of most enterprise level datasets has simply outgrown many of the inadequate older systems and processes built to manage it, and we continue to produce new data at stunning rates. There is a general consensus among data scientists today that the amount of data in the world doubles every 12 months.  

    • Automation and Continuity: Within an organization, business data moves through a set process—akin to the software lifecycle for the DevOps analogy—where data is ingested in one form and exits in another. Engineers must build data pipelines, test them and change them before they are deployed. By adopting the standards for automation and best practices outlined above, you’ll ideally have a constant stream of data flowing through a pipeline. This unlocks the potential to obtain real-time insights from data, shortening the length of time it takes to turn raw data into valuable business information.
    • Cultural Shift: When completed correctly, DataOps changes who within an organization becomes a data user. A higher level of transparency and an assurance of quality means that data is more openly available to decision makers within non-technical teams. Consider the business value-add for a sales team who wants to expand into new markets and are able to access a wealth of customer data.
    • Machine Learning: We talked about the role of feedback loops in the xOps mindset. When machine learning modelling meets this mindset, you improve the quality of the model being produced because you’re unlocking a higher quality of data through version control, continuous integration and continuous deployment. Improved insights in machine learning offer nearly unlimited potential for extracting value from DataOps.

    — Hannah Leslie

  • Enterprise AI Books to Read This Spring

    Enterprise AI Books to Read This Spring

    If you are anything like me, and curious about how new enterprise tools will transform our relationship with work and business-technology, then you are probably reading a lot about artificial intelligence. There’s likely no other topic that’s been written about in more depth in recent years, yet much of it is unrealistic or poorly reasoned hype.

    This is why I’ve been reading more books from trusted authors on the subject. In previous years, I’ve read books such as Erik Brynjolfsson’s and Andrew McAfee’s Race Against the Machine, and Martin Ford’s book Rise of the Robots: Technology and the Threat of a Jobless Future. Both were great books that brilliantly framed AI. Last year, I wasn’t as fortunate.

    Last year, my reading included The Master Algorithm by Pedro Domingos. It was an interesting look at machine learning algorithms, and specifically the quest for the algorithm—the master algorithm—that would create all future algorithms without the need for humans. This book proved a disappointment because the author seemed as convinced, almost dogmatically so, that general AI success was as close around the next corner as the AI pioneers of the 70s and 80s believed success for them was around the next corner.

    The next book up for me was Life 3.0 by Max Tegmark. While this was a good read, it proved to be too much of an analysis of what has been already written than a book that significantly pushed the subject of AI forward in a new or unique way.

    Hopefully, my next batch of AI reads will prove better. I realize I’m playing catch-up here, as most of these aren’t new books. But they do look to be important books on the subject.

    Here’s what I currently have lined up.

    AI Super Powers

    Author: Kai-fu Lee.

    Following the defeat of the world’s top player of the game Go, to Google’s AlphaGo AI, the government of China set ambitious plans to become the global AI hub by 2030. In his book, Kai-fu Lee contends that China has quickly caught up to the U.S. and that dramatic changes from AI developments are happening much more quickly than many expect.

    In his book, Lee examines universal basic income, and examines what jobs may be enhanced with AI the possible solutions to the biggest changes AI promises to bring to us all.

    Artificial Intelligence: A Modern Approach 4th Edition

    Authors: Peter Norvig and Stuart Russell.

    At 1152 pages, Artificial Intelligence: A Modern Approach (due out this April) isn’t a light read. The latest edition of this book looks across the entire field of AI, and a deep dive into machine learning, deep learning, transfer learning, multi-agent systems, robotics, natural language processing, causality, probabilistic programming and more.

    I have high hopes for this book, as it’s treatment appears to be exactly what I’m looking forward to reading: an overview of the state of AI without going too light on the treatment of each topic.

    Applied Artificial Intelligence: A Handbook For Business Leaders

    Authors: Mariya Yao, Adelyn Zhou and Marlene Jia.

    This book is made the list because it’s AI as it can be applied to business-technology. It promises to be a roadmap on how to use  data, technology, design and staff to solve enterprise business problems.

    “We teach you how to lead successful AI initiatives by prioritizing the right opportunities, building a diverse team of experts, conducting strategic experiments and consciously designing your solutions to benefit both your organization and society as a whole.”

    That’s exactly what enterprises need to do to succeed at integrating AI into their companies in order to get value from this growing enterprise technology.

    Machine Platform Crowd

    Authors: Andrew McAfee and Erik Brynjolfsson.

    I learned and enjoyed both Race Against the Machine and The Second Machine Age. The latest book from MIT Principal Research Scientist Andrew McAfee and Director of the MIT Center for Digital Business Eric Brynjolfsson, looked at the impact of machine intelligence and big data. In this book, the authors look at another form of augmented intelligence: our collective intelligence, and what it means for transportation, medical research, financial services and more.

    That’s it for the reading list for now. I’d appreciate hearing what AI-related business books you are reading this spring and summer.

    — George V. Hulme

  • Why Enterprise Survival Depends on a Context Driven Approach to Data

    Why Enterprise Survival Depends on a Context Driven Approach to Data

    Enterprises are sitting on an enormous amount of data that could be fueling their automation journey. Much of the information needed to run a business is existing data. However, much of an organization’s data is inaccessible and unsearchable. This so-called “dark data” is hidden away in paper forms, spreadsheets, emails and literally billions of PDF documents. This dark data is locked away and mostly unusable by the systems and AI required to power the autonomous business of the future. Enterprises are trying to achieve 21st-century outcomes while being shackled to 20th-century data practices. 

    I recently read a report from NewVantage Partners on a survey they did concerning how big data and AI were driving digital transformation. While 92% of the survey respondents said they were increasing investment in big data and AI, there were some interesting findings that suggest organizations are not realizing the ROI they had hoped. For instance:

    • Only 31% have a data-driven organization.
    • Only 28% stated they have a data culture.
    • 53% are not treating data as a business asset.
    • 38% have not realized any measurable outcome from their investment.

    Now, there’s a whole list of reasons and hurdles that are preventing organizations from realizing better value and outcomes from their efforts to become data-driven. But, could the underlying approach to becoming data-driven be flawed? Could organizations be struggling because they continue to apply a legacy mindset and approach to data acquisition and storage? Could these challenges be a contributing factor in 70% of digital transformation projects failing?

    Just take a look at the growing number of stories, market reports and surveys on digital transformation and you’ll see a graveyard of failed projects, lack of success and high turnover rates. The current approach to digital transformation is not working. 

    In our push to become data-driven, we capture data and attempt to transform it into computable bits and then stuff the data into rigid structures. This legacy approach to capturing and representing information outside the human brain results in a proliferation of single-use data that doesn’t scale with the business. We always lose some of the meaning, relationships and context. Despite all the advancements in technology, we’re still trying to model the information based on how we’ve historically stored data. We’re still producing flat data.  

    In the autonomous enterprise of the future, and the digital workers that will run them will need rich, contextual data. The true power and value of digital transformation will only be unlocked by the adoption of a semantic data-driven approach. The future of work will be built on digital workers powered by machine learning, artificial intelligence and knowledge graphs. It’s not enough for an organization to be data-driven anymore–they need to become contextually-driven. We need to start capturing the meaning to achieve a better understanding of the data.

    “I liked the idea that a piece of information is really defined only by what it’s related to, and how it’s related. There really is little else to meaning,” said Sir Tim Berners-Lee, the inventor of the World Wide Web.

    Context is all about the meaning. Context is more than just a piece of data. Context is the background in which an event, task or process takes place. It is the deep, multi-dimensional understanding of data and the relationships between the entities. Understanding the context can lead to accelerated insight through AI and machine learning. To be competitive, enterprises need more than data; they need context to enhance productivity and efficiency. But our current approaches to capturing and storing data have historically failed to capture this valuable context. As the amount of data grows exponentially, organizations will realize semantic data is key to survival.

    — Kevin Harbauer

  • Stellus Launches Next Generation NAS Platform

    Stellus Launches Next Generation NAS Platform

    Stellus Technologies launched a network-attached storage (NAS) system capable of processing unstructured data at 10 times the speed of existing systems.

    Based on a software-defined file system written on top of a key-value store, the Stellus Data Platform can be acquired either as an appliance or in the form of a virtual image that IT teams can deploy themselves on-premises, in the cloud or at the network edge.

    Ken Grohe, senior vice president and chief revenue officer for Stellus, said the operating system created for the Stellus Data Platform has been under development for the past five years thanks mainly to $70 million in funding from Samsung. As a provider of memory technologies widely used in storage systems, Samsung has a vested interest in advancing the adoption of next-generation storage systems capable of supporting a wide array of emerging applications, he said.

    NASThe appliance Stellus provides is built around 8TB hot-swappable, non-volatile memory express (NVMe) solid-state drives (SSDs). The base configuration is delivered in a 5RU2 rack that comes with 184TB of storage. The largest configuration comes in a 17RU rack to provide access to 1.475PB of storage capacity. The Stellus Data Platform can deliver throughput rates of 20 to 80+ GB/s reads and 19-70+ GB/s write throughput, according to the company.

    The Stellus Data Platform is designed specifically to address the needs of emerging applications in, for example, the medical and life sciences sector accessing massive amounts of data that existing NAS systems are either too slow or costly to address. The adoption of most of those applications are now being driven by DevOps teams rather than traditional storage administrators, Grohe said.

    Just as significantly, he said, the Stellus Data Platform is designed from the ground up to allow IT organizations to scale-up performance and capacity independently. That “Scale Through” architecture means IT teams don’t have to add additional “dead wood” to scale performance in the form of additional storage units that end up having only a small amount of data loaded on to them, said Grohe.

    In addition, the key-value store at the core of the platform enables searching and identifying unstructured data using metadata to eliminate the need for data caches and global data maps.

    Most big data analytics and artificial intelligence (AI) applications are deployed in the cloud today in part because legacy storage systems in on-premises environments can’t support them economically. The limiting factor, however, has not necessarily been the underlying technology, but rather the need to be backward-compatible with existing storage systems. As a startup, Grohe said Stellus enjoys the luxury of not being encumbered by a classic case of the innovator’s dilemma.

    It’s too early to say to what degree Stellus might supplant rivals such as Dell Technologies, NetApp and Hewlett-Packard Enterprise. Grohe said Stellus has signed up more than 500 customers to participate in its beta program. The challenge going forward will be determining to what degree Stellus will be employed to support a growing niche of emerging applications versus becoming a new foundation on which a next-generation NAS platform will be used to also support mainstream applications.

    — Mike Vizard

  • AIOps: The Path to Greater Resilience and Uptime

    AIOps: The Path to Greater Resilience and Uptime

    Large organizations with complex IT infrastructures are facing a significant challenge when it comes to data. The complexity is such that many IT operations teams are struggling to detect, or predict, the complex failure conditions and circumstances that can adversely impact resilience and uptime.

    Monitoring a multitude of systems is a complex task, as the data generated by any system can be vast and far too overwhelming for manual IT operations processes, particularly in relation to analyzing, correlating and prioritizing alerts and remediation.

    Companies have much to gain from a better understanding of the data being created from monitoring tools and practices: increasing the resilience of systems, and the speed of remediation cycles delivers significant cost and time savings. Harnessing advanced technologies to deliver more intelligence to teams in charge of performance monitoring is becoming increasingly important, which is where AIOps can deliver significant value on top of existing monitoring and automation tools.

    Cutting Through the Noise

    Telemetry tools deliver staggering amounts of data. The Microsoft Azure telemetry platform, for example, records 10 petabytes of data per day. When dealing with this volume, the ability to intelligently decipher the severity of alerts becomes vital, especially as many of them may be duplicates or not meaningful in any way. Yet it is IT operations teams that are tasked with understanding how systems are performing, and where resources should be diverted to ensure services are always up and running. To do this effectively, the root causes of alerts must be identified quickly and correctly.

    AIOps is increasingly becoming the go-to solution to the problems inherent in dealing with big data. By applying data collection, data modeling and data analytics techniques and using machine learning algorithms to establish patterns, the noise of the data is reduced as operations teams gain more intelligent insights into alerts. So, providing more context to issues flagged within systems is the goal. Eventually, somebody has to take action when an alert is created, so the more intelligence engineers have, the easier it will be to remediate.

    The less context delivered with alerts, the more time engineers will need to resolve the issue, as the manual remediation process will likely involve interaction with other teams. Take application performance monitoring as an example. In my experience, operations teams do not have an end-to-end understanding of the applications that are live in the production environment. If an alert is created, the ops team will typically seek assistance from a member of the developer team who built it, or another colleague with a deeper understanding of the application, which of course extends the lead time of the remediation process.

    Key Considerations for Deploying AIOps

    By establishing data patterns through machine learning and other advanced analytics technologies, the following key drivers for alerts assessment using AIOps can be established:

    Frequency: Discovering how regularly alerts are created and whether it warrants automation for analysis, correlation, prioritization and remediation.

    Coverage: Telemetry is of course vital to AIOps, as the machine learning algorithms will only be as good as the data they ingest. Putting in place the right telemetry tools is therefore vital to ensure the right data is being gathered. Failing to do so increases the risk of missing critical issues.

    Impact: Not all problems require equal energy and resources. Certain issues might only occur every few months and may not be critical. In such situations, it is not prudent to invest significant amounts of time and money into developing the machine learning algorithms that will identify where and when an alert will be created. Again, cutting through the noise of the telemetry will help to identify high-priority alerts.

    Probability: What is the likelihood of certain issues recurring? Can automation tools, informed by the machine learning algorithms, be implemented to deal with these, i.e. learn where and when they are likely to occur so they can be dealt with by the AIOps platform autonomously?

    Take, for example, the issue of CPU or memory going beyond a certain threshold. Rather than manually contacting customers when an issue is flagged to inform them that services are about to go down before being spun up again, an AIOps platform can automate this process, so remediation happens seamlessly.

    Latency: Automation of the remediation process for an alert deemed critical requires increased investment in AIOps. Getting good data from the early stages of AIOps processes will reduce the amount of time taken to both flag and deal with a problem. Containerized microservices, for example, take very little time to recover, so very little investment is needed when applying AIOps tools that automate remediation. Database recovery, however, is a more complex process, the automation of which would require significant investment.

    What’s the Customer Value?

    For service providers, AIOps is essential if they are to attain the highest levels of resilience, the measure of which all comes down to the Service Level Agreement (SLA). The SLA is measured by a percentage that corresponds to the time systems and are up and running over a year. So, 99.99% uptime is good, 99.999% is great, but 99.9999% is where service providers want to be. To put this into context, downtime for an entire year in the latter scenario amounts to just 32 seconds.

    Increased automation is the only way to achieve optimal SLA. To do this, service providers are increasingly turning to AIOps solutions.

    Operations teams are in charge of the IT strategies that align with business objectives. For service providers, a key part of this is, of course, ensuring maximum availability and optimal performance of the customer-facing environment. Understanding when and where faults appear, and improving resilience against them, is the best way to ensure this. That is where AIOps truly adds value.

    — Prashant Jain

  • CAP Theorem No Match for Mainframes

    CAP Theorem No Match for Mainframes

    One of the biggest problems in modern distributed computing is data management. These days data are spread out all over the place. As a result, getting the data you need, exactly when you need it, becomes a problem. Wider distribution creates more network latency. If you have a system in which some machines are in California and others are in New York, logic dictates it’s going to take more time for data to travel coast to coast than from within the same geolocation. It’s a matter of common sense.

    The data management problems created by distributed systems became more pronounced with the growth of the internet, which is, by nature, highly distributed. Eric Brewer, a computer scientist at the University of California, Berkeley, published a paper in 1999 that describes the dynamics of data management in a distributed system as a set of immutable constraints. Brewer published the work as the CAP principal. Three years later Seth Gilbert and Nancy Lynch at MIT proved the principal to be a theorem. Today, the CAP theorem is a central consideration when designing and implementing applications at web-scale.

    Simply put, the CAP theorem asserts that when your data is distributed over a variety of machines and geolocations, if you want the latest, most accurate data, you’re going to have to wait. And, if you don’t want to wait, the data you get will probably be stale. However, if data resides on a single machine, then everything changes. In such a case, you can have the latest data on demand, at any time.

    Of course, having a single data source implies having a machine that supports the demands of web-scale. Sadly, the x86 machine has yet to be made that can fit the bill. But a mainframe can. And, not only can it handle an enormous operational load, under the right set of circumstances it also can be a cost-effective way to meet the need at hand.

    Allow me to elaborate.

    The CAP Theorem Explained

    Before we go into the details as to why a mainframe computer such as an IBM Z or LinuxONE can make the problems presented by the CAP theorem go away at a reasonable price, let’s look at the theorem itself.

    Essentially, the CAP theorem asserts that of the three aspects of data management in a distributed system—consistency (C), availability (A) and partition tolerance (P)—you can have only two at any point in time.

    Figure 1: The CAP theorem dictates that of the three aspects of data management in a distributed system, only two are available at any point in time.

    In other words, if you want the latest information (consistency) from your data source immediately (availability), that information needs to be stored in a single, fail-safe location (partition tolerance). If you want to spread the data out over many machines (partition tolerance), either the data will be accurate at some point in time but not immediately (consistency) or you can have data immediately but it might be stale, awaiting an update to the most recent state (availability).

    The CAP theorem is more than theoretical. It has an actual operational impact on companies. A telling example is a problem that Netflix experienced when implementing its Monet system.

    Netflix Gets CAP-Ped

    Back around 2018, Netflix decided to implement an internal ad management tool under design using GraphQL, an open source API specification originated at Facebook that was becoming popular in many companies that operate at web-scale. In fact, GraphQL is still growing in popularity. So Netflix decided to take the plunge toward adopting GraphQL with an internal project named Monet.

    The way Netflix decided to implement Monet was to have the GraphQL API server act as a facade for existing RESTful APIs. The thinking was to reap the benefits provided by GraphQL, yet reuse as much existing code as possible. The idea was good, the actual deployment fell short. Once Monet was put into production, it ran dog slow.

    It turns out the REST APIs were distributed among a variety of data centers (see Figure 2). Network latency was killing them.

    Figure 2: Initially Netflix’s Monet system queried data from a variety of data centers.

    The company’s solution was to bring all the REST APIs into a single data center (see Figure 3).

    Figure 3: Netflix improved Monet’s performance by moving all RESTful APIs into a single data center.

    The result? Once all data resided in the same data center, performance improved dramatically. It makes sense. Shorter distance to travel means less latency in the system overall.

    Now, how this relates to the CAP theorem is interesting. Remember, according to CAP, as partitioning goes down (P), consistency (C) and availability (A) go up. So, while it’s true that Netflix was not using a single machine as a data source, moving everything into a single data center created what one could consider a single virtual data source. Yes, there might be latency issues within the data center. But, rack-to-rack latency is nowhere near as high as the latency between geolocations. We’re talking the difference of 1 to 10 milliseconds as opposed to hundreds, if not thousands, of milliseconds.

    In Netflix’s case, the company addressed the constraints imposed by the CAP theorem by reducing the breadth of partition at the geolocation level. All the data lived in the same data center. Imagine if partition size could be reduced at the machine level. What if instead of having data distributed among many machines within the same data center, data actually resided within a single machine? Partition size would be reduced to one. Then, applications could have the latest data immediately. All that’s needed is a single machine that can handle the load. This is where using a mainframe comes into play.

    Mainframes Are Built for Big Data and Agile DevOps

    Mainframes since their inception were intended to be able to store and process enormous amounts of data. Today 80% of the world’s data still resides on mainframes. As mentioned in a previous article about this topic, one mainframe can process 2.5 billion transactions daily. That’s the equivalent of 100 Cyber Mondays in a single day. That’s a lot of computing power easily equivalent to more than a few racks of commodity hardware in a data center. In fact, in some situations, one mainframe can provide the computing power found in an entire data center.

    So, let’s go back to the Netflix problem. What if instead refactoring those distributed RESTful APIs into a single data center, the time and money spent was used to refactor those RESTful APIs into a single mainframe? Yes, there are some technical hurdle to overcome, but considering the running Linux on a mainframe is standard these days, it’s not as if we’re talking transforming a bunch of Python code into COBOL. Databases from standard RDBMS such as DB2, Oracle to the NoSQL MongoDB can run perfectly well on a mainframe. There’s work to be done, but reinventing the wheel is not one of the tasks. Using a mainframe is a viable solution except for one sticky little problem: The conventional thinking is that using a mainframe is prohibitively expensive. Thus, the bias toward racks of commodity hardware or simply going with a cloud service.

    Mainframes as a Cost-Effective Solution

    The general perception in most IT shops is only big banks, insurance companies, governments and large-scale manufacturers use mainframe computers. They’re just too expensive for small to medium-size business. While many mainframes come with a price tag that can exceed $1 million, there are still surprisingly affordable models. For example, an entry-level IBM z114 will run about $75,000.

    If you’re a company spending $50,000 a month on AWS or Google Cloud computing resources, then for the $600,000 a year cost you’re spending on a cloud provider and the latency that comes with it, your company can well afford not only an entry-level IBM Z system to handle a production load, but also a second one to serve as backup, should failure occur.

    In terms of system administration, just about all the tools commonly used to support modern CI/CD processes today in the DevOps world have corresponding equivalents for mainframe implementation. As with any adoption of unfamiliar technology, there is a learning curve. But, again, we’re not reinventing the wheel. The tools, resources and expertise exist. Adoption is just a matter of following standard change management practices typical in medium-to-large-size companies that are well-positioned and use a mainframe to take advantage of implementing a cost-effective, no-partition data source strategy.

    Integrating Mainframe Technology into a DevOps-Driven Ecosystem

    Mainframes are capable of doing DevOps in an open, agile and collaborative fashion. The tools, resources and expertise required to support a mainframe’s participation in CI/CD processes exist today. For example, Zowe is an open source software framework that provides solutions that allow development and operations teams to securely manage, control and develop on the mainframe like any other platform. DB2 DevOps Experience for z/OS is one of the first commercial products built on Zowe. It allows IT the control over the mainframe database, its business rules and standards, while allowing development teams the ability to work independently using self-services and integration with modern CI/CD tools to speed their application release time with better quality and development practices.

    When you think about it, in certain mission-critical situations, using a mainframe to have the latest data at any time not only makes sense from a technological point of view, but it’s also attractive in terms of return on investment. The trick is to be able to identify the right use case. Of course, if you have a situation in which 90% of your users are in California and you have a mainframe situated in Bangalore, the laws of physics will apply. Moving the bits from the mainframe to user will take time. But, if you have situations in which user and data are in continental proximity, using a mainframe as a standard data source begins to look very attractive.

    Again, having the right use case is key. But, everything else being equal, using a mainframe for sensitive, data-driven applications warrants significant consideration.

    Putting It All Together

    The constraints on distributed computing imposed by the CAP theorem remain a challenge to those who design and implement large applications at web-scale. The “pick two” limitation CAP imposes can be an annoyance for simpler websites that provide content intended for the general consumer. But, for large scale, data-intensive applications, having accurate data immediately is not optional, it’s essential. Thus, using mainframe technology to create a no-partition data infrastructure becomes a viable way to address the limitations imposed by the CAP theorem. Under the right conditions, making a mainframe the central repository for your application’s data provides a significant return on investment hard to match when compared to commodity-based cloud providers.

    The nice thing about using a mainframe is when it comes to the CAP theorem, you still only get to pick two, but the two you get to pick are consistency and availability. In the scheme of things, those are the two that count.

    — Bob Reselman

  • Application Performance Management for DataOps

    Application Performance Management for DataOps

    Over the years, application performance management (APM) tools have been utilized typically to measure and monitor the performance of software such as web applications. They do so by tracking such things as request response times and error messages, thus providing relevant data to be analyzed by the DevOps team, which then can make the appropriate changes to optimize their software. But what about the effectiveness of the data workflows that contribute to the functionality in many of the applications out there today? How can performance monitoring toolsets be extended to provide these workflows with opportunities for continuous improvement?

    Data Pipelines and Performance Challenges

    Below, I will discuss the use of APM tools in the context of DataOps. I will delve into the metrics and insights an APM tool must provide to engineers so they can support data pipelines efficiently, showing how the effective use of data-centric APM tools can provide extensive value to a DataOps team and their organization.

    Data Pipelines and DataOps Engineers

    A successful DataOps implementation requires creating automated data workflows that analyze large amounts of data quickly, while providing high-quality analysis as an end product. It’s easy to see how the job of a DataOps engineer can be a complicated one. It is their job to build and manage these pipelines to help the business achieve certain goals that rely upon the resulting data analysis. In other words, DataOps engineers are responsible for making sure high-quality data is made available as quickly as possible.

    Consider a common scenario with which we are all familiar: online shopping. With each purchase you make from a particular e-commerce website, the system typically provides recommended products for you to buy. This is often the result of a data workflow that produces data analysis for a recommendation system, which is then leveraged by the application to provide the end user with these recommendations. This pipeline is likely to be made up of several processes.

    These processes include those for loading large sets of raw data into a distributed file system for storing data across a cluster. This would be followed by several processes that need to run on this cluster to format and analyze this data to produce the quality data for the recommendation engine being leveraged by the web application. A lot is being done with large sets of data. And, as is usually the case with so much happening, there is also much that can go wrong.

    The Challenges in Managing Data Pipelines

    There are several major performance challenges that come with the management of the types of pipelines mentioned above. Below are a few of the common threats to data pipeline performance:

    • Scalability: One of the more obvious challenges when dealing with the construction of a data pipeline is scalability—in other words, the effectiveness and efficiency of the pipeline when it needs to deal with a much larger dataset. It’s likely when the jobs were written to deal with such data transformation and analysis that they were tested against smaller sets of data in development environments. What happens when this is no longer the case? Maybe now instead of processing a relatively small amount of data in a test environment, it needs to process much more as it runs in production. Will the processes still perform adequately to deliver quality analysis in a time-efficient manner?
    • Application Failures: Data pipelines rely upon parallel processing. But what happens when you experience the failure of one or more of these parallel-running processes that are executing to clean and analyze the data? This can throw a wrench in your ability to relay quality, complete analysis to the business unit. Being able to resolve failures quickly and get the pipeline back up and running plays a huge role in the success of a particular data workflow.
    • Fragmentation: Most data pipelines consist of a variety of tools that handle different tasks (such as ingestion, storage, processing, analysis, etc.). With so many different tools in the mix, pipelines can become fragmented. Tool incompatibility issues can arise and data formats can vary. It’s important to be able to unify data within the pipeline, even as it moves between disparate tools.

    APM Tools to the Rescue

    Manual vs. Automated

    Enabling DataOps within your organization requires continuous feedback and continuous improvement to your data workflows. A good place to implement these strategies is with the use of APM tools to help manage the performance of these pipelines.

    Consider the processes for improvement that occur without the use of APM tools. It would be on the IT and DataOps folks to locate performance-related problems manually within these pipelines and begin digging through log files to find the cause. A significant amount of time likely would be added to resolving these failures due to the lack of a centralized location to view this information, the lack of automated alerts to notify the team to these performance snafus and the lack of assistance in deriving insights from collected performance data. APM tools can provide an intuitive UI for debugging such issues and can provide the type of insights needed to detect and resolve these problems quickly.

    That being said, monitoring for big data processes is quite different from other types of application monitoring. Certain APM software will be better-equipped for dealing with data pipeline-related challenges than others. Much of this depends on the ability of the APM tool to integrate with the platforms and frameworks used within the data workflow, as well as the metrics the performance management software collects and utilizes to help the DataOps engineers refine their processes.

    What Types of Metrics Can Help Determine the Causes of Issues Within a Data Pipeline?

    Earlier, I discussed data pipeline performance challenges stemming from scalability issues and process failures. Let’s discuss what types of metrics should be collected to resolve issues like these in a data workflow.

    Before delving into the metrics, it’s important to note that having full-stack data is essential. By full-stack data, I mean not just infrastructure metrics (which are the most obvious type of metric folks tend to think about when managing data pipelines), but also other data points such as user data, application data and resource data. All this data needs to be unified and correlated across all systems in the pipeline to manage it effectively.

    Consider scalability, for instance. The APM solution you are using must be able to track several key metrics to help identify scaling-related bottlenecks within the data pipeline. These include:

    • The Amount of Data Being Processed: How can we identify a scalability issue without knowing how much data is being processed by a particular run? The answer: We can’t. Being able to quantify the amount of data being handled by each iteration of the data pipeline is the first step in detecting if the workload is the problem.
    • The Length of Each Run for Each Individual Process: The APM software being used by an organization must be able to integrate with the structure of the data pipeline to report the length of each run for each individual job running within the workflow. If a particular job was completed in one minute when processing X amount of data but took 20 minutes when processing 10X, then we have diagnosed a potential scalability issue and narrowed it down to the root cause in a time-efficient manner. Maybe the fix is as simple as tuning some SQL, and the scalability issue can be rectified in relatively little time.
    • Resource Utilization: Certain processes may require more resources than others to complete in a timely fashion. The ability of the APM tool to track the amount of resources being utilized by each process is key in determining if simply reallocating certain resources can fix problems with performance.

    Example issues:

     

     

    Unravel is an example of an APM tool that can track all the above while organizing the data within a user interface that provides DataOps engineers with just about all they need to diagnose data workflow performance issues from one screen. In addition, Unravel employs AI solutions to help take the metrics they collect and provide the engineers with specific insights as to how they can optimize their pipeline. This helps decrease the amount of time DataOps personnel spend on managing existing processes, allowing them to spend more time creating new processes to improve the quality of analysis being provided to the business.

    Conclusion

    APM tools are no longer solely for ensuring application availability. APM tools for big data, such as that from Unravel, have allowed for performance monitoring and analysis within data workflows, helping to ensure the availability of crucial, high-quality data analysis. Any organization looking to improve the quality of their data analysis processes should be looking to extend their APM toolset to include those for big data.

    This sponsored article was written on behalf of Unravel.

    — Scott Fitzpatrick

  • Adobe Strategy Revolves Around DevOps

    Adobe Strategy Revolves Around DevOps

    At the Adobe Summit 2019 conference this week, Adobe detailed that its Adobe Experience Cloud customer experience ecosystem is reliant on a set of best DevOps processes that is wrapped around a modern microservices architecture and layered on top of big data platform deployed on the Azure cloud.

    Adobe last year acquired Magento, provider of an e-commerce platform, and Marketo, provider of a marketing automation platform. Since then, Adobe has been working to integrate those offerings with its existing marketing application to create the Adobe Experience Cloud.

    Anjul Bhambhri, vice president of platform engineering for Adobe, said this set of software-as-a-service (SaaS) applications are being integrated around a shared common data model that is layered on a microservices-based architecture of containers running on Kubernetes clusters. On top of that, Adobe is layering a series of machine learning algorithms, dubbed Sensei, to embed artificial intelligence (AI) across the entire portfolio, he said.

    As that work progresses, Adobe plans to allow customers and partners to leverage the data stored in the Adobe Experience Cloud to train their own AI models using data curated by Adobe, Bhambhri noted. That data model, known as the Experience Data Model (XDM), is also being shared with partners including Microsoft and SAP, which last year committed to an ambitious initiative to build a common data model in collaboration with Adobe. XDM makes use of open source Kafka messaging software to integrate the various data pipelines that are required to integrate these applications at scale in near real-time, Bhambhri said.

    Those efforts complement an existing initiative under which Adobe is exposing open application programming interfaces (APIs) to encourage developers to build applications that extend Adobe Experience Cloud.

    Rather than building their own customer data platforms, organizations can drive their digital business initiatives using a big data platform managed by Adobe on their behalf. Many organizations have been building their own platforms to achieve that goal with mixed success, first deploying a data warehouse on top of a big data platform and then working to integrate various applications to provide a richer digital customer experience. Bhambri noted Adobe is pouring considerable resources to obviate the need for organizations to acquire, deploy and manage their own IT infrastructure to achieve that goal.

    IT leaders must decide whether they want to reduce the time, money and effort required to build these custom platforms in favor of using a managed service from Adobe—which, arguably, would enable them to pour more resources into developing applications that, as a result, would come to market faster.

    In the meantime, it’s clear the need to embrace digital business transformation will drive more organizations to embrace DevOps processes. The question now is to what degree they need to define their own set of best DevOps processes versus simply lifting the DevOps best practices that organizations such as Adobe have already proven.

    — Mike Vizard