devopsy.kinsta.cloud

Category: DevOps Toolbox

  • Electric Cloud Aims to Make DevOps More Accessible

    Electric Cloud Aims to Make DevOps More Accessible

    Electric Cloud today unfurled the Winter 2019 edition of the ElectricFlow platform for managing DevOps, which makes it easier for different classes of users to tailor the platform to meet their specific needs.

    Sam Fell, vice president of marketing for Electric Cloud, said the latest release of ElectricFlow allows end users to choose from different personas based on their role in the organization, and each persona has a user interface crafted to surface the level of detail they most likely require. That approach is intended to make the company’s DevOps platform more accessible to wider variety of end users, he said.

    The latest edition of ElectricFlow also adds a customizable customer login experience that is more task-oriented. Users can now add bookmarks/favorites to access specific pages faster.

    Finally, Electric Cloud is now including release and pipeline templates that make onboarding easier, based on user created events/triggers and code check-ins in a Git repository.

    Fell said that while DevOps platform such as ElectricFlow increasingly have incorporated higher levels of automation by relying more on machine learning algorithms, humans should not be left behind. Electric Cloud has been focusing more of its development effort on making it easier for DevOps professionals of all skill levels to derive value from the platform, he said, noting current DevOps platforms tend to make available far too much information than the average IT professional can ingest, added Fell.

    Less clear is how the complexity inherent in DevOps processes extends the time it takes for an organization to employ DevOps pervasively. While DevOps platforms are undoubtedly powerful, they also can be intimidating. In many instances, it’s simply easier for IT administrators to continue to rely on tools they already know to manage a specific task even when that tool is not as robust or rich as what might be accessible in a more modern DevOps platform. What often is ascribed to cultural IT inertia may have its roots in the simple fact that IT professionals are not always equally excited about having to learn and master new tools.

    The number of IT organizations that have been able to employ DevOps processes across their entire organization are still a comparative minority. Most organizations today have what can best be described as pockets of DevOps processes, usually driven by a small, cross-function team that is highly motivated to update a specific set of applications continuously. IT organizations are, of course, hoping to replicate the success of those teams across the rest of the organization. However, the rest of the IT organization doesn’t always have the same level of engineering skills—the spirit may be willing, but the initial DevOps hurdles needed to be overcome are simply too high.

    It may be a while before IT organizations sort this all out. More accessible tools are only one part of the equation. But IT leaders almost inevitably find that giving their teams access to easier tools will, at the very least, start moving their organization in the right direction.

    — Mike Vizard

  • Actifio Looks to Meld DataOps with DevOps

    Actifio Looks to Meld DataOps with DevOps

    Actifio and IBM have formed an alliance under which IBM will make available data management software developed by Actifio under its own name as part of an effort to close the gap between DevOps and DataOps.

    At the same time, Actifio will make available a software-as-a-service (SaaS) offering for the first time, dubbed Actifio Go, which promises to make it easier to manage and move data across multiple clouds.

    Actifio CEO Ash Ashutosh said with the rise of both DevOps and DataOps there is now a unique opportunity to unify the management of pipelines across the enterprise. While storage administrators have been creating data pipelines for databases and applications for decades, the pipelines developers create to consume that data have been developed and managed separately. Now there is a nascent DataOps effort based on a set of best practices for automating the management of data that borrows many of the same principles being advanced by DevOps proponents for the building and deploying of applications.

    Actifio’s core Virtual Data Pipeline technology advances DataOps, Ashutosh said, by making it easier to copy a virtual instance of a data set running in a production environment and update it with the most recent copy of that data at any point in time. That capability is not only being employed to, for example, copy data into the cloud, but also to make data readily accessible to DevOps teams as needed on a self-service basis, he said, adding the goal is to make it easier for developers to dynamically consume the latest version of any data set as needed, with a minimal amount of intervention required on the part of a storage administrator.

    IBM plans to make an instance of Virtual Data Pipeline available as part of the IBM InfoSphere portfolio of data management software it provides. That agreement extends Actfio’s existing reseller relationship with IBM.

    Ashutosh conceded there is much work to be done before DataOps becomes a natural extension of DevOps. Most storage administrators today are not all that familiar with how the data pipelines they create are consumed by applications. Most storage administrators continue to think in terms of volumes and files rather than in terms of managing data pipelines, he said.

    But as those application environments become more dynamic thanks to the rise of microservices, pressure to modernize data management will only increase, Ashutosh noted. For example, more DevOps teams will be integrating the ability to copy data directly into their continuous integration/continuous delivery (CI/CD) platforms. Achieving that goal will be easier if the software employed to manage data within the context of a DevOps processes is invoked as a SaaS application, he said.

    Most IT organizations are a very long way from melding DevOps and DataOps. Most are still struggling with mastering the fundamentals of DevOps. But the convergence of DevOps and DataOps is all but inevitable. The only thing that remains to be seen is the degree to which IT organizations will proactively seek to achieve that goal versus waiting for developers to force the issue.

    — Mike Vizard

  • ‘DevOps Engineer’ Title Lands in Top 5 of Dice Salary Survey

    ‘DevOps Engineer’ Title Lands in Top 5 of Dice Salary Survey

    The position of DevOps Engineer is gaining some serious respect, and with that comes cash.

    In Dice’s recently released “2019 Tech Salary Report,” DevOps Engineer now ranks in the Top 5 of all tech salaries, with an average of $111,683. The survey is based on the responses of 10,780 technology professionals conducted between Oct. 22 and Dec. 13, 2018.

    It’s the first time that the title of DevOps Engineer has appeared in the Top 5, according to Dice. The one caveat is that the position might not be as popular as the annual salary suggests; rather, more professionals now identify with this position.

    What it does mean is that the DevOps Engineer title is now ranked with more established tech titles, including senior leadership positions CEO, CIO, Vice President and Director, with an average salary of $142,063; System Architect, at $129,952; Tech Management, including Strategist and Architect, at $127,121; and Product Manager, at $114,174.

    Overall, the average salary for tech workers remained relatively flat year over year. In 2018, the average salary totaled $93,244, a less than 1 percent increase from 2017, when the average salary stood at $92,712, according to Dice.

    In an email, Dice CTO Klavs Miller noted that the increasing use and adoption of various cloud-based technologies is pushing enterprises to invest more in DevOps engineers to take advantage of that compute power to develop new apps.

    “Many DevOps Engineer job requirements include a heavy focus on the creation of infrastructure as code, essentially transforming operations into the management of codebases that create infrastructure (in cloud services or native cloud infrastructure rather than servers and hardware),” Miller wrote. “We are also seeing and experiencing first-hand that more and more organizations are moving toward having combined development and operations teams around a domain, rather than splitting along traditional functional lines.”

    In his email, Miller noted that while there might be few tech pros that use the “DevOps Engineer” title, it doesn’t mean that there’s an actual lack of engineers with the skills to help enterprises with their DevOps projects.

    Instead, it’s a matter of looking for talent that might already be there.

    “There is probably a shortage of talent that label themselves as ‘DevOps Engineers,’ but employers should look at the skills required by those roles instead,” Miller noted.

    “There’s a growing demand for a skill that is still evolving with market trends,” he added. “As an example, DevOps is not nearly as common as its more common sibling, software engineers, but it has become more popular each year according to our salary survey and among employers who are sourcing Dice for these positions.”

    The types of enterprises looking for DevOps talent are varied. Right now, insurance company Geico and hardware store giant Lowe’s are each looking for specific DevOps titles, according to Dice.

    If private business isn’t your area, the Louisiana Economic Development agency is currently advertising for an entry-level DevOps position.

    In addition to specific tech titles, the Dice report shows that several types of programming languages that are now associated with the DevOps discipline are commanding significant salaries in the market.

    For instance, those with expertise in Google’s Golang programming language can command salaries of up to $132,827, according to the Dice survey.

    A little further down the line, and keeping with Miller’s emphasis on cloud-based technologies driving much of the DevOps conversation, those with knowledge of Amazon’s DynamoDB and Redshift can command salaries of more than $125,000 on average, the study found.

    Additionally, platform-as-a-service (PaaS) skills can net a developer an annual salary of more than $122,000.

    In his email, Miller noted that it’s difficult to draw a direct line between needs for certain programming languages and the growing popularity of DevOps. The growing reliance on cloud and the types of digital transformation projects enterprises are engaged with now show how the market has shifted.

    “There are some assumptions that we can’t confirm, but DevOps shares programming languages with other roles that are also in demand, as employers adopt cloud-based technology and open source becomes more pervasive in companies’ tech stacks,” Miller added.

    — Scott Ferguson

  • OverOps Employs Machine Learning to Measure Application Reliability

    OverOps Employs Machine Learning to Measure Application Reliability

    OverOps is adding a series of dashboards to its analytics platform that employ machine learning algorithms to identify which software application anomalies in a pre-production environment are likely to have the most impact on a production environment.

    Company CTO Tal Weiss said one of the biggest challenges DevOps teams face is prioritizing which issues to fix first in an application. The Reliability Scoring capability being added to the company’s namesake platform for analyzing code employ machine learning algorithms to keep track of introduced errors, regressions and slowdowns during every phase of pre-production. In addition to enabling DevOps teams to visually identify the root cause of an issue via a single click, that data is then used to identify which issues will impact reliability.

    Weiss noted that the Reliability Scoring can also be employed to set thresholds of acceptable application performance that would also be used to determine whether there is an issue. The data collected using that capability then can be fed back into a continuous integration/continuous deployment (CI/CD) platform to set up gates that would then prevent suspect code from being promoted, he said.

    The primary benefit, however, should be a major reduction in the number of troubleshooting instances any DevOps team should need to engage in after an application is deployed in production, said Weiss. In fact, the Reliability Dashboards will provide a central mechanism through which developers, testing professionals and IT operations teams can collaboratively track and measure the progress of an application development project, he added.

    The Reliability Scoring tool OverOps developed employs “micro-agents” that operate between application code and the hardware. Weiss said those micro-agents can not only capture data that was never previously available, they deduplicate, classify and gate critical issues as they move into staging and production. The Reliability Dashboards visualize that data in Grafana, a set of open source tools for visualizing data. DevOps teams can also opt to feed that data into any existing dashboards they may already have in place.

    In general, Weiss said the metrics surfaced by the Reliability Dashboard are critical to justifying investments in DevOps over the long term. Organizations expend a lot of time and effort making the transition to DevOps. But most of them don’t really know to what degree they are getting better at building and deploying applications. Reliability Scoring dashboards provide the historical context required to make those assessments, he said.

    Far too many DevOps teams are still starved for data needed to make critical decisions. In the absence of that data, teams address issues as they arise, rather than based on the impact they might have on the business. Worse yet, it’s difficult to identify new issues over those that might be another manifestation of a known issue that suddenly becomes a much higher priority.

    Regardless of the root cause of any problem, however, DevOps teams would do well to remember things that get measured are usually the first things to get fixed.

    — Mike Vizard

  • Salt Security Unveils Platform to Secure APIs

    Salt Security Unveils Platform to Secure APIs

    Salt Security, fresh off raising an additional $10 million in funding, announced a platform that promises to advance the state of DevSecOps by making it possible to detect when application programming interfaces (APIs) are being probed by cybercriminals looking for weaknesses to exploit.

    Salt Security CEO Roey Eliyahu said the Salt Security API Protection Platform employs machine learning algorithms to first establish a baseline for how APIs are being invoked across an IT environment. The software-as-a-service (SaaS) platform then monitors activity to surface any anomalous behavior indicative of cybercriminals engaging in reconnaissance of API traffic flows, he said.

    Armed with that analytics, DevOps teams can then prioritize fixes to any potential security hole cybercriminals might be able to exploit. That approach advances DevSecOps because the Salt Security technology makes it possible to feed requests to address those security issues directly into a continuous integration/continuous deployment (CI/CD) platform—critical, he said, because all too often developers are unclear regarding what cybersecurity professionals are asking them to fix whenever a cybersecurity issue arises.

    Eliyahu noted that, in today’s business environment, security has become a major concern. According to the Open Web Application Security Project (OWASP), nine of the top 10 vulnerabilities in its current Top 10 report include an API component. Insecure APIs were ranked as the third most severe threat to cloud computing in 2018 by the Cloud Security Alliance (CSA), and Gartner predicts that by 2022 API abuses will be the most frequent attack vector resulting in data breaches for enterprise web applications.

    The challenge organizations face is each API is unique to each application and the organization that builds them, which means each has unique logic and, by extension unique vulnerabilities, said Eliyahu. Existing cybersecurity tools lack granular understanding of these because they are designed to protect organizations from known attacks. Attacks that target API logic go unnoticed by these tools because they appear to be authenticated use cases, and then employ subtle methods to probe each for unique vulnerabilities, he noted.

    To thwart those attacks, the Salt Security technology automatically discovers all APIs and associated functionality, said Eliyahu. That’s important because it’s not uncommon for organizations to lose track of the many that might have been exposed over the course of an application’s lifetime.

    APIs in general are often overlooked because much of the focus in any organization tends to be on business logic. But as the number of applications and services continue to proliferate across the enterprise, the number of APIs being exposed has increased dramatically. Those numbers will increase more as organizations embrace microservices to build applications that are more easily updated.

    The issue, of course, is how to secure APIs that often are developed and maintained by teams that weren’t involved in the integration of the original application. Because of that issue, API management has emerged as a discipline unto itself within many organizations. The challenge now is finding a way to ensure those APIs are employed as intended, versus becoming an avenue for data exfiltration that no one is aware of until it’s long too late.

    — Mike Vizard

  • DevOps Challenges and Version Control: What the Survey Results Tell Us

    DevOps Challenges and Version Control: What the Survey Results Tell Us

    Putting DevOps into practice is an ongoing experiment for many organizations. Developers often struggle with continuous integration (CI) or continuous delivery (CD) performance, testing delays and other bottlenecks. While not the answer to everything, version control—that long-established foundation of software development toolchains—has become a central part of DevOps, particularly DevOps at scale.

    This is underlined by the results of “DevOps Challenges and Version Control,” a recent survey conducted by devopsy.kinsta.cloud on behalf of Perforce Software, which provides some interesting insight into their experiences. This includes the role of version control systems within DevOps in practice, plus the top challenges the companies surveyed face.

    A total of 467 people from organizations across the globe responded, with roles ranging from DevOps engineer to QA analyst and CTO. Before we look at the challenges and issues they face—not just with their version control system but DevOps in general—first let’s look at the good news: Most version control systems are a good fit for their current DevOps initiatives.

    For instance, more than half of all respondents said the most important features of their current version control system are:

    • Improved collaboration.
    • Improved CI.
    • Better support for distributed teams.
    • Quality.
    • Faster commits.

    These were followed by compliance/security, at 48 percent. These jibe well with the main DevOps challenges they also cited:

    • CI/CD issues (a third cite CI/CD workflow streamlining as their top priority).
    • Prioritizing backlog and development projects.
    • Managing build artifacts and containers
    • Developer complaints about slow commits and merges
    • Working with open source software.

    These are all things that VCS can help fix.  For instance, these days, most version control systems transactions are automation for CI, rather than user requests.

    CI Challenges

    So far so good, but users want more from their version control system. Almost a third would like better integration with the software development toolchain. Other improvements on the wish list include security with fine-grained permissions, handling large files or repos and file access for teams in multiple locations or working remotely and improved CI performance.

    CI is critical to good DevOps and giving developers real-time feedback theoretically improves quality, but if the version control system’s speed and performance does not match demand, then bottlenecks are created. The last thing anyone wants is developers sitting around waiting for builds to happen. The survey’s respondents said testing (28 percent) and waiting for code reviews (23 percent) were the two points in CI that slow everything down the most.

    People Want More from their Version Control Systems

    So, what’s the answer? Version control that combines speed and performance, of course, but also built-in automated code review to alleviate the frustration of time spent waiting for tests and other tasks. Given that 64 percent are based in multinational locations, 7 percent work remotely and a mere 29 percent work in the same office, clearly the version control system needs to support dispersed working environments, too. In many situations where the version control system is being used for multiple artifact types and not just code, this needs to be covered, too.

    Likewise, most respondents have Git somewhere in the organization and understandably so: it is open source, users can get started with Git quickly and it is suited to small teams. However, Git struggles to scale, and, given some of the challenges that respondents quoted, this is clearly something many are experiencing. Enterprise strategies are emerging that give developers the Git interface they’re familiar with, but also the scalability, global replication and high performance for moving files and performing builds.

    With so many perceived challenges cited, it is no surprise that almost half of all those surveyed said they plan to replace their existing version control system. The top reasons cited were scalability issues, poor cross-platform integration and lack of DevOps support, plus the perennial pain point of license costs.

    Understandably, switching version control systems can seem daunting, but equally, based on the survey’s responses, many organizations are clearly living with systems that are expensive to maintain and have productivity issues that are getting in the way of, rather than supporting, DevOps. Conversely, choosing version control system strategies and tools can tangibly and positively impact on achieving DevOps success.

    — Tim Russell

  • AIOps: How to Solve Performance Issues in Complex Data Pipelines

    AIOps: How to Solve Performance Issues in Complex Data Pipelines

    AIOps takes center stage to topple barriers as modern data application growth accelerates

    There seems to be a chasm between DevOps and big data teams in the enterprise, and the bridge over that chasm is long and treacherous.

    In fact, data analysts who are charged with managing and making sense of the growing volumes of data in most organizations traditionally have been excluded from the strategy-making and operations functions of DevOps personnel. Yet, data analysts are critical to the success of software development and at every stage of the software delivery pipeline.

    DevOps Under Siege as Modern Data Applications Proliferate

    Let’s take a deeper look. We know that DevOps personnel are on the front lines in ensuring that all systems, both new and existing, perform as expected in a production environment. But, even today, DevOps teams struggle in dealing with technical challenges when faced with complex data pipelines and the underlying big data stack that supports them, including:

    • Root cause analysis — determining if an application issue is due to application code, config errors or resource contention.
    • Correcting inefficiency in code.
    • Improperly configured auto-scale clusters.
    • Lack of actionable insight into how clusters were tuned at both a cluster and job level.
    • Difficult access to support engineers and promotion of DevOps collaboration.

    And these are just the tip of the iceberg.

    The pressure is on. On average, companies with data-driven environments garner up to 50 percent higher market value than their competitors, according to IDC, and big data is a contributing factor. Thus, big data implementations have become a top priority for businesses of all sizes, and on multiple fronts: revenue acceleration, creation of new products and contributing to better, quicker decision-making. But the unfortunate truth is that, as big data projects move from pilot to production, they invariably encounter serious performance problems that frustrate both data analysts and IT operations.

    What’s more, DevOps and big data personnel face a mutual problem: lack of centralized, top-down visibility into their end to end data pipelines and applications. And, as a result, they face challenges in pinpointing the source of problems—namely, that problems can become visible anywhere in the stack, but their root cause could be elsewhere in in the stack. And, these problems result from root causes as diverse as bad code, inefficient data partitioning, mismatched system configuration settings, best-guess resource allocation and infrastructure issues. Furthermore, a big data stack running multiple applications faces resource contention and prioritization issues that can reduce both individual application performance and overall system performance.  And streaming applications create their own challenges;: for example, if a streaming application starts to lag in processing data in real-time, diagnosing the root cause takes considerable time and effort.

    The industry is faced with a reality where the problems are outrunning the solutions. In the case at hand, organizations have relied on brute force and expensive custom solutions for making analytics-based recommendations more user-friendly to improve individual application and full-stack performance.

    AIOps Takes Aim

    AIOps, or artificial intelligence for IT operations, is gaining steam in the enterprise, with many new use cases appearing as data volumes grow and systems become more complicated. Gartner expects AI to pervade every aspect of managing IT, including application performance management (APM), IT infrastructure monitoring, cloud platform monitoring, performance management, capacity management, the delivery tool chain (CI/CD) and IT service management.

    In fact, AIOps is on a tear: Just three years ago, it was only a concept. Today, Gartner predicts that by 2019, 25 percent of global enterprises will have implemented an AIOps platform to support two or more major IT operations.

    But even as it continues to evolve, AIOps today can deliver automated recommendations to solve expensive, challenging and time-consuming problems with big data deployments and across entire complex stacks. Without AIOps, there would be simply too many variables to correlate manually.

    AIOps makes APM for modern data application possible. It does so by helping to eliminate painstaking, trial-and-error configuration, troubleshooting and remediation workflows that have long plagued big data operations teams. And, for the first time ever, dataOps and DevOps teams can troubleshoot, support SLAs, remediate issues and manage large-scale deployments in an agile and dependable manner.

    It’s a new environment for the modern data center. As part of an APM solution, AIOps is hard at work helping automate and guarantee the performance of big data applications and addressing the primary challenges that DevOps faces with the unrestrained growth of big data.

    It could still be a bumpy ride—organizations of all kinds face unprecedented increases in data volumes. But now, working together, DevOps and AIOps are better-equipped to contend with the challenges.

    — Bala Venkatrao

  • How AIOps Makes DevOps Less Noisy

    How AIOps Makes DevOps Less Noisy

    For DevOps engineers, “noise” is the enemy of productivity. In this context, the noise we’re talking about is unnecessary or low-priority alerts and notifications that distract engineers from identifying serious issues—and ultimately can cause alert fatigue syndrome, in which alerting systems are ignored altogether.

    Without the application of a well-constructed noise reduction plan, alert noise will end up taking away from the focus of solving the actual problem that may be causing a more critical issue.

    So, how can you fight alert noise? You might be tempted to assume that the answer lies in reducing the number of alerting tools you use, but that’s probably unrealistic. Teams today are dealing with complex application stacks and multiple tools and data sources, all of which can produce their own alerts for issues that may be interdependent or isolated. You simply can’t take tools out of your stack.

    What you can do, however, is analyze and manage alerts more effectively. Dealing with alert noise ultimately boils down to developing an effective problem analysis strategy. Let’s explore how, with a specific focus on how AIOps factors into keeping alerts manageable.

    Reduce the Time in Analyzing Alerts

    While reducing noise is important, DevOps teams risk losing valuable time solving critical problems if they also have to create and/or add systems to properly analyze alerts and their sources and priority level. Maintaining this system also would be a source of unnecessary labor that may reduce the alert noise, but also reduces the amount of time that DevOps staff can work on the solution to the root cause— which brings up another consideration. When coming up with a plan, the amount of complexity involved in adding the monitoring tools necessary to cover an entire application and/or service can be equally laborious compared to not having a solution at all. At this point, we may be merely shifting the labor emphasis.

    This is Where AIOps Comes In

    Now, let us envision a way that a team can gain full insight into a root cause without exposure to alert noise. The team doesn’t need to analyze every single alert, which would include:

    • Duplicate alerts.
    • Alerts from multiple sources (tools, microservices).
    • Informational alerts (warnings).

    Instead, the team can let AIOps analyze and classify the stream of alerts and identify critical issues. DevOps no longer would need to spend valuable collaboration time analyzing every link in the toolchain. When a team has to analyze every tool, the team could produce false positives, redirect resources toward the wrong solution, spend too much time trying to identify the root cause and create excessive alert noise. Let us envision a way in which the team does not need to spend excessive time in analysis and can experience more effective collaboration in service of solving the actual problem.

    AI and machine learning can relieve this manual process through AIOps. Automated collection of data across the application stack, alert correlation, identifying root cause and decision-making can be leveraged by a team to get to the problem immediately without all of the messy work of manual analysis, sorting and identifying root cause. This level of automation allows for needed human interaction through collaboration across teams, which can result in faster resolution, decreased downtime and effective communication of key events to company staff and end users.

    Analyzing to Predict Future Problems

    Another benefit of AIOps and the power of automated data collection, analysis, issue identification and decision-making is the ability to predict future events. A DevOps team can leverage these insights into effective preventative measures. The ability of AIOps to produce these insights will help a team find ways to automate and/or improve operations. Reducing alert noise no longer needs to be a goal because it is happening through a smooth interaction between the human team and the AI.

    There is a growing need for DevOps teams to utilize the power of AIOps. While human teams are not replaceable, redundant and time-consuming tasks are. The ability of AIOps to effectively traverse a complex toolchain with isolated issues to analyze alert importance, correlate alerts, collect and analyze relevant data, identify root cause and produce holistic actionable insights eliminates inefficiencies in problem-solving and reduces alert noise.

    Making DevOps less noisy is a goal with its roots in effective use of information. Alerts serve to inform, but thanks to the increasing complexity of applications, alerts become noise. Leveraging the power of AIOps serves to do a lot of the dirty work for a team so that noise can be reduced—but more importantly, it can make operations run smooth and keep customers happy.

    — Phillip Lorenzo

  • How AIOps Helps You Get More Out of DevOps

    How AIOps Helps You Get More Out of DevOps

    DevOps may seem like an overused term. It involves taking an application’s source code and running it in an environment. It can cover the processes, the technology or even the people that maintain the technology that is running those very processes.

    At its heart, though, DevOps is about helping developers to be self-sufficient when it comes to the basic operations around getting their application to a real environment. Once things are running in a test or production environment, there are other IT functions that are required to keep it running and performing well. While there are teams that specialize in these functions, such as site reliability engineers (SREs) and system administration, they are usually overwhelmed by the sheer amount of data their areas of responsibility generate without considering the data that applications going through their DevOps processes are also creating.

    This is what makes AIOps the perfect complement to the DevOps space.

    What AIOps Does

    What AIOps does best is break down silos within IT. Having data from all the functional areas processed and handled by modern machine learning (ML) in a single place allows correlation, which leads to faster problem identification and resolution.

    Before machine learning was introduced, for humans to process data, more or less they operated in three ways:

    • They knew exactly what data to look for, such as when a specific business function isn’t working. There is a typical message in the monolithic application that handles all the core business.
    • They routinely sifted through data only from their area of expertise.
    • When they went outside their area, any logs and other data they had access to often was summarized data, which has time gaps and can miss a lot of detail that proper correlation needs.

    Now with ML, the AIOps data platform can perform trend analysis, look for patterns and exceptions across all the data that is being generated in the environment and highlight what it considers important through web interfaces or using an event management solution for after-hours support.

    Over time, as the AIOps platform continues to learn the environment, it can be given the ability to automatically resolve certain error conditions, as it will be able to identify them faster than human operators and often resolve them before they become service-impacting.

    How This Helps DevOps

    Machine learning aids DevOps processes in three ways when working with logs and data:

    • Knowing exactly what data to look for: In a true polyglot DevOps environment that is leveraging microservices, it is next to impossible for any one person to know the most common error messages that might occur. In this model where a new business function can be conceived, written in one of many languages, and deployed into production without Operations even being notified, ML allows the AIOps platform to quickly identify what “normal” looks like for a new microservice and start looking for anomalies.
    • Only routinely sifting through data from their area of expertise: When working in DevOps, developers need to be able to quickly find data from multiple areas in IT that is related to whatever they are trying to diagnose. Using AIOps, they will be able to quickly pull data from systems directly related to their incident. Whether it is as simple as using timecodes or a correlation GUID (globally unique identifier) that is put in the logs purposefully to aide tracking, AIOps will guide the developer through the other systems’ data.On the flip side, if there is an infrastructure problem, the same AIOps platform can help correlate what downstream services may have been impacted, from failed builds to Kubernetes pods that were restarted.
    • Summarized and incomplete data: The true value of machine learning is that it has the ability to process a higher volume and variety of data at a much higher speed and accuracy than a human, and sort and index it for later use. Not having to summarize the data to one data point every five minutes could result in a problem being discovered that has occurred often but not consistently, as it may have been dropped in the summarization.

    Conclusion

    AIOps is very young in the technology industry, but has great promise as it builds on previous generations of IT operational analytics products, combined with the latest in machine learning capabilities. DevOps is a key field in IT that has more logs than it knows what to do with. Combining the technologies and expertise in these two growth areas will have a huge positive impact on productivity across enterprises with improved service management.

    — Vince Power

  • Red Hat Adds Support for Nested Workflows to Ansible Automation Framework

    Red Hat Adds Support for Nested Workflows to Ansible Automation Framework

    Red Hat today announced it is adding support for nested playbooks created using the open source Ansible automation framework within one another, which, in turn, will make it easier to automate complex IT processes.

    Bill Nottingham, product manager for Ansible, said the goal is to enable DevOps teams to reuse existing Ansible playbooks to kick off a series of nested IT jobs and workflows using version 3.4 of Ansible Tower, the console Red Hat provides for creating and managing Ansible playbooks.

    Version 3.4 of Ansible Tower also now makes it possible to slice an IT job into a series of smaller jobs that can be distributed across a cluster to accelerate execution. That capability makes it possible to scale out the processing of an Ansible playbook, he said.

    Finally, Red Hat also announced that Ansible Tower. 3.4 is now able to run on Red Hat Enterprise Linux in Federal Information Processing Standard (FIPS) compliant mode.

    Nottingham said Red Hat expects the ability to reuse Ansible playbooks within a set of nested processes will encourage IT organizations to create more playbooks now that it’s easier to reuse them as a set of modular components. IT organizations can even make launching jobs dependent on the completion of other jobs using a scheduling mechanism within Ansible Tower, he added.

    Although adoption of IT automation frameworks has been uneven, the last few years has seen IT organizations rely more on them to manage tasks as IT environments become more complex. That level of complexity will only increase as IT organizations deploy more applications that are based on microservices created using containers that need to be updated frequently. IT automation frameworks also go a long way toward making up for what has become a chronic shortage of skilled IT administrators. That capability is especially critical as both the number of applications and the rate at which they are being updated continue to accelerate.

    Most IT administrators have limited programming skills, which is why a declarative approach to IT automation in the form of an Ansible open source project has been gaining significant traction. In fact, thanks to automation, traditional IT metrics such as the number of servers being managed per administrator increasingly are becoming obsolete.

    It may take a little while longer before IT automation frameworks are the rule than the exception. But as the IT community now approaches the end of the second decade of the 21st century, reliance on manual processes that were developed at the end of the last century is no longer viable in this era of distributed workloads. In fact, regardless of where applications are deployed, the expectation in the DevOps era is that existing IT operations teams will be able to manage them without having to hire additional IT administrators.

    — Mike Vizard