devopsy.kinsta.cloud

Tag: full-stack monitoring

  • Overcoming IT Complexity With Full-Stack Observability

    Overcoming IT Complexity With Full-Stack Observability

    Organizations embarking on digital transformation journeys tend to develop complex infrastructures. They update their legacy applications while adding multi-cloud, virtual and cloud-native capabilities. Eventually, IT pros find themselves managing diverse, complex and distributed networks including cloud, system, application and database infrastructures.

    To rein in the resulting complexity, organizations tend to amass monitoring and managing tools. The goal is to simplify systems oversight—but instead, silos tend to develop because teams use a wide variety of tools to manage their networks or infrastructures. 

    This piecemeal approach worsens operational blind spots and delays problem resolution. It also increases security exposures. Soon, overwhelmed IT pros can’t keep pace with app modernization or infrastructure dynamics because they’re awash in complexity.

    Though this scenario is common, it’s not inevitable. IT teams can ease their digital transformation journey by implementing an integrated and cost-effective full-stack, end-to-end monitoring service capable of overcoming complexity and silos. 

    Organizations require full-stack observability.

    How Observability Differs From Traditional Monitoring

    Observability goes beyond traditional monitoring. Traditional monitoring helps IT organizations understand the actual state of their infrastructure and applications. It captures and processes volumes of infrastructure and application telemetry data and notifications and displays which components are up, which ones are down and which ones have changed. 

    Usually, traditional monitoring focuses on a specific network, cloud or infrastructure. It tracks application and infrastructure elements so IT pros can identify anomalies and investigate problems as they arise.

    Monitoring relies on metrics-oriented dashboards built to assess telemetry data against manual or basic statistically relevant thresholds. Monitoring tools are invaluable, but they don’t offer cross-domain correlation, service delivery insight, operational dependencies, or predictability. Modern systems have complex multi-cloud environments and a deluge of telemetry data: This counts as a shortfall.

    Observability does more. It measures the internal states of systems by examining the outputs and looks at applications and systems in their entirety—from the end-user experience to server-side metrics and logs.

    But observability also includes monitoring as a critical element. To gain observability, you must first collect information through monitoring. Observability uses the insights and metrics generated through monitoring to understand what’s causing the issue at hand. 

    Monitoring aggregates and displays the data to determine whether the systems are operating as expected. The analysis of this information is compared with expected outcomes and objectives. This allows IT pros to understand the state of their infrastructure and applications. 

    Silos are thus avoided because complex environments are viewed in their entirety.

    Observability in Action

    Once in place, observability allows IT organizations to continuously improve performance, availability and digital experience across complex, diverse and distributed hybrid and cloud environments.

    With observability, organizations can quickly find and resolve anomalies. However, full-stack observability goes beyond monitoring and expedited problem resolution: It supplies insights, automated analytics and actionable intelligence through cross-domain data correlation, machine learning (ML) and artificial intelligence for IT operations (AIOps). It works across massive real-time and historical metrics, logs and trace data.

    Observability goes beyond the silos and the piecemeal approach associated with traditional monitoring. And when observability isn’t limited—when it includes ML and AIOps—it uses the large volume of gathered data and supplies insights, automated analytics and actionable intelligence to help IT staff expedite problem resolution. It also enables ITOps, DevOps and security organizations to achieve consistent, optimized and predictable business service delivery with continuously improved digital experience and IT productivity. 

    The result is customers and employees benefit from better-run systems. The technology provides organizations of all sizes and industries with comprehensive, integrated, and cost-effective functionality through cloud-connected on-premises or software as a service (SaaS) deployment flexibility.

    When embarking on digital transformation journeys, organizations don’t need further complexity—certainly not while they update legacy applications and add a plethora of modern services and capabilities to their stack. The key to reducing complexity is observability.

    Observability simplifies the transformation process. It reduces operational noise to benefit ITOps, DevOps and security teams. They can become more proactive in issue and anomaly detection to achieve optimum IT performance, compliance, and resilience. With full-stack observability, any organization—no matter its size or industry—can reduce IT complexity while readying itself for digital transformation.

  • Could Full-Stack Observability Help Lessen IT’s Burden?

    Could Full-Stack Observability Help Lessen IT’s Burden?

    The year 2020 was a landmark year for many reasons, other than the obvious. The IT world saw the first time that cloud service revenues dwarfed spending on data centers. Across the board, budgets have been increasing around cloud adoption. And in this ‘new normal,’ digital experiences are slated to remain just as crucial of a touchpoint as ever. In fact, new findings show that for 85% of users, digital applications are absolutely essential to their daily lives.

    Rising digital demands mean reliance on the cloud is here to stay. But if organizations don’t right-size their cloud workloads, expenses could get out of control. Plus, maintaining world-class user experiences requires a keen insight into the overall performance of apps, systems and solutions. If teams aren’t continually monitoring application performance and business KPIs, the bottom line could suffer.

    I recently met with Joe Byrne, AppDynamics Regional CTO, to discover what full-stack observability (FSO) means for today’s cloud-heavy applications. According to Byrne, DevOps and SRE teams should be maintaining a holistic view of the entire pulse of the application—including non-functional metrics (NFM) alongside traditional security and functional monitoring. Doing so could reduce time spent on root cause analysis and help ensure code is continually meeting business objectives.

    What is Full-Stack Observability (FSO)?

    In the classic definition, observability focuses on measuring the internal status of a system based on external outputs. Application performance monitoring (APM) typically tracks telemetry around logs, metrics and dependencies. IBM, for example, calls the observability trend “a natural evolution of APM data collection methods.”

    Full-stack observability (FSO), said Byrne, takes this one step further to track business-related key performance indicators (KPIs) alongside additional, more traditional software metrics. “FSO helps understand that software is performing optimally and that it is meeting business objectives,” said Byrne. In this way, FSO takes a holistic view to determine not only if an application is functionally sound but if it’s meeting its end goals.

    FSO would consider critical performance actions, such as user experience telemetry. For example, Byrne explains, an e-commerce website could find value in tracking performance levels such as login time, check out time and payment initiation. Tracking such metrics is necessary to pinpoint roadblocks and create actionable steps to remediate holes in the user journey, said Byrne. Increased visibility here could also help engineers quickly respond to issues to better optimize the user experience. In addition, aggregating information at a higher level—to view total products sold per day or total revenue per day, for instance—could inform overall progress.

    How FSO Could Reduce IT Stress

    “We’ve reached a point where people have moved away from brick and mortar and are going online,” said Byrne. These online spaces are more omnipresent — and valuable — than ever. The AppDynamics 2021 App Attention Index found that the average digital consumer uses 42 applications daily. Furthermore, 85% of people now say digital services are a critical part of how they get by in life.

    Mounting digital reliance has resulted in rising digital complexity. Therefore, said Byrne, companies have been moving to the cloud for the ease of scalability, elasticity and the capability of cloud functions. Cloud-native abilities like serverless, Kubernetes, identity and access management (IAM) and other add-on services are helpful perks to avoid reinventing the wheel.

    But of course, lifting and shifting all your current architecture to the cloud is costly; plus, it won’t necessarily eliminate application bugs. The user journey is increasingly vast, and broken touchpoints along the way could hinder performance. This is complicated by increasing third-party dependencies. Yet, more often than not, it is the application’s custom code that is having the issue, explained Byrne.

    Perhaps the most significant benefit of full-stack observability is a reduction of remediation time. “If you don’t have perspective, it can take a long time to run down an issue,” said Byrne. “The cost of finding a root cause can be so expensive.” Having a quick window into network, application and infrastructure hiccups is necessary to quickly address issues and save potentially lost revenue.

    Shifting Performance Monitoring Left

    An Amazon study found that every 100ms of latency costs companies a 1% loss in sales. Knowing the reality of high user expectations, software companies must be especially proactive in how they scale to meet the needs of consumers. As 72% of users believe application problems are the brand’s responsibility, the onus is on companies to avoid negative brand reputation.

    To spot performance hangups early on, Byrne encourages a shift-left attitude toward observability around business outcomes. To date, the shift-left mantra has focused more on code validation and security—necessary elements to rapidly deploy software updates. Now, adding performance testing could be the next iteration to further correlate each software push with end business outcomes.

    “You want to see and validate as much as possible before production,” said Byrne. “Performance testing is a key piece in the overall success of an application.”

    This could involve testing the load time for a specific endpoint—if a key action doesn’t meet a certain threshold, then it would not be promoted. Incorporating performance monitoring within a CI/CD process is a burgeoning practice also referred to as BizDevSecOps.

    There’s No Going Back

    The new normal is upon us, and it’s hard to imagine going back to any semblance of a world where there’s a decreased reliance on technology. From airline bookings to banking and grocery orders—software applications are now intrinsic to nearly every facet of modern life. To compete in such an economy, finely-tuned modular customer experiences that continually optimize business outcomes will truly set companies apart from the rest.

    In short, to maintain high software standards, it’s essential to break down operational silos and work together as a team—the DevOps creed. End-to-end ecosystem visibility could help enable that, ensuring, in Byrne’s words, that “applications perform in a way that meets the needs of the business.”

  • Choosing the Right Monitoring Tools for a Full-Stack Solution

    Choosing the Right Monitoring Tools for a Full-Stack Solution

    According to the marketsandmarkets report on APM tools, the monitoring tools market is expected to grow to $4.98 billion by 2019, at a CAGR of 12.86 percent and the DevOps market would attain market size of $8,763.8 million by 2023, growing at a CAGR of 18 percent during 2017 -2023 (source: markets business insider), with more focus on monitoring and performance management, delivery and operations management, lifecycle management, analytics and other solutions.

    Selection Criteria for Full Stack Monitoring Tools

    With the amount of DevOps tools and new tools coming up in the market, it is highly essential to choose the right tool for the right need before investing in it and expecting the desired outcomes. One has to make several considerations and analysis in choosing the right tool for the need. The monitoring tool you select should be capable of doing full stack end-to-end monitoring and should enable faster troubleshooting and quick remediation.

    Following are some of the key criteria that can be used as factors for the monitoring tools analysis.

    Monitoring Features

    The mix of platforms, operating systems, devices and applications can be extremely diverse in the enterprise, which needs monitoring of infrastructure, platforms, cloud environments, application performance, network monitoring, orchestrations, containers, databases, end user monitoring, covering heterogeneous workloads and its performance. Check out what features are supported in the monitoring tools which suits the need.

    Infrastructure Monitoring

    The infrastructure monitoring tools should measure response time, availability, uptime, CPU usage, memory usage, disk usage, process level usage and load of the servers, components, storage, databases, virtual systems, network switches, security, user permissions, performance and throughput on the application and web servers, DB level queries and DB transactions, latency, transfer rate, connection time of databases, and provide time series data of the measurements, aggregation of data with process level drill down and history of trends. Tools that can do infrastructure monitoring should be able to provide most of the above mentioned measurements.

    Network Monitoring

    Network monitoring tools should provide performance metrics such as bandwidth, latency, responsiveness, different port level metrics, network packets flow, CPU usage of hosts and provide custom metrics and aggregation of these metrics. Network monitoring tools need a comprehensive platform that works across varied network topology (heterogeneous networks (wired/wireless), cloud-based networks and software defined networks). Choose the tools that are capable of auto discovery of nodes, devices, routers, firewall rules, ports and protocols, automatic rules, thresholds and alerts, and which are application-aware, that can detect the applications, services in the network with network path, network flow analysis and provide graphs/tables about network status to determine if the performance problem arises from network or application itself, and can generate network topology map.

    Containers Monitoring

    Monitoring containers/micro services include capturing container level states on CPU usage, memory, network, disk usage, processes running inside the containers, as well as usage of containers hosts, container level logs, containers running/stopped, orchestration level metrics on pods states, cluster stats, container networks, application endpoints at container level, pod level, and system level, performance metrics at container levels as well as services running in it, etc. Containers run in clusters and monitoring ephemeral container workloads are necessary in production; choose tools that provide the above capabilities for container monitoring beyond the out-of-box monitoring data provided by the container orchestrators.

    Application Performance Monitoring

    Application performance monitoring is a key component in full-stack monitoring, where logs are collected and centralized with profiling and tracing available on the application and its dependencies, provides measurements on performance such as end user response time, availability, throughput, error rate, page loads, slow pages, error occurrences, any third-party JavaScript slowness, browser speed, track SLAs, page bloats, distribution of customer base across geographical areas, checks for end user transactions and simulate end user actions with synthetic transactions and provide detailed drill down and correlation data from end user experience, providing full visibility across the application stack. Choose APM tools that cover the profiling, tracing, end user experience monitoring, log monitoring and log analytics, providing data correlation with drill-down capability on transactions and full-stack visibility, at a minimum.

    Cloud Monitoring

    Cloud providers offer monitoring capabilities for the workloads running in their cloud environment providing log trails, performance metrics, network level metrics, in-depth visibility of application components, measurements on microservices, PaaS services, and also offers storage and analysis on instrumentation data. The tools choice can depend on the support for interoperability and integration to multiple cloud providers, exchange of data between different databases/storages, perform transformation/enrichment of data and that which provides visualization of data with correlation and business insights from end user experience perspective with drill down approach and performs auto-remediation, auto-discovery, baseline and predict metrics.

    Deployment Model and Pricing

    Monitoring tools are available in on-premises, SAAS cloud-based deployment models, open source/commercial versions and, depending on the complexity in configuring the tools, maintenance and the support needed, appropriate deployment model and pricing/licensing needs, an appropriate tool can be chosen; however there seems to be increasing trend in the usage of SaaS-based monitoring tools.

    Agent-based/Agentless Approach of Monitoring

    Monitoring tools collect the data from the servers and applications using agent-based/agentless, push- or pull-based approach. The collectors can act as metrics/log shippers, which push the data to the centralized server for storage and processing. Agent-based approach may involve deploying the monitoring agents in the machines considered for monitoring and this can be deployed to the target machines, through configuration management tools/infra automation scripts such as Chef, Puppet or Ansible.  Tools may support auto discovery of machines, which can register itself as the client machine for monitoring. Agent-based monitoring can help monitoring the fellow agents as well. Choose the tool that suits the needs and the effort needed in maintaining the agent-based monitoring tools.

    Alerting

    Notifications set up on the predefined thresholds and rules are a must-have for monitoring tools. Tool should provide capability to send mails and web hooks, integrate to collaboration/ChatOps tools such as Slack or Hipchat and provide the ability to define custom thresholds and run batch alerts at intervals.

    Dashboards 

    Monitoring data and metrics can be used for decision-making and stakeholders’ buy-in, for which centralized end-to-end dashboards on metrics of different sources and components are needed. Many commercial tools provide aggregated as well as detailed correlation information and segregated data of applications under monitoring. The tools must provide out-of-the-box dashboard capability and capable to add/update new dashboards views and create graphs, create custom queries depending on the need.

    Customizability

    Many monitoring tools may provide the features for monitoring different workloads, but determining whether the tools provide customizations on defining custom metrics, custom dashboards, custom checks and plugins, community support on extending the tools features, customizable workflows, customizing thresholds and rules and the ability to write customized queries can influence your choice of tools for monitoring.

    Extensibility

    Despite performing monitoring, things may break; hence, monitoring tools should have extensibility to the collaboration tools which can alert through the proper channel to the team to act upon. Integration with ticket-based controls and incident management workflows, auto-ticketing mechanisms, auto-remediation tools and REST API support are some of the key items to look for when evaluating the monitoring tools.

    Production Grade

    Determine how the monitoring tools can perform in the given load and the support for production cluster setup. The tool should be capable of responding within the defined seconds, for the defined set of servers and time interval and store time series data collected from different sources (servers, devices, applications, etc.) and check whether the monitoring tool can scale up depending on the load. Scalability, performance can largely influence the decision of selecting the monitoring tools.

    Analytics – AIOps

    Commercial monitoring tools provide capability of running machine learning techniques, which can perform anomaly detection in the data, help predict the capacity and demand and maintain inventory based on the metrics. Depending on the advanced use cases the business demands, choose the tools which can provide these capabilities that can accelerate the IT operations maturity level.

    Ticketing and Incident Remediation

    Despite deep monitoring, defects may leak and SRE teams may have to triage and resolve the issues. Autoticketing and autoremediation workflows can be defined in the ITSM tools, which can integrate with the monitoring tools and trigger the remediation workflows based on the alerts defined and notify the SRE teams through different channels. Choose monitoring tools that can be easily integrated with the ITSM tools and provides out-of-box or options to develop custom workflows without much complexity, to achieve use cases such as auto-ticketing based on threshold alerts in monitoring and autoremediation.

    Conclusion

    When choosing the full-stack monitoring tools, choose the tools that offer monitoring capabilities for varied workloads including containers, IoT and comprehensive platform with user interface and integrated dashboards, along with integration and interoperability between the IT operations tools, ITSM tools and AIOps tools, which provide event correlation and analytics Choosing the tools that are right-sized and address the skillset, technology needs, pricing and deployment models, with a focus on must-have features, will enable SRE teams to triage and remediate the issues faster.

    — Lavanya Subbarayalu

  • What to Consider in a Full-Stack Monitoring Solution

    What to Consider in a Full-Stack Monitoring Solution

    In the DevOps space, there are various tools available for monitoring. When building a full-stack monitoring solution, however, selecting the right monitoring tools and designing monitoring based on IT operations’ needs is a critical decision.

    Design Considerations for Implementing a Full-Stack Monitoring Solution

    The following are some of the key design considerations while implementing full-stack monitoring:

    Distributed Monitoring

    Consider choosing the topology that is best-suited to the monitoring needs of the applications. For example, small teams with fewer applications to monitor can choose to implement monitoring tools in a single-server standalone topology, while large IT operations teams with many different monitoring applications can choose a distributed architecture in which the master nodes contain metadata of the monitoring tool, while monitoring and slave nodes are centralized and act as the data collectors and data nodes, then connected together as a centralized monitoring system. Monitoring tools clusters should contain a minimum recommended master nodes and data nodes for high availability. If a network outage occurs, monitoring systems may not be able to detect the underlying problems; however, with distributed slaves, if one slave is offline, a new slave node can be added to the cluster, which performs checks and writes the results back to the data nodes. Choose on-premises or SAAS-based monitoring tools depending on your workloads (cloud/on-premises). Pricing also varies and should be considered.

    Availability

    Site reliability engineering (SRE) teams focus on the reliability, uptime and availability of the applications and thus monitor continuously to identify any potential risks/outages. This mandates that the monitoring tools to be up and running all the time. Most monitoring systems are set up in the same network they monitor, and if a network outage occurs, the monitoring system can’t monitor for security threats. Thus, monitoring tools must be configured inside a high-availability cluster, with availability zones and geo-redundancy regions for availability, backup and disaster recovery.

    Scalability

    It is common to have multiple monitoring systems to monitor multiple locations or data centers with thousands of hosts. Data from each host is collected multiple times daily, which can result in huge load and performance hits. In a distributed monitoring system with slaves distributed, data checks can be performed locally in the data center, the data can be processed and slaves can then send the results to the master. Based on the load, the hardware CPU cores can be utilized to run multiple checks in different threads or slaves can be added to the cluster for scalability and redundancy.

    Storage

    Monitoring systems collect and use data from several workloads for various monitoring needs. This results in continuous monitoring, which generates volumes of data accumulation and can result in slow performance of the monitoring tools to analyze the data, demanding high storage needs to store the data and impacting faster troubleshooting. Monitoring systems use databases that are document-based or time series-based. Time series-based databases can store data with time as points, and modern monitoring systems offer plugins to collect data from various sources and provide a built-in schema of the metrics to be stored to the database. Consider customizing the plugins depending on the metrics needed to be stored, as well as aggregation of data based on the granularity and the necessary data storage timeframe. Data can be aggregated and stored in weekly or monthly tables in the database to minimize the overhead of maintenance cost of the monitoring tools. Configure retention and backup policies to retain the aggregate data for machine learning and anomaly pattern detections.

    Performance

    Performance of the monitoring tools should be tested, given the size and capacity of the master and slave monitoring nodes and the number of hosts to be monitored. Performing various checks in the hosts and then returning data to the server nodes involves lot of processing with high compute needs. Use queues or a transport mechanism to manage the data transfer from the clients to monitoring systems to avoid data loss due to network outage or latency. Most monitoring systems offer a built-in or support a queuing mechanism that performs retries to read/write operations in case of data loss. The slave agent nodes can improve the performance of the monitoring systems by performing data enrichment and data transformation locally, then forwarding the results to the master agents for better and faster reporting. Decide the time intervals you need to monitor the data based on the SLOs defined to optimize the performance of monitoring tool. Monitoring systems can use agent-based or agentless monitoring approaches: In an agent-based system, the data is collected using binaries installed in each host to be monitored, while an agentless monitoring system uses a pull-based approach to collect data from individual hosts. When using an agent-based approach, consider the memory footprint of the agent installation: Use optimized queries for search capabilities/indexes on the primary fields and aggregations in the monitoring system database to improve performance of the data retrieval, graphing and reporting.

    Security

    Configure security with permissions in the monitoring systems. The slave nodes/master systems should communicate securely with each other and access the endpoints and various ports of different processes for performing checks. Configure security at all levels of applications, different components, middleware software and firewalls to allow access to the ports and processes the monitoring systems need to access and monitor. Limit the security access to the defined stakeholders as per the need and administrator permissions to configure the monitoring tools for policies, thresholds, integration with other tools, managing users, backup, retention and other factors.

    Automation

    Because monitoring systems use either agent-based or agentless monitoring modes to collect data, agent binaries or plugins must be installed in every host to be monitored, which can be a tedious and time-consuming task. While a few of the monitoring tools can perform auto-discovery of the hosts and the agent’s binaries, custom configurations can be pushed to the hosts through configuration management tools and configured automatically to perform the checks and forward the instrumentation to the centralized monitoring system. The infrastructure scripts for the monitoring systems need to be version-controlled, and scheduled jobs and maintenance activities—such as cleanup of old data and running enabling batch scripts on the monitoring data to trigger remediate actions—can be deployed using configuration management and CI tools, reducing much of manual effort and maintenance costs. Configuring the monitoring of provisioning, de-provisioning and use of resources can be done through automation scripts in the monitoring tools.

    Testing the Monitoring tools

    Test your monitoring tools as you would any other application. Create test cases to test the configurations used for various checks; test whether right data has been reported from the right host with appropriate labels and tags; check the endpoints and the policies configured; test the latency/data transfer between the agents and master nodes; test whether the data correlation provided, if any, are giving the accurate information with time-bound; test to ensure the dashboards, alerts and visualization graphs are reporting proper data; as well as any machine learning and remediation scripts functional testing. Configure test environments to install monitoring tools in the staging environments to test the behavior, performance and the accuracy of the monitoring data before moving to production environments.

    Choosing Metrics to Set Up

    Metrics provides insights into the system behavior and choosing metrics that are not relevant can add more overhead and poor monitoring. Choose the metrics that provides the health of the overall platform such as throughput, latency and error metrics that can give insights of erroneous results from several sources, success metrics and performance metrics. Configure metrics for individual resources that measure utilization, saturation, errors, availability, performance such as response times, disk/CPU/memory usage, packets inflow/outflow, etc. Choose metrics to configure based on the events logs, such as scaling up/down hosts, containers up/stopped, build or deployment failures, profiling and stack traces, which can provide data correlation, and configure the metrics to enable drill-down from the work-level metrics to resource-level metrics to the event-level metrics, with required granularity and tagged to the scope of the resource. For an efficient full-stack monitoring, configure all the types of metrics to enable faster troubleshooting.

    Defining SLOs

    SRE teams focus more on the uptime, reliability and availability of an application and many factors can impact the reliability. Most commonly, application design can impact the performance of the application. The service-level objectives (SLOs) have to be designed with the desired objective and setting up target for the objective, with a clear definition of how the objective is measured and the conditions under which they are valid. Configure SLOs related to performance and availability that can be measured within the SLA with defined error budgets, and standardize the service-level indicators that can be created as templates for reuse across multiple projects.

    Designing Alerts

    Monitoring systems should know what to monitor, how to alert and what channels to use to communicate the alerts. Alerts can be defined on various criteria such as monitoring the potential issues/events and alert them before they occur (host, disk, memory, CPU metrics); monitoring the expected system behavior and key user actions for green status and alerting if the system behavior deviates monitoring the critical resources and components of the application and its impacts that affect various other parts of the applications and alert if the resources fail. Alerting also could be based on black-box monitoring, in which the alerts are configured based on externally visible factors and real users’ impact, or based on white-box monitoring. Check historic values and remediation actions in the past to determine the threshold values and the priority levels, where high-priority alerts can be sent as pages to on-call engineers and low/medium priority alerts are sent as secondary notifications. The alerts should be designed to provide contextual information such as links to appropriate graphs/logs/attachments that can contain deep information about the events occurred along with correlations. Identify the right team to receive the alerts and the communication channel that best suits the engineers in on-call rotations and responders with escalation plans and mitigation plans. Configure the frequency and the priority level of the alert based on the use cases. Striking balance between low-alerting and overalerting is important and alerting needs to be continuously improved and fine-tuned to avoid false-positive alerts.

    Visualization and Dashboards

    Metrics are collected and stored as time-series data, wherein data can be best represented as a graph with time on the x-axis and one can notice the value changes over time. Configure graphs and dashboards to provide a holistic view of different components that can be visualized over various time scales to understand long time trends as well as recent changes that might be affecting the systems currently. Configure generalized dashboards to visualize the current health metrics as on-screen displays and focused dashboards to drill down with contextual information that can be used for troubleshooting and specific use cases such as capacity planning.

    Incident Remediation

    Designing a well-defined incident remediation process involves integrating monitoring system alerts to create an auto-ticket in the IT service management (ITSM) tools and posting which incident/problem management workflow must be triggered. Identify the potential use cases for auto-remediation, in which a reliable source can identify the potential problem and get the same responses without any manual input for decision-making. For example, a workflow can be designed to integrate with knowledge base to check for known issues and from known components or sources to validate the issue and run auto-remediation actions and notifications bound with SLAs. Extending monitoring systems for auto-ticketing, auto-investigation, auto-remediation and auto-notification through multiple communication channels to the on-call engineers will accelerate full-stack monitoring.

    Machine Learning Analytics

    With volumes of data being generated and stored in monitoring systems, consider machine learning techniques such as predictive machine learning, in which algorithms learn how system behaves normally, provides a resulting model of normal behavior and predicts a range of observation points that lie outside the range that are marked as anomaly in the data, which can be used for capacity planning use cases. Design test cases for testing the machine learning algorithms to avoid any false-positives that incorrectly identify anomalies or result in malicious events going unnoticed as normal behavior.

    Understanding the needs of monitoring in your environment and then choosing the right tools, and implementing them with various design considerations in place, with continuous monitoring and optimizing further, can accelerate SRE (IT operations) teams to achieve full-stack monitoring.

    — Lavanya Subbarayalu

  • Benefits and Drivers of Full-Stack Monitoring

    Benefits and Drivers of Full-Stack Monitoring

    As organizations undergo digital transformations, there is increasing need to modernize the way IT operations is managed and to deliver applications faster, while at the same time delivering business value by managing applications’ availability, performance and outages in a cost-effective way.

    Because outages have a huge impact on revenue, there is a growing demand for site reliability people and monitoring systems, which can enable them to handle applications’ uptime, performance and reliability with a proactive and predictive monitoring approach.

    This article series focuses on the need for full-stack monitoring in today’s digital operations, the considerations for selection of monitoring tools and design considerations for implementing a full-stack monitoring solution.

    Why is Full-Stack Monitoring Necessary?

    Full-stack monitoring refers to the capability of monitoring all of the following:

    1. Infrastructure monitoring—Monitor servers, network, containers, cluster orchestrators, cloud, middleware, app servers, web servers and databases.
    2. Application performance monitoring—Perform log monitoring and analytics, synthetic transactions, sql transactions, data correlation with metrics and logs, end user experience monitoring, profiling and transaction-level tracing with code-level visibility.
    3. Performing end-end monitoring across the workloads in on-premises/cloud/hybrid environments through a centralized dashboard/visibility across the stack.
    4. Provide alerts, analyze, act and remediate the incidents faster, using the instrumentation data leveraging IT service management (ITSM), ChatOps and collaboration tools along with monitoring tools.
    5. Perform Analytics on the data being used to derive insights and predict demands.

    The following are some of the needs driving enterprises to implement full-stack monitoring:

    • Business agility: Business needs to deploy very frequently to sustain the market demands
    • Application reliability: Site/application platform is expected to be reliable, available and scalable, as outages impact revenue and may result in losing business and customers.
    • Bandwidth bottlenecks can impact the application performance, leading to potential business loss and needs network-level deep monitoring.
    • Failure can happen at any level and go unnoticed, leading to overhead in recovery and remediation; hence, monitoring from various sources and at different angles with varied levels are needed.
    • Application complexity and dependencies: Applications use multiple-third party components, multiple hosts and several underlying services, components and infrastructure. If any of these do not function as expected, it can degrade an application’s performance and result in issues/defects going unnoticed—and often reported by end users.
    • Cloud born applications, which are multi-tiered and multi-layered, use mix workloads (PaaS + IaaS) and monitoring of different sources/workloads has to be done for distributed hybrid environments (multi-cloud).
    • Use of virtualization and container technologies, which have enabled environments with dynamic and ephemeral states and such workloads, need monitoring at the orchestration and container levels.
    • Lack of end-to end monitoring tools and lack of visibility across all IT assets and workloads, which has increased the need of centralized monitoring data. The SRE teams need to get stakeholders’ buy-in based on the monitoring data to push back on the bad releases and take calculated risks. They also need to cater different types of monitoring data to different stakeholders and line of business (LOB) owners and overcome cultural challenges to get stakeholders’ buy-in.
    • IT operations needs to forecast an application’s behavior, capacity and performance, based on how the system performs in the given conditions. Machine learning of monitoring data and patterns can help predict capacity and anomaly.
    • Large volumes of data are collected from multiple sources giving out metrics, logs, etc., for various components of the stack. The IT operations team needs in-depth information about the components and dependencies along with metrics to provide contextual/correlation information. They need to correlate the data to determine symptoms and causes for faster troubleshooting. As such, they need monitoring tools that can provide correlation out-of-box to reduce the effort involved in triaging.
    • Configuration changes: Deployment of application/infrastructure/services are done through infrastructure as code automation scripts, which may contain issues in configuration. Lack of testing infrastructure, services and application end to end can increase the number of issues to be resolved; hence, there is a need of monitoring right from provisioning of resources, usage, maintenance and de-provisioning of resources.
    • Lack of end user perspective and business metrics from traditional tools, coupled with the need to eliminate toil and perform faster triaging/post-mortem and automated incident resolution, demands use of new-age full-stack monitoring tools.

    Growing business demands and agility, application reliability, deriving business insights and end user experience analytics are the key drivers for implementing full-stack monitoring. Hence, the IT operations traditional approaches are rapidly being transformed to digital operations (SRE operations), along with DevOps practices being widely adopted across enterprises.

    In the second and third articles in this three-part series, we’ll look at the considerations for selection of monitoring tools and design considerations for implementing a full-stack monitoring solution.

    — Lavanya Subbarayalu