devopsy.kinsta.cloud

Tag: Prometheus

Attaching Evidence to Alerts: an Enrichment Sidecar for Alertmanager

Every on-call engineer knows the ritual. A page lands in Slack: the 5xx rate on a service is above the threshold. The message tells you that something is wrong and nothing about what. So you open Kibana or Loki in another tab, set the time window to the last few minutes, filter by the service, and start reading. In my team this took two or three minutes on every alert, and these are the worst minutes of the incident, because nothing is being diagnosed yet.

The obvious request is to put the last few error lines into the alert message itself. I tried to do this inside Alertmanager, and it cannot be done there. It helps to understand why before building anything around it.

Why Alertmanager Cannot Do It

Alertmanager renders notifications through Go templates, and a template can only use what is already attached to the alert: the labels and the annotations which Prometheus put there when the rule fired. There is no mechanism which would call Elasticsearch, Loki or any other HTTP API at the moment a notification is sent. Annotations are rendered from the alerting rule at evaluation time, inside Prometheus, and Prometheus does not talk to log stores either. Fetching data at send time is simply not a capability of the design.

So the logs have to come from a separate process which stands next to Alertmanager.

The Sidecar

What runs in production now is a small service next to Alertmanager, and its shape came out of two failed attempts, so I will describe it through them.

My first attempt was a bot which received alerts and posted them to Slack itself, with the logs already attached. It looked good in a demo, but it was a mistake because the bot had become the paging path. Any bug in my code, any timeout of the log store, and nobody gets paged. I threw that version away quickly. Alertmanager has to stay the thing which posts the alert, exactly as before, and whatever adds context has to be allowed to die without anyone noticing. So the enricher hangs off a second receiver: the route which sends the alert to Slack also duplicates it to a webhook, and behind the webhook stands my service. When it is down, the page arrives as it always did, just without the extra.

The second attempt posted the logs as its own message into the channel, right after the alert. Within a day the channel was hard to read: two messages per alert, and during an alert storm they interleaved, so the logs stood three messages away from their page. The fix was to reply into the thread of the message which Alertmanager had already posted. The channel stays as it was, one message per alert, and whoever opens the incident an hour later finds the logs from the exact minute of the page sitting under it.

Finding the Message to Reply To

That thread reply is the one interesting engineering problem in the whole service. The enricher receives the alert from Alertmanager, and it has to find the Slack message which corresponds to this alert, and nobody tells it which one that is. When the webhook fires, the service reads the last couple of minutes of channel history and looks for a message which carries the same alert fingerprint. Our Slack template rendered the fingerprint into the message, so the match was exact; before I added that, I matched on the label set which appears in the message text, and that worked too. Once the message is found, the reply is one API call with thread_ts.

Then production corrected the details. Alertmanager re-sends a firing alert on repeat_interval, and my version at first posted the same log dump under the same message again and again, so now the service remembers which fingerprints it has already enriched. The lookback for matching is limited to about two minutes, since replying to some old message which happens to carry the same labels is worse than staying silent. The query into the log store is built from the labels of the alert (service, namespace, severity) and takes the last few error-level lines, ten or so. I keep it that short on purpose: the reply is a hint for the person, and the full picture is in the log store anyway. One more thing I did not expect: an empty result is also worth posting. When the reply says there were no matching error logs in the last five minutes, the person on call can broaden the investigation to the infrastructure or the alerting rule itself, and this saves the same minutes.

What It Changed

The service is a few hundred lines, and in operation it has been boring, and boring is what I want from anything standing near paging. The visible change for the team: when you open Slack on a page, the first evidence is already under it, and the phase of “which dashboard, which index, which time window” is gone from incidents. Time to diagnosis dropped on every incident which had logs behind it, without any change to the services themselves.

Logs turned out to be only the first thing worth attaching. The same service can reply with anything which Alertmanager cannot know at render time: a runbook link picked by label, the marker of the latest deploy of the affected service (the first question in half of the incidents anyway), or a dashboard link already scoped to the incident window. Alertmanager alerts, and the sidecar carries what the responder reaches for in the first minute.

If you run Prometheus with Slack, the whole thing is one small service plus one extra webhook receiver in the Alertmanager config. Do the idempotency bookkeeping first; it is the first thing which bites.

Daniil Romashov is a Site Reliability / DevOps engineer working on reliability and observability for high-load systems. Open-source work: github.com/youngpabl0.

October 8, 2026
Scalable Jenkins Management: Empowering Enterprises With Centralized Control for 150+ Instances

If you have ever tried to keep more than a handful of Jenkins instances healthy at the same time, you already know the problem: every master drifts a little differently. Plugins fall out of sync, storage quietly fills up, one team’s “quick fix” becomes another team’s outage, and nobody has a single view of what is actually running across the organization. Multiply that by 150+ instances spread across on-premise data centers and multiple clouds, and manual administration stops being an option. That was the starting point for a project we shipped in three phases over about 20 months: a centralized UI and automation layer that gives platform teams one place to upgrade, back up, roll back, monitor and manage access for every Jenkins instance in the company — without each team having to become Jenkins administrators themselves.

Why Centralization Was Overdue

Jenkins is famously easy to stand up and famously easy to let sprawl. As the instance count grew past the point where any one person could reason about the whole fleet, a handful of recurring pain points kept surfacing: inconsistent plugin versions that broke pipelines after ad-hoc upgrades, backups that existed on some masters and not others, no unified alerting when an instance ran low on disk or memory, and job configurations — especially non-lightweight checkouts — quietly burning far more compute than they needed to. None of these problems are exotic. What made them expensive was scale: a fix that takes ten minutes on one instance takes days when it has to be repeated, inconsistently, across 150 of them.

Architecture: A Control Plane, Not a Bigger Script

Rather than writing a longer shell script, we designed the system as a proper control plane. The frontend is a React and Redux dashboard with Material-UI components, talking to a Python backend split across Flask and FastAPI for synchronous admin actions and real-time data syncing. Underneath that sits the part that does the actual work: idempotent Ansible playbooks for upgrades and backups, Kubernetes and Helm for dynamic Jenkins agent provisioning, and ArgoCD driving GitOps-style rollouts across AWS and Azure. Observability runs through Prometheus and Grafana for metrics and the ELK stack for log aggregation and alerting, so the dashboard is showing live fleet health rather than a static inventory.

Figure 1 — The centralized control plane layered above the Jenkins fleet.

Idempotency turned out to matter more than almost anything else in the design. Because playbooks can be re-run safely, rollbacks stopped being a stressful, manual scramble and became a predictable, repeatable action — which is what let us expose a “self-service” layer to teams instead of routing every request through a platform engineer.

What the Dashboard Actually Does

● One-click upgrades and rollbacks, executed as idempotent Ansible plays across any subset of instances

● Differential backups to S3-compatible storage via rsync, scheduled centrally instead of per-instance

● Dynamic slave/agent provisioning through Kubernetes Helm charts, scaling capacity to actual job demand

● Centralized alerting for downtime, storage, CPU and memory thresholds via Prometheus and ELK

● Advanced job reporting: failed jobs, unused jobs, non-lightweight checkouts and high-resource jobs, surfaced so teams can fix their own inefficiencies

● Org-wide access management and audit logging, supporting SOC 2 and GDPR requirements

● A self-service portal so application teams can perform routine tasks without filing a platform ticket

Process and the Hard Parts

We ran this as a standard Agile delivery: bi-weekly sprints, daily stand-ups, and the usual Scrum ceremonies, starting with stakeholder interviews and Figma wireframes before any backend code was written. Integration testing simulated the full 150+ instance fleet in Docker and Kubernetes before anything touched production, and the test suite combined Jest for unit tests, Cypress for end-to-end UI coverage and Locust for load testing under concurrent fleet-wide operations. The genuinely hard problems were not the individual integrations — they were the seams between them. Keeping the dashboard’s view of fleet state synchronized in real time without introducing lag took several iterations of the polling and event model. Rollbacks needed to guarantee data consistency even mid-failure, which is what pushed us toward strictly idempotent playbooks. And because the fleet spanned on-premises hardware alongside AWS and Azure, we ended up writing custom adapters just to give every environment a consistent interface to the control plane.

Results

The clearest way to describe the payoff is in the numbers teams saw after rollout: significantly lower administrative overhead, a meaningful cut in infrastructure spend from better resource allocation and less downtime, and a large drop in time engineers spent on manual Jenkins upkeep — time that went back into actual product work.

Figure 2 — Reported reductions in overhead, infrastructure spend and manual task time following full rollout.

What’s Next

This kind of fleet-wide control plane is not a novel idea on its own — there is a long history of tools for orchestrating multiple Jenkins instances. What made the difference here was combining self-service automation with genuinely actionable job reporting, so that the platform team’s job shifted from firefighting individual instances to setting guardrails the rest of the organization could operate within safely. The next phase looks at predictive resource scaling and extending the same self-service model to non-Jenkins CI/CD tooling as the organization’s pipeline mix diversifies.

August 24, 2026
Reducing MTTR: A Practical Guide to Correlating Incidents with AIOps
At 2 a.m., a payment service starts throwing errors. Within minutes, your observability stack fires off 40 alerts: Elevated latency on three services, a spike in 5xx errors, a memory warning on a downstream cache and a handful of dependency timeouts. Somewhere in that noise is the one signal that actually explains what broke. Finding it manually, alert by alert, is how a 5-minute fix turns into a 45-minute outage.

This is the core problem AIOps was built to solve — not more dashboards, not more alerts but correlation. By automatically grouping related signals across metrics, logs and traces, AI-driven correlation collapses an alert storm into a single, prioritized incident with a probable root cause attached.

In this guide, you’ll learn how incident correlation actually works, how to set it up on top of your existing observability stack and how to measure whether it’s actually reducing your mean time to resolution (MTTR). By the end, you’ll have a practical framework for moving your team from manual alert triage to AI-assisted incident response.

Why MTTR Breaks Down Without Correlation

MTTR is typically broken down into four phases: Detection, triage, diagnosis and remediation. Most teams invest heavily in detection — that’s what observability tooling is good at. The bottleneck almost always sits in triage and diagnosis, for a simple reason: One root cause typically produces many alerts, and someone has to manually figure out which alerts belong together before the diagnosis of the actual problem can even begin.

A few patterns show up repeatedly:

Alert storms mask root cause. A single upstream failure can cascade into dozens of downstream alerts, each looking like an independent incident.

Signals live in silos. Metrics, logs and traces often sit in different tools or dashboards, so the engineer on call has to manually stitch together a timeline.

Tribal knowledge bottlenecks diagnosis. The person who ‘just knows’ that a cache-eviction warning usually precedes a checkout failure isn’t always the person on call.

AI-driven correlation directly targets this middle stretch of the incident life cycle — the time between ‘something is wrong’ and ‘here’s what’s actually wrong’ — which is usually where the most time is lost.

Prerequisites

Before you begin, make sure you have:
1. An existing observability stack producing metrics, logs and traces (StackGen’s ObserveNow or a Prometheus/Grafana/Loki/Jaeger-style setup)
2. Alerting already configured on key services, even if it’s currently noisy
3. Admin access to configure integrations and correlation rules
4. At least one recent incident with alert history you can use to validate correlation output
5. Estimated time: 30–45 minutes for initial setup, plus 1–2 weeks of tuning based on real incidents.
Step 1: Connect Your Observability Stack

AI-driven correlation is only as good as the data it can see. If your metrics live in one tool, logs in another and traces in a third with no shared identifiers, correlation has nothing to work with.

Start by making sure your telemetry shares common context — service name, environment and ideally a trace ID or request ID that threads through logs and traces.

# Example: Shared labeling convention across metrics, logs and traces

service: checkout-api

environment: production

trace_id: “{{request.trace_id}}”

If you’re using ObserveNow, connecting metrics, logs and traces into a single pane is largely a matter of pointing your existing Prometheus, Loki and Jaeger (or equivalent) instances at it — StackGen integrates with these directly rather than requiring a rip-and-replace.

Expected Result: A unified view where a single service’s metrics, logs and traces are queryable together, ideally with a shared identifier connecting them.

Troubleshooting: If correlation later feels inaccurate, this is almost always the first place to look — inconsistent service-naming or missing trace propagation breaks the AI’s ability to link signals correctly.

Step 2: Configure AI-Driven Correlation Rules

With unified telemetry in place, the next step is telling the correlation engine what ‘related’ should mean for your environment. Most AIOps platforms, including Aiden AI Copilot, combine two approaches:

Topology-Aware Correlation: Using service dependency maps to group alerts from services that call each other

Pattern-Based Correlation: Using historical incident data to learn which alert combinations tend to co-occur

# Example: Correlation policy

correlation_policy:

time_window: 5m

group_by:

– service_dependency

– shared_trace_id

confidence_threshold: 0.75

The time_window matters more than it looks. Set it too short and you’ll miss slower-cascading failures; set it too long and unrelated incidents start getting merged. Five minutes is a reasonable starting point for most microservice architectures — narrow it down if your services are tightly coupled with fast failure propagation, widen it for slower, more decoupled systems.

Expected Result: Incoming alerts are grouped into incidents rather than arriving as isolated tickets.

Troubleshooting: If everything gets grouped into one mega-incident, either your confidence threshold is too low or your time window is too wide. Tighten both and re-test against a known incident.

Step 3: Test Against a Real Incident

Don’t wait for a live outage to find out if your correlation setup works. Pull the alert history from a recent real incident and replay it against your new configuration.

Ask three questions about the output:
1. Did the correlation engine group the alerts that actually belonged together?
2. Did it surface a root-cause candidate that matches what you now know actually happened?
3. Did it leave out anything that should have been included or wrongly include something unrelated?
This is also where Aiden/AIOps automated root cause analysis is worth validating directly. It should be able to point to the earliest anomaly in the correlated group and rank likely causes, not just group alerts together and leave the analysis to you.

Expected Result: The correlated incident matches (or comes close to) the retrospective understanding of the incident, with the root-cause candidate ranked at or near the top.

Step 4: Monitor, Tune and Automate Remediation

Correlation isn’t a one-time setup. It improves while identifying more of your real incident patterns. Over the first few weeks:
- Review every correlated incident and confirm or correct the grouping.
- Watch for recurring false groupings (two unrelated services that keep getting merged) and adjust topology mappings.
- Track false-negative correlations — alerts that should have been grouped but weren’t.
Once correlation accuracy is consistently high, you can layer-in automated remediation for well-understood incident patterns — for example, automatically restarting a service instance when a specific correlated pattern (memory leak signature + latency spike + no recent deploy) is detected with high confidence. This is where Aiden for SRE typically comes in: Using the correlated, root-caused incident as the trigger for a pre-approved remediation action, rather than waiting for a human to read the alert and act.

Expected Result: A shrinking gap between detection and remediation, with routine incident patterns increasingly self-resolving.

Testing Your Setup

Before trusting correlation in production, run it through a structured validation pass:
- Replay 3–5 historical incidents of varying severity and complexity.
- Compare AI-correlated groupings against what your team determined manually during the actual incident.
- Check root-cause ranking accuracy — was the true cause in the top 1–2 candidates?
- Confirm the time to first correlated alert is meaningfully faster than your team’s manual triage time.
If correlation accuracy is inconsistent across incident types, it’s usually a sign that your topology mapping is incomplete for certain services, which is why it is worth revisiting Step 1 before adding more correlation rules on top of gaps in the data.

Real-World Results

Teams that move from manual alert triage to AI-driven correlation typically see improvement concentrated in the triage and diagnosis phases specifically, since that’s the part of MTTR correlation is designed to compress:
- Alert Volume per Incident: Often reduced from dozens of individual alerts down to a single correlated incident
- Time to Root-Cause Identification: Meaningfully faster when root-cause ranking is accurate, since engineers stop manually cross-referencing dashboards
- On-Call Burden: Fewer distinct pages per incident, since related alerts are grouped rather than firing independently
Your specific numbers will depend heavily on how mature your topology mapping is and how much historical incident data the correlation engine has to learn from — treat the first few weeks as a tuning period, not the final benchmark.

Next Steps

Now that you have AI-driven correlation running, you can:
- Extend correlation rules to cover infrastructure-layer signals (node health, network) alongside application-layer telemetry
- Pilot automated remediation on your highest-confidence, lowest-risk incident patterns first
- Expand from Aiden for SRE into Aiden for DevOps if pipeline and deployment failures are a significant source of your incident volume
August 13, 2026
A Modern Approach to Multi-Signal Optimization
In modern cloud operations, we’re not just facing a data tsunami; we’re caught in a signal storm. An application’s throughput metric looks flat, but its underlying pod central processing unit (CPU) usage is spiking. A new deployment manifest is applied successfully via kubectl, but user latency is slowly creeping up. These signals are a mix of complementary and conflicting information, creating a confusing picture that paralyzes decision-making.

The problem isn’t the signals themselves, but our approach to them. We consider all metrics equally important, which leads to confusion. Without a framework, a spike in a pod’s CPU usage is given the same weight as a drop in user sign-ups. This lack of structure is the root cause of the analysis paralysis many engineering teams feel today.

This confusion has specific negative impacts. It leads to slow incident response (MTTR) as engineers waste time toggling between Grafana dashboards, Datadog traces and kubectl logs, manually trying to correlate conflicting signals. It also drives costly ‘best guesses’. When a pod begins restarting, the loudest symptom is a resource issue. Therefore, the reflexive action is to scale up behavior that contributes directly to the industry-wide 32% cloud waste statistic. Most importantly, you cannot build reliable automation on a foundation of chaotic, unclassified signals.

To escape the storm, we need a deliberate multi-signal optimization strategy. This strategy’s success hinges on a critical first step: Analyzing and classifying all signals into a logical framework. Only then can we build a solution that is not just fast, but also precise and cost-effective.

Creating Order with Metric Classification

The practical first step in this strategy is to organize the chaos. It’s how you make sense of the noise before you can act on it. We do this by implementing a framework called the signal hierarchy.

Outcome metrics are the ‘what matters’ or ‘north star’ metrics. These define business and user success and are what you are solving for. They are always application-level indicators such as p99 latency, application error rates, cost per transaction or user conversion rates.

Primary metrics are the ‘external drivers’ or ‘causal signals’. These are independent events that act upon your system causing it to react. In the Kubernetes world, these include changes in user traffic from an ingress controller, application programming interface (API) requests per second, a code release or a configuration change or the trigger of a CronJob.

Secondary metrics are the ‘internal symptoms’ or ‘diagnostic signals’. They show how the system’s infrastructure is responding to the primary drivers. These are the classic infrastructure metrics we’re all familiar with: Pod CPU and memory usage, network I/O and disk saturation.

The goal of this classification is to understand the role of each signal. This simple act of categorization allows a fix or a resolution to be found systematically and logically: It prioritizes negative changes in outcomes, looks for causes in primary signals, and uses secondary signals for evidence and diagnosis.

The Strategy in Action: From Classification to Correlation

Let’s walk through how this multi-step strategy works in a real-world Kubernetes environment. Imagine a sudden, sharp increase in the p99 latency for an e-commerce checkout service running as a deployment.

The first step is real-time classification. As signals stream in from Prometheus, the Kubernetes API and CI/CD tools, it should go through a classification system to get classified appropriately. The p99_latency_spike would be tagged as an outcome. A pod_cpu_high metric would be tagged as a secondary. The other two signals are identified: A recent increase in api_requests_per_sec and a deployment_event from ArgoCD are tagged as primary. This classification system, which could be a static classifier or an ML-based classifier, provides an essential context for what comes next.

With signals now classified, we need an approach to correlate them. This process prioritizes the negative outcome (high latency) and immediately looks for a causal primary signal to explain it. Here, it finds the ‘bad deployment’ narrative: The p99_latency_spike (outcome) began shortly after the deployment_event (primary), which was followed by the pod_cpu_high metric (secondary). These three signals complement each other perfectly.

However, what if the signals were conflicting? Imagine the latency is high (outcome), but pod CPU and memory are normal (secondary) and ingress traffic is flat (primary). The lack of complementary infrastructure symptoms is just as important. It tells the engine to correctly rule out a resource bottleneck in the service itself and widen its search for other primary causes such as the failure of a downstream API dependency.

The diagnosis is now based on a coherent, correlated story; not a single, noisy alert. In our first scenario, the system generates a precise operation — kubectl rollout undo — instead of the simplistic and costly kubectl scale deployment. This generated operation is then dispatched through a safety pipeline for verification before execution.

The result is that a precise and effective action was taken. The right action was taken because the system first classified the signals to understand their roles, then correlated them to find the true narrative.

Proactive Classification: Focusing on the Metrics That Matter

The signal hierarchy isn’t just a reactive tool for incident response. Its most profound impact comes from applying it proactively. For any given application, we can perform this analysis ahead of time, creating a focused model of what truly drives its behavior and performance.

Instead of waiting for an issue, we can analyze a service’s telemetry landscape in a steady state. Out of the hundreds of metrics an application performance monitoring (APM) tool may expose for a single service, we can pre-identify the critical few. For our checkout service, this means defining that p99_latency and payment_error_rate are its core outcome metrics, and ingress_traffic and inventory_api_calls are its key primary drivers.

The immediate benefit is a dramatic improvement in efficiency. When an alert on an outcome metric fires, the system’s initial hypothesis formation is no longer a search across hundreds of potential signals. It begins with a much shorter, pre-validated subset of high-impact metrics, making the correlation step exponentially faster. But the true power of this pre-identified model is that it allows us to move beyond rapid reaction into the realm of prediction. By continuously monitoring anomalous patterns in the key primary and secondary drivers, the system can predict imminent issues and take corrective action before the user-facing outcome metric is impacted. This is the shift from firefighting to fire prevention.

This proactive approach also yields a powerful secondary benefit: Optimizing the observability stack itself. By identifying which metrics are truly causal and which are simply redundant or highly correlated symptoms, teams can make informed decisions to reduce their monitoring overhead. This allows you to potentially stop measuring, storing and paying for unnecessary telemetry without losing any meaningful visibility, thereby reducing the cost of your observability platform.

From Strategic Insight to Business Impact

A disciplined multi-signal strategy is more than a technical improvement; it’s a direct driver of business value, moving organizations from costly reaction to profitable proaction. The impact of this strategic approach is clear, measurable and significant.
- Crushing Cloud Waste: By strategically correlating traffic patterns (primary) with resource usage (secondary), we’ve found that over 60% of production workloads can be tuned for cost improvements of 20% or more.
- Unlocking ‘Long Tail’ Savings: For the bottom 50% of an organization’s resources (such as staging and development environments), this strategy consistently yields average cost reductions of 30–40% by understanding usage schedules and dependencies.
- Boosting Performance and Reliability: By automatically correlating negative outcomes to their primary triggers, root cause analysis becomes an instantaneous process. This is why our model leads up to 78% less application downtime.
The key to conquering modern cloud complexity isn’t another dashboard or another tool. It’s adopting a holistic strategy. This strategy must begin with the foundational step of classifying signals to create order from chaos. From this foundation of clarity, you can build powerful, intelligent automation that saves money, protects revenue and — most importantly — allows your engineers to focus on innovation.

Stop chasing individual signals in a storm of data. It’s time to adopt a multi-signal optimization strategy and let an autonomous engine turn that chaos into clear, actionable and profitable intelligence.

KubeCon + CloudNativeCon North America 2025 is taking place in Atlanta, Georgia, from November 10 to 13. Register now.
October 27, 2025
Grafana Labs Makes Observability More Accessible

At a GrafanaCon event today, Grafana Labs unveiled an update to its core Grafana visualization platform. The company is offering an Explore Metric module which it says can identify an issue’s root cause — and to do so without anyone having to launch a query against the open source Prometheus platform that is at the core of the Grafana Cloud observability platform.

That’s not the only enhancement. Version 11 of Grafana also makes it easier to edit and share dashboards, in addition to streamlining the number of alerts that a single incident might generate. In addition, Grafana Labs is now making available a curated edition of the OpenTelemetry software dubbed Grafana Alloy that it will extend and support alongside the rest of its open source software portfolio.

Finally, the company is updating Loki log management to make it simpler to streamline the storage of log data.

Grafana Labs CTO Thomas Wilkie said that collectively these updates will make it easier for a wider range of IT professionals, including application developers, to invoke the capabilities of a Grafana Cloud platform based on open source software.

As IT environments become more complex, the need for observability platforms becomes more pronounced. IT teams typically have access to a raft of monitoring tools for tracking pre-defined metrics but their ability to launch queries to surface the root cause of an issue has been limited. As observability platforms continue to evolve, the line between observability and monitoring continues to blur, said Wilkie.

Less clear is to what degree observability platforms might enable IT teams to rationalize existing monitoring tools. In the meantime, the pace at which IT teams are embracing observability varies widely from one organization to another. IT teams need enough expertise to launch relevant queries to get the full return on their investment in an observability platform. In the absence of that expertise, acquiring an observability platform becomes less compelling, in the short term.

In theory, there soon should come a day when machine learning algorithms and other forms of artificial intelligence (AI) make it easier for organizations to identify issues without having to craft queries. But it’s not likely AI will eliminate the need for site reliability engineers (SREs), said Wilkie. Instead, AI will make it simpler to onboard new members to a DevOps team while deploying more applications than ever, he added.

It’s still early days so far as adoption of AI and observability is concerned but there will be an inevitable convergence. The overall goal should be to make it easier to troubleshoot highly distributed application environments that today span cloud, on-premises and edge computing applications. That’s critical because as application environments become more distributed the overall size of most of the IT teams tasked with managing them is not increasing. As such, that pace at which DevOps workflows can be automated will prove crucial.

In the meantime, IT teams should expect it will become more difficult to triage complex IT environments before it hopefully becomes simpler thanks to the rise of observability, automation and AI.

Photo credit: tito pixel on Unsplash

April 9, 2024
Optimizing Prometheus Implementations With Managed Services

Prometheus is a leading metrics-based open source monitoring solution and time series database used across the IT community to collect metrics from systems, applications and services. According to the 2021 Cloud Native Survey from the Cloud Native Computing Foundation (CNCF), the use of Prometheus in production has reached 65%, an increase of 43% year-over-year.

“As cloud-native stacks have grown and matured, observability has become crucial for visibility into application metrics, performance, alerting, and insights,” said Chris Aniszczyk, CTO of CNCF. “Prometheus is a key element of this, enabling organizations to build a foundation of observability.”

If you are using Prometheus, it is probably because of its many advantages including reliability, easy initial implementation and its open source nature. However, in addition to these benefits, Prometheus also brings several challenges if you are deploying and maintaining the solution on your own.

Facing the Four Main Prometheus Challenges

Prometheus users face four primary challenges that can be alleviated by the right service provider:

Challenge One: Support

Prometheus users face the dilemma posed by any open source tool: There is no vendor behind the solution to deliver technical support. It is easy to take tech support for granted, but when you are working with a technology that is not supported, even a small issue can become a substantial problem. A commercially-backed product can provide a critical level of expertise you do not have in-house, and take on responsibility for keeping the system up and running.

Challenge Two: Maintenance

The greatest challenge you’ll likely face with Prometheus is the amount of time your team must spend configuring and maintaining the system.

Just one example of how Prometheus drains your resources is the use of third-party exporters to expose metrics in the Prometheus format. These integrations are designed by members of the open source community an do not come with any support; upgrades are hit-or-miss.

Exporters are difficult to maintain. First, there are often multiple exporter options and choosing the right one for each of your services is time-consuming. Second, you can easily have hundreds of different services running in your environment and your team can spend hours every day making sure the exporters are working properly.

Another maintenance issue involves PromQL, the Prometheus query language. PromQL is a custom query language, and it is very complex and difficult to learn. Having to rely on your one PromQL expert for all Prometheus queries across the organization is simply not practical or sustainable.

With all these maintenance requirements, you will have some team members spending all their time managing and maintaining a Prometheus environment, eating away at resources that could be used to focus on business needs and innovate new applications.

Challenge Three: Scalability

Users tend to deploy Prometheus in silos, with a separate server for each cluster. Over time, you may have tens or even hundreds of Prometheus servers in your environment that you are not aware of. This leads to scalability issues regarding consistency, maintainability and a global view of your metrics.

In addition, storage is a major hindrance to scaling Prometheus, because the system typically stores metrics for about two weeks. If you need to analyze metrics older than a couple of weeks, you run into a serious problem.

Challenge Four: Correlating Data

Data does not have much value if you cannot correlate it and, unfortunately, Prometheus does not provide correlation for metrics to other elements in your environment like processes or cloud regions. This makes it difficult to pinpoint the source or impact of a problem.

Options for Solving Prometheus Challenges

Options are available to help you with Prometheus but most do not solve the four major issues outlined above. The following are your options:

APM: APM vendors all have some level of support for Prometheus, but this is an add-on to their platform. They offer an ingest tool that can transform Prometheus data into a format you can query natively within the APM platform. In reality, they are not helping you utilize Prometheus but instead are forcing you to work within the constraints of their system.

Cloud tools: Some cloud hyperscalers offer solutions to support Prometheus. These tools can help with deployment and infrastructure maintenance, but you are still stuck doing all the other configuration and maintenance work.

Managed service: The preferred option is a Prometheus managed service, which can alleviate all four of the main challenges posed by open source Prometheus. The right optimized managed service provider delivers support, maintenance, long-term storage and data correlation.

Must-Have Capabilities for Your Prometheus Managed Service Provider

The following are essential capabilities you should look for when choosing a Prometheus managed service provider:

Prometheus compatibility: You need an optimized managed service provider whose platform is designed to be natively compatible with Prometheus, consuming data in the Prometheus format and allowing you to run PromQL queries on the data.

Simplified PromQL: To eliminate the reliance on PromQL experts for every query, a managed service provider should simplify the use of PromQL, without breaking compatibility.

Exporter management: The right optimized managed service provider will be able to streamline the selection, deployment and management of exporters, saving you a significant amount of time and effort.

Automated discovery: Tracking and optimizing all your services can be challenging. For this reason, a Prometheus-managed service should be able to automatically discover the services running in your environment and help guide you with configuration.

Metric Storage: One of the shortcomings of Prometheus is the lack of long-term storage. Consequently, a Prometheus managed service provider should provide a database for long-term storage of metrics, so you can perform comparative analyses of the data, from month to month and even year-to-year.

Prometheus compatibility: If you already use open source Prometheus, your teams are used to using things like PromQL and Grafana dashboards. You should select a managed service provider that is Prometheus-compatible so your team can use the same dashboards and queries they have always used to view the data.

Conclusion: Optimized Valuable Tool

Prometheus is not a standalone solution. But it is an optimized valuable tool to collect metrics for performance analysis and it is appreciated by your architects, developers and IT Ops pros. While Prometheus may have some limitations, it is worth finding a solution to these issues; that solution is a managed Prometheus service. The right managed service provider can give you the best of both worlds—all the benefits of Prometheus without the time, resources and expense needed to run it effectively.

September 16, 2022
Grafana Labs Adds Visual Tools to Simplify Prometheus Queries

At its GrafanaCONline event, Grafana Labs today announced an update to the open source Grafana dashboard. The update adds visual query tools to make it easier for IT professionals of any skill level to launch queries against the Prometheus monitoring platform or the company’s Grafana Loki log aggregations framework.

In addition, Grafana said the open source edition of Grafana OnCall, a scheduling and paging application, is now also available as open source software that can be used in on-premises IT environments. Previously, Grafana OnCall was only available as part of the managed Grafana Cloud service.

Archana Kesavan, director of product marketing for Grafana Labs, said version 9.0 of the core Grafana platform makes Prometheus monitoring more accessible. Prometheus is being advanced under the auspices of the Cloud Native Computing Foundation (CNCF). It is most widely used in Kubernetes environments, but can also be used with monolithic applications. The latest version of Grafana adds support for an exemplars overlay for traces and Prometheus sparse histograms.

Other capabilities added with this release include a dashboard preview and title search capability, support for envelope encryption by default, an expanded navigation bar and a revamped heatmap panel that renders more quickly than its predecessor.
‘

Finally, the alert system that Grafana Labs first introduced as an option is now a default setting.

Kesavan said in addition to making it easier to launch queries, the Grafana Labs approach also limits the ability of a poorly constructed query to impact clusters running Prometheus.

In general, the adoption of Prometheus has risen alongside Kubernetes. At least initially, DevOps teams that adopt Kubernetes often employ Prometheus to monitor the platform. Less clear is the degree to which IT teams will continue to rely on Prometheus once cloud-native applications are deployed in a production environment. Many IT teams already have a monitoring platform they employ that they choose to extend to Kubernetes environments. Conversely, however, some organizations are opting to replace legacy tools with the Prometheus monitoring platform to monitor both Kubernetes environments and legacy monolithic applications.

At the very least, Prometheus has established a de facto standard for collecting monitoring data from Kubernetes environments that can be queried by a wide variety of tools.

One way or another there will soon be fleets of Kubernetes clusters deployed across the extended enterprise that will need to be monitored. DevOps teams will naturally need to decide for themselves how best to monitor those clusters and, ultimately, observe them in a way that enables queries to be launched and issues surfaced before they become a major problem.

In the meantime, DevOps teams might want to consider consolidating many of the monitoring tools they use today as part of an effort to streamline operations. Rather than having to correlate events across a dozen tools to determine the root cause of an issue, the goal should be to have enough tools to ensure verification without spending hours sorting through conflicting reports.

June 14, 2022
Prometheus Associate Certification will Demonstrate Ability to Monitor Infrastructure

New associate certification exam from CNCF and The Linux Foundation will test foundational knowledge and skills using Prometheus, the open source systems monitoring and alerting toolkit

Valencia, SPAIN, KubeCon + CloudNativeCon Europe – May 18, 2022 – The Cloud Native Computing Foundation® (CNCF®), which builds sustainable ecosystems for cloud native software, and The Linux Foundation, the nonprofit organization enabling mass innovation through open source, today announced the new Prometheus Certified Associate (PCA) exam. The PCA is a pre-professional certification designed for engineers or application developers with special interests in observability and monitoring within the Prometheus ecosystem.

The PCA is modeled after Kubernetes certifications such as KCNA, CKA, or CKAD and is intended for candidates who have passed these certifications or have completed Prometheus-specific training or Cloud Engineer bootcamps. The exam is now in Beta testing and will be generally available later in 2022.

According to CNCF’s 2021 Cloud Native Survey, the use of Prometheus in production has reached 65%, an increase of 43% year-over-year. Production users include organizations like Adidas, Slack, JPMorgan Chase, Verizon, and Uber. Prometheus was the second project to join CNCF in 2016 and the second project to graduate in 2018, following only Kubernetes.

“As cloud native stacks have grown and matured, observability has become crucial for visibility into application metrics, performance, alerting and insights,” said Chris Aniszczyk, CTO of CNCF. “Prometheus is a key element of this, enabling organizations to build a foundation of observability. The PCA exam will allow developers to demonstrate their proficiency in Prometheus ecosystem and cloud native observability concepts.”

PCA will consist of a multiple-choice, online certification exam testing entry-level skills and knowledge around best practices for monitoring cloud native applications and infrastructure and the ability to improve application performance through observability data. Successful candidates will be able to recognize when something is wrong with an implementation, monitor and gain insight into performance, debug issues, and feed data into other systems, including QA, security, and automation. The existing “Monitoring Systems and Services with Prometheus” training course can be helpful for those new to Prometheus.

“There is an ongoing talent shortage in the cloud space, and the only way to address it is to get more individuals trained and certified in these emerging technologies,” said Clyde Seepersad, SVP, and general manager of training & certification at The Linux Foundation. “We have seen tremendous interest in Prometheus training as adoption of this impactful technology has increased, which is a fantastic starting point. This new certification will allow individuals skilled in this technology to verify their abilities to employers, helping advance their careers while beginning to address this talent gap.”

The exam is being developed in partnership with Certiverse. As demonstrated with The Linux Foundation Certified IT Associate (LFCA) and Kubernetes and Cloud Native Associate (KCNA) exams, the company’s online exam development platform brings high-quality certifications to market with the speed required to keep IT content current and relevant.

More information about the exam and topics to be covered in it is available here. Those interested in enrolling in the Beta may do so for free here. Additional details on exam availability will be announced in the coming months.

About Cloud Native Computing Foundation

Cloud native computing empowers organizations to build and run scalable applications with an open source software stack in public, private, and hybrid clouds. The Cloud Native Computing Foundation (CNCF) hosts critical components of the global technology infrastructure, including Kubernetes, Prometheus, and Envoy. CNCF brings together the industry’s top developers, end users, and vendors, and runs the largest open source developer conferences in the world. Supported by more than 500 members, including the world’s largest cloud computing and software companies, as well as over 200 innovative startups, CNCF is part of the nonprofit Linux Foundation. For more information, please visit www.cncf.io.

About the Linux Foundation

Founded in 2000, the Linux Foundation is supported by more than 1,000 members and is the world’s leading home for collaboration on open source software, open standards, open data, and open hardware. Linux Foundation’s projects are critical to the world’s infrastructure including Linux, Kubernetes, Node.js, and more. The Linux Foundation’s methodology focuses on leveraging best practices and addressing the needs of contributors, users and solution providers to create sustainable models for open collaboration. For more information, please visit us at linuxfoundation.org.

The Linux Foundation has registered trademarks and uses trademarks. For a list of trademarks of The Linux Foundation, please see its trademark usage page: www.linuxfoundation.org/trademark-usage. Linux is a registered trademark of Linus Torvalds.

# # #

Media Contact:
Katie Meinders
The Linux Foundation
PR@CNCF.io

May 18, 2022
GitLab Revamps Observability Strategy After Acquiring Opstrace

GitLab plans to make observability a more integrated element of its continuous integration/continuous delivery (CI/CD) platform after acquiring Opstrace.

Opstrace’s open source project exposes a horizontally scalable application programming interface (API) created by Cortex for Prometheus, an open source monitoring platform being advanced under the auspices of the Cloud Native Computing Foundation (CNCF). It then makes use of a Loki API to collect log data.

DevOps teams can also point an existing instance of Prometheus to Opstrace or any instance of Fluentd or Promtail software they are using to collect log data, with support for a wide variety of other APIs, such as the Datadog agent, planned. A command line interface (CLI) is provided to set an instance of Opstrace that creates a controller through which the collecting of log data is managed.

Kenny Johnston, senior director for product management for GitLab, said as part of an ongoing effort to make observability more accessible, GitLab plans to integrate Opstrace by default with GitLab Monitor for organizations that use either the self-managed version of GitLab or the software-as-a-service (SaaS) platform.

Opstrace currently collects logs and metrics and support for distributed traces is planned. The goal is to extend the overall scope of observability capabilities that GitLab already provides within its CI/CD platform, noted Johnston. The goal is to make it simpler for DevOps teams to employ observability capabilities and interrogate data to optimize workflows without acquiring a separate observability platform, he added.

GitLab is betting that, as Prometheus continues to gain traction, more DevOps teams will prefer to extend its reach into the realm of observability rather than having to manage a separate platform.

Although observability has always been a core tenet of DevOps best practices, achieving it has proven to be a challenge. At best, most organizations can continuously monitor IT environments based on predefined metrics that identify when a specific platform or application is performing within expectations. Observability combines metrics, logs and traces—a specialized form of logging—to instrument applications in a way that makes it simpler to troubleshot issues without having to rely solely on a limited set of pre-defined metrics that have been created to monitor a specific process or function.

The rate at which DevOps teams will embrace observability will naturally vary. In 2022, the biggest obstacle might not be the technology as much as it is simply understanding what queries can help DevOps teams better understand the root cause of an IT issue before there is a major disruption. In the long term, it’s expected that machine learning algorithms will leverage the data collected by observability platforms to automatically identify issues that might lead to a disruption long before it actually occurs.

One way or another, the ability to discern events even within the most complex IT environment is steadily improving. Whether that’s accomplished as an extension of monitoring or by embracing true observability may not be all that relevant—as long as the ultimate goal is achieved.

January 10, 2022
Best of 2021 – 11 Open Source DevOps Tools We Love For 2021

As we close out 2021, we at devopsy.kinsta.cloud wanted to highlight the most popular articles of the year. Following is the eighth in our series of the Best of 2021.

DevOps isn’t just a cultural shift — it requires great tools to come to fruition. Below, we’ve pulled together a list of some of the most well-loved DevOps tools available today. But, throwing loads of money into fancy SaaS solutions can quickly gobble up the cloud budget. These DevOps tools all are open source, and enable everything from container builds and orchestration to microservices networking, configuration management, CI/CD automation, full-stack monitoring and more. Here are some of our favorite open source DevOps tools for 2021.

1. Kubernetes

With the ubiquity of microservices and container-based software, it’s no surprise that Kubernetes tops this year’s list of open source DevOps tools. Kubernetes, adoption of which rose by 48% in 2020, is used to orchestrate containers. Instead of releasing microservices manually, Kubernetes can automate deployment, maintenance and scaling of groups of containers in production. Kubernetes, sometimes written as K8s, is hosted by the Cloud-Native Computing Foundation (CNCF).

2. Docker

Docker, the software, is a free and open source platform used to build, ship and run an application as a lightweight container. Containers package up the binaries, libraries, configuration files and dependencies required for a program to run. Over the past decade, containers have played a key role in agile development, and Docker containers led the revolution. At its core is the Docker Engine. Docker Hub is also an excellent resource for finding and sharing prepackaged functions as containers. Also, to plug container vulnerabilities, it may be helpful to use open source container auditing tools like Docker Bench or Anchore.

3. Istio

Microservices are a handy development style, yet they bring new development and architecture concerns. Namely, how do we apply networking policies like security, encryption, observability and telemetry elements consistently across all our services? Well, service mesh is one answer. Service mesh places a sidecar proxy next to each container and abstracts these networking capabilities to a control plane. Istio is one such open source service mesh that has seen widespread adoption. Istio is built on Envoy, opening it up to plugins and extensibility options. We should also mention Linkerd and Kuma as viable open source service mesh alternatives.

4. GitHub Actions

GitHub is arguably the most popular source control and software collaboration platform on the planet. The GitHub platform itself, based on Git, has seen some significant updates in the past few years. Most notable is the GitHub Actions capability. GitHub Actions enable software packages hosted on GitHub to accept inputs and trigger other processes. This could help automate some cool DevOps workflows within GitHub, such as code reviews, branch management or CI/CD processes — the combinations of possibilities here are endless. GitHub Actions are, essentially, YAML files hosted in GitHub repositories that leverage GitHub webhooks. Though this is more of a feature than an open-source tool, we feel it’s important to include here. Actions is free for public repositories with a limit of 100 actions.

5. Jenkins

A big part of the DevOps philosophy is finding ways to automate and deploy new iterations more efficiently. Part of this goal is creating a streamlined continuous integration and continuous delivery (CI/CD) pipeline. Jenkins is an open source automation server with hundreds of plugins to automate the building, deployment and testing of software projects. Although GitHub Actions could theoretically replace a CI server in the future, CI tools like Jenkins, CircleCI, TravisCI and GitLab Community Edition still are preferred by many DevOps teams.

6. Prometheus

Metrics and alerting systems are crucial for site reliability engineers to visualize applications and react to issues. Prometheus, a graduated CNCF project, is a well-loved open source monitoring solution. A Prometheus server collects time-series metrics by scraping HTTP endpoints and generates a system to interact with this data, offering deep querying, visualization, storage and other capabilities. Check out this Awesome Prometheus list for Prometheus introductions and additional resources.

7. Ansible

Ansible is all about automation. Ansible, an open source project sponsored by Red Hat, can be used to automate things like cloud provisioning, networking, deployment, configuration management and other tasks. Ansible has a simple yet effective architecture that is relatively easy to assemble — all you need is a text editor and command line. You describe your infrastructure in a text document and organize your desired states in a playbook. For an example in practice, see how OpenIO uses Ansible. “Ansible is our standard tool not only to deploy the OpenIO core, but also our WebUI, OIO-FS and all upcoming options,” writes Cédric Delgehier, Ops at OpenIO.

8. Chef

Chef is another infrastructure-as-code (IaC) solution for automating your configuration management. Chef uses Ruby to automate server configurations and works well with all major cloud service providers (CSPs). This can be very useful when creating and provisioning large quantities of machines. Like other automation tools on this list, the user describes their components and their states in a declarative format. In Chef, these are known as “recipes,” which can be grouped together into “cookbooks.” You can’t diss Chef for not being on-theme!

9. Terraform

Terraform is another IaC tool that can be used for initiation of building, versioning and further automation using configuration files. “Terraform is a tool for building, changing and versioning infrastructure safely and efficiently,” as it is described on GitHub. Terraform follows an “execution plan” that a user creates with high-level syntax. One unique aspect of Terraform is its emphasis on versioning — this allows you to version the blueprint of your service just as you would your software.

10. JAMStack

As I’ve covered before, JAMStack combines JavaScript, APIs and markdown for constructing web-based applications. While more of a “headless development” methodology than a single open source tool, JAMStack projects are often built using open source components. For example, JAMStack often leverages open source headless content management systems such as Ghost, Strapi and/or Netlify CMS.

11. ELK Stack

ELK Stack is the union of three open source projects maintained by Elastic: Elasticsearch, Logstash and Kibana. With these three components, developers can take in and log data from any source and create helpful visualizations. This centralized logging is enabled with a NoSQL database for storage with Elasticsearch, processing and data collection with Logstash and visualization with Kibana. Increased visibility is vital for data analysis and helps identify errors to reduce mean time to recovery (MTTR).

Rounding Out The DevOps Toolchain for 2021

Programming isn’t only about shipping quality code, it’s about efficient execution. This drive to improve operations across the board is truly putting Ops into everything. With awesome open source DevOps tooling, it means more and more architects can adopt a DevOps approach within their deployment models.

DevOps as a practice, as well as its underlying technology, is always evolving. In 2021, a lot of effort is being placed on wrangling the effects of introducing a microservices development style. We’ve noticed the maturation of container orchestrators and service mesh, as a result. Blueprinting infrastructure-as-code and creating automated, repeatable configurations is another must-have for implementing automated build and release pipelines. Not to mention, many no-code and low-code platforms are opening up DevOps capabilities to non-programmers, albeit from a proprietary perspective.

It’s also helpful to note that some popular open source DevOps tools have been acquired — like Docker and Chef — blurring the lines between their business and their open source roots. When it comes to open source tools, it’s a good practice to adopt vendor-neutral tools with solid community support representing diverse stakeholders. This will help future-proof your project. Even though open source is “free,” users must ensure the benefits outweigh the time-to-onboard as well as the operational overhead of self-hosting.

DevOps tools are beneficial to automate software deployment. Here, we’ve attempted to list some of the most exciting tools to help automate that process. Of course, many new tools are emerging in the DevOps space, and we have hardly scratched the surface. Do you have a favorite DevOps tool that’s not mentioned above? Let us know in the comments below!

December 23, 2021