devopsy.kinsta.cloud

Author: Tomer Levy

  • There’s No Value in Observability Bloat. Let’s Focus on the Essentials

    There’s No Value in Observability Bloat. Let’s Focus on the Essentials

    The observability market paired with cloud security has been predicted to reach $70B by 2026, but that market prediction is a myth, and I’ll tell you why.

    If observability was a grand buffet and every company needed a feast of features and capabilities of this proportion, maybe this prediction would be valid. But most companies don’t need an over-the-top feast. Legacy APM and observability solutions are serving up an all-you-can-eat buffet when the reality is that most companies only need the essential ingredients to satisfy their observability strategy. And the cost for that can be much more modest. The $70B observability market prediction is, therefore, suspect, considering the true needs of most organizations.

    Complex cloud ecosystems, the rise of applications built on microservices and Kubernetes, and the overall scale and complexity of applications have led to exponentially more data being created in the pursuit of observability. Storing and processing all that data is expensive, and as the volume of data balloons, so does the cost.

    And while the complexity of business-critical systems and the volume of data have scaled, so have the overall cost of those legacy tools—in direct proportion. Partially, this is driven by the features and instrumentation added that only contribute to the bloat and valueless telemetry.

    Customers aren’t seeing value in any of this bloat, and for the most part, all that’s been accomplished is increased complexity in deploying and operating the system, with a commensurate lengthening of time to value. Plus, pricing models haven’t significantly changed despite the advancements in the market. The fact is, organizations can’t really say that they have gotten more insight into the overall performance of their mission-critical systems. What they are saying, however, is that these tools are becoming overwhelming to manage and overwhelmingly expensive, and developer productivity is not increasing.

    What organizations need and want are real-time insights into the health and performance of their cloud infrastructure and applications–specifically the confluence of logs, metrics and traces within the context of modern observability. There is little tolerance for software bugs, slow web experiences, crashed apps and other service interruptions, and the responsibility lies on developers, engineers and ITOps teams to quickly resolve production issues before they impact customer experience.

    There is a call for observability, but rather than an explosion of features and noise, what’s being called for is a narrowing of focus.

    Observability Has an Efficiency Problem

    Telemetry data gathered from the distributed components of modern cloud architectures needs to be centralized and correlated for engineers to gain a complete picture of their environments.

    Engineers need a solution with critical capabilities such as dashboarding, querying and alerting, and AI-based analysis and response, and they need the operation and management of the solution to be streamlined.

    What’s important for them to know is that it’s not necessary to spend more to ensure peak performance and visibility as their environmental complexity grows.

    Legacy players experiencing some maturity in the market may have conditioned them to believe this is true—they have set a standard for higher costs correlating with incremental amounts of data, no matter the quality of that data. Companies are being fed the lie that all this data may be relevant and they’ll be doing themselves a disservice by neglecting even one byte of it.

    Data Does Not Inherently Equal Insights

    No doubt, more data is being generated, but most of it is not relevant or valuable to an organization. Observability can be optimized to bring greater value to customers, and that’s where the market is headed.

    Call it “essential observability.” It’s a disruptive vision to propose a re-architected approach to observability, but what engineers need is a new approach making it easier to surface insights from their telemetry data while deprioritizing low-value data. Costs can be reduced by consuming only the data that enables teams to maintain performance and drive smart business decisions.

    Essential observability is cutting out the fluff and only focusing on—and paying for—the impactful data. It’s about arming engineers with meaningful insights so they can quickly resolve production issues and business disruptions. Organizational performance increases without inflating data complexity and resource consumption.

    The Essential Observability Revolution

    Observability tools have evolved beyond the model of linear price increases as complexity and data scales. For companies (especially growth-stage organizations) that want a partner to help them make more informed decisions and keep their costs more reasonable (i.e., stop recklessly spending for useless data), they need to focus on tools offering them the essentials.

    Solutions are available, offering targeted features and automated capabilities out of the box, delivering the insights needed to understand and fine-tune data. These solutions provide continuous support, reduce complexity and accelerate time to value.

    The next revolution in observability isn’t features and functionality but providing insights to make data-driven decisions that keep essential observability cost-effective while not compromising the value derived.

  • Where Does Observability Stand Today, and Where is it Going Next?

    Where Does Observability Stand Today, and Where is it Going Next?

    As we move into the second half of 2023, it’s only natural to engage in the annual ‘state of the industry’ reviews and take a general inventory of the past year while looking out into the immediate future. Observability has gained traction among DevOps and platform engineering teams over the past year and is poised to continue that velocity. But my observation is that the current observability model is broken.

    Too Much Data

    Long story short: Many organizations (and engineers specifically) agree that they are sending too much data to their monitoring tools, resulting in an inability to garner valuable insights because the systems are massively overloaded. Sending this high volume of data also costs too much and requires an inordinate amount of time for analysis, which many organizations don’t have. Put simply–more data doesn’t mean more clarity; in fact, we see the opposite.

    Massive adoption of cloud-native technologies such as Kubernetes that heighten the challenge of efficiently monitoring environments is also accelerating this problem. Beyond increasing costs and complexity, many teams seem to feel that their current approach doesn’t create the desired level of observability for Kubernetes, full stop.

    Another major issue we see on the observability landscape relates to the overall maturity of monitoring processes and teams. In many instances, even when organizations have effective tools, they struggle to interpret the data produced into targeted conclusions that their teams can translate into effective actions. And this is further amplified by the outstanding engineering talent shortage because good people are harder to find and retain than ever before.

    We also see a lot of pervasive challenges related to organizations’ desire to use popular open source monitoring tools-especially if they are attempting to do so in an integrated manner or to support rapidly growing deployments at scale.

    Overwhelmed by Observability Data?

    In summary, if you’re overwhelmed by observability data, if complexity and costs grow out of control and you’re also struggling to make sense of the available information and improve your cloud applications, we think that means that the current observability model really is broken.

    To that end, gaining a deeper perspective into end user opinions on these topics, among others, is the precise goal of the annual DevOps Pulse Report, currently in its sixth year and now in the survey stage. We predict that this year’s research project will underline the aforementioned challenges and help frame how teams might seek to address them.

    What types of findings do we hope to unearth? Last year, respondents revealed that despite a growing perception of DevOps maturity, it was actually taking them more time to respond to emerging problems. Roughly 64% of respondents reported that MTTR during production incidents was over an hour, compared to only 47% the previous year.

    Last year’s report also highlighted the reality that monitoring complex environments remains difficult with over 52% of respondents citing Kubernetes, microservices and serverless as their primary challenges in gaining improved observability.

    Both of these previous findings also seemed to underline the growing issues of data overload and spiraling complexity. We’ve also seen previous evidence of a lack of maturity around distributed tracing—a critical observability practice aimed directly at increasing visibility into complex architectures—despite the fact that 75% of respondents listed it as a priority in last year’s research.

    Is your team struggling to make sense of its available monitoring data, or do so in a way that aligns with your current processes or budgets? Hopefully, the 2023 DevOps Pulse Report, set to publish early this year, will put some numbers behind these assertions and help us all understand how to move forward.

    If you want to participate, you can do so here.

  • DevOps in 2016: The Year of Implementation

    DevOps in 2016: The Year of Implementation

    DevOps is the default software delivery approach for modern IT organizations and hyperscale startups that have incorporated the culture into their everyday integration and delivery activities.

    However, over the past year, these organizations have also started to recognize and acknowledge that there are challenges in certain aspects of the DevOps approach, including precarious security infrastructures, R&D team member burnout, and the need to close gaps between operations and development with dedicated personnel such as site reliability engineers (SREs).

    In 2015, we saw larger enterprises beginning to migrate to Amazon Web Services (AWS) and sketch out their plans to incorporate agile and DevOps methodologies. The implementation process necessitates changing cultural norms, and incorporating new tools and organizational processes. And with DevOps’ great benefits, my prediction is that in 2016, these enterprises will move from the drawing board to implementation.

    1. Companies will stop talking and start implementing. In 2016, DevOps culture will finally become mainstream (and not only for startups and tech-savvy companies) as more traditional enterprises jump on the bandwagon and start implementing real DevOps principles. IT leaders will need to figure out how to restructure their teams, adapt DevOps values, concepts and tools, and incorporate continuous integration and continuous delivery practices to do a better job of meeting user expectations of fast and agile delivery.

    Having said that, traditional IT organizations that deal with sensitive data and tight change management processes — such as those in the health, finance, and some government sectors — cannot compromise their strict IT delivery standards, especially if they have to comply with security and compliance protocols. Therefore, 2016 will still be just the beginning for enterprise-grade DevOps implementation in several specific industries and verticals

    2. SREs will bridge the gap. The growing movement of site reliability engineers (SREs) is being led by Google and has been introduced to the world as part of a practical and more mature DevOps adoption. SREs are software engineers who work in operations and are in charge of availability, change management, emergency response, and monitoring (among other things).

    Even early DevOps adopters such as LinkedIn have begun looking for dedicated individuals to fill SRE positions, which require ops as well as code-level management skills. In 2016, we will see an increasing number of discussions on SREs as well as more job opportunities for them, making this new class of engineers key players in DevOps implementation.

    3. High demand will mean high salaries. What we have seen within our company as well as among our customers is that dev and ops engineers with strong experience in working with and implementing DevOps are still a rare breed. As more and more IT leaders recognize the need to move to DevOps, the higher the demand and compensation will grow in 2016 for the limited number of experienced engineers.

    According to Puppet, DevOps engineers make noticeably higher salaries than most other IT positions and 55% of U.S. DevOps engineers earn $100,000 per year or more. While large IT organizations are building their own education and training plans in anticipation of the move to DevOps, the demand for experienced engineers will still continue to grow in 2016 and beyond.

    4. We will have more Big Data benchmarks. Over the past two to three years, the IT industry has embraced Big Data. Technologies such as Hadoop and MongoDB are commonplace now when it comes to extracting values from large data repositories. Today, companies in industries ranging from Adtech to online gaming to finance are already utilizing Big Data to enhance their core capabilities.

    Over the past year, we saw the rise of more solutions that help with implementation and presentation that aid in generating insights and enhancing user experience. Beginning next year, these big data solution vendors will leverage their unique positions and advanced technologies (such as machine learning) as well as their vast amounts of data to create specific industry benchmarks and best practices that will ultimately enhance customer satisfaction.

    5. Some companies will reverse-migrate to on-premise. Don’t get me wrong, we’ll still see AWS, Azure, Google Cloud Platform (GCP), and others continue to grow their user bases among next-gen companies as well as traditional organizations such as, the Guardian, which recently failed to build an OpenStack private cloud and consequently decided to go all-in with AWS.

    However, companies that have been using the public cloud for the past year or two will see their cloud bills start to soar (if they haven’t already) and look around for cloud solutions that better meet their resource-utilization levels and business model. I think we’ll see more of this so-called “reverse cloud migration” as the cloud business matures in 2016. This is actually a sign that this market is maturing and that people are past the super-hype phase and seeing the pros and cons of the public cloud.

    In Summary

    Of course, no one can predict the future, but the events of 2015 may hint at what we will see in 2016 — and that points toward an increasing implementation of DevOps practices.

    Some companies will manage to complete their DevOps cycles and have full-blown, continuous deployment straight to production. However, the majority will be challenged by even the basic implementation processes and needs such as which DevOps tools to use, how to reconstruct their organizations, and what state of mind to have.

    Still, DevOps will eventually be the norm when it comes to software development and delivery — it is just a matter of time.

  • 9 Open Source DevOps Tools We Love

    9 Open Source DevOps Tools We Love

    While we have been building our SaaS company, we have been investing much time in researching and deciding which tools to include in our DevOps toolkit. We’ve based these decisions on our years of experience in the IT industry, dealing with infrastructure for the most part. From building a petabyte-scale, data analytics infrastructure, our architecture, tools, and processes have become key components of our technology and operations. We’ve taken great care in selecting, benchmarking and constantly improving our tool selection.

    As a company that is building a solution on top of an open-source stack (ELK), our team is heavily involved in the open source community, contributing to multiple projects such as Camel and Kafka while customizing tools to fit our needs. The vast majority of tools we use internally are open-source ones. By sharing the toolset that we’ve collected and honed over time, we hope to foster a discussion within the DevOps community on what further improvements can be made.

    With that, we welcome you to enjoy browsing through the following list that we’ve created. Some tools you may have known about for years, while others may be new. Of course, we invite any and all feedback, especially from those of you who are using alternatives!

    Must-Have DevOps Tools

    1. Nagios (& Icinga) 

    nagiosInfrastructure monitoring is a field that has so many solutions… from Zabbix to Nagios to dozens of other open-source tools. Despite the fact that there are now much newer kids on the block, Nagios is a veteran monitoring solution that is highly effective because of the large community of contributors who create plugins for the tool. Nagios does not include all the abilities that we had wanted around the automatic discovery of new instances and services, so we had to work around these issues with the community’s plugins. Fortunately, it wasn’t too hard, and Nagios works great.

    We also looked into Icinga, which was originally created as a fork of Nagios. Its creators aim to take Nagois to the next level with new features and a modern user experience. There is a debate within the open source community about the merits of Nagios and its stepchild, but for now we are continuing to use Nagios and are satisfied with its scale and performance. The switch to newer technology, such as Icinga, may be appropriate in the future as we progress.

    2. Monit

    monitSometimes the simplest tools are the most useful, as proven by the simple watchdog Monit. Its role is to ensure that any given process on a machine is up and running appropriately. For example, a failure occurs in Apache, Monit will help to restart the Apache process. It is very easy to setup and configure and is especially useful for multi-service architecture with hundreds of micro-services. If you are using Monit, make sure to monitor the restarts that it executes in order to surface problems and implement solutions (rather than just restarting and ignoring the failure). You can do this by monitoring Monit’s log files and ensuring that you are alerted to every restart.

    3. ELK – Elasticsearch, Logstash, Kibana – via Logz.io

    logz.ioThe ELK Stack is the most common log analytics solution in the modern IT world. It collects logs from all services, applications, networks, tools, servers, and more in an environment into a single, centralized location for processing and analysis. We use it for analytical purposes (e.g., to troubleshoot problems, monitor services, and reduce the time it takes to solve operational issues). Another use for this tool is for security and auditing (e.g., to monitor changes in security groups and changes in permissions). After receiving alerts on these issues, it is easy to act on unauthorized users and activities. We also use ELK for business intelligence, such as monitoring our users and their behavior. You can set up your own ELK or buy it as-a-service. We’ve written a guide for the community on using ELK to monitor your application performance.

    Disclaimer: Logz.io is our ELK-as-a-service that we use in our own environment. You can say that we eat our own dog food.

    4. Consul.io

    consulConsul is a great fit for service discovery and configuration in modern, elastic applications that are built from microservices. The open-source tool makes use of the latest technology in providing internal DNS names for services. It acts as a kind of broker to help you sign and register names, enabling you to access service names instead of specific machines. If, for example, you have a cluster of multiple machines, you can simply register them as a single entity under Consul and access the cluster easily. We praise this tool for its efficiency, although we still feel there is more that can be done with it. If you also use it, it would be great to hear about your own use case.

    5. Jenkins

    jenkinsEveryone knows Jenkins, right? It’s not the fastest or the fanciest, but it’s really easy to start to use and it has a great ecosystem of plugins and add-ons. It is also optimized for easy customization. We have configured Jenkins to build code, create Docker containers (see the next item), run tons of tests, and push to staging/production. It’s a great tool, but there are some issues regarding scaling and performance (which isn’t so unusual). We’ve explored other cool solutions such as Travis and CircleCI, which are both hosted solutions that don’t require any maintenance on our side. For now, however, since we’ve already invested in Jenkins, we’ll continue with it.

    6. Docker

    dockerEverything that can be said about how Docker is transforming IT environments has already been said. It’s great…life changing, even — (although we’re still experiencing some challenges with it). We use Docker in production for most services. It eases configuration management, control issues, and scaling by allowing containers to be moved from one place to another.

    We have developed our SaaS solution with a twelve-layer pipeline of data processing. Together with Jenkins and Docker, we have been able to run a full pipeline across all layers on a single Mac. It would be wrong to say that there aren’t any complications with Docker, as even small containers can take a significant amount of time to build. However, we want to ensure that our developers are as satisfied as possible and enable them to work rapidly. With all of the management involved in storage, security, networking — and everything surrounding containers — this can be a challenge.

    We see Docker progressing and look forward to welcoming the company’s new management and orchestration solutions. For those who might be having issues with Docker, we’ve also compiled a list of challenges and solutions when migrating to Docker.

    7. Ansible

    ansibleAgain, simplicity is key. Ansible is a configuration management tool that is similar to Puppet and Chef.  Personally, we found those two to have more overhead and complexity to our use case– so we decided to go with Ansible instead. We know that Puppet and Chef probably have a richer feature set, but simplicity was our desired KPI here. We see some tradeoffs between configuration management using Ansible and the option to simply kill and spin new application instances using a Docker container. With Docker, we almost never upgrade machines but opt to spin new machines instead, which reduces the need to upgrade our EC2 cloud instances. Ansible is used mostly for deployment configuration mostly. We use it to push changes and re-configure newly-deployed machines. In addition, its ecosystem is great, with an easy option to write custom applications.

    8. Collectd/Collectl

    Collectd/l are nifty little tools that gather and store statistics about the system on which they run and are much more flexible than other tools. They allow users to measure the values of multiple system metrics and unlike other log collection tools that are designed to measure specific system parameters, Collectd/l can monitor different parameters in parallel. We use these two tools to measure customer performance parameters and ship them to our ELK-as-a-Service platform. We’ve specifically wrapped a Collectl agent in a Docker container and push it with Ansible to all of our servers. It collects information every couple of seconds and then ships it to ELK to allow us to run reports and send alerts. If you’d like to see a specific example of how we do this process in our environment and how others can do the same, we’ve created a guide for everyone.

    9. Git (GitHub)

    githubGit was created 10 years ago following the Linux community’s need for SCM (Source Control Management) software that could support distributed systems. Git is probably the most common source management tool available today. After running Git internally for a short period of time, we realized that we were better suited with GitHub. In addition to its great forking and pull request features, GitHub also has plugins that can connect with Jenkins to facilitate integration and deployment. I assume that mentioning Git to modern IT teams is not breaking news, but I decided to add to it to the list due to its great value to us.

    Other picks?

    The modern DevOps world is full of outstanding and unique open source tools—it’s a jungle out there. We found the tools listed here to be the best in breed and think they should be included in every DevOps engineer’s shortlist.

    Or am I wrong? What open source DevOps tools are in your toolkit? I’d love to hear your own recommendations and experiences in the comments below.