devopsy.kinsta.cloud

Tag: logging

  • Logging: Turning Good DevOps Teams into Great Ones

    Logging: Turning Good DevOps Teams into Great Ones

    The competition is on for DevOps teams that are expected to rev up production on a multitude of projects, with teams and companies aiming to stay atop the global software product marketplace. The complexity of code creation and the pace of deployment has picked up in recent years, and DevOps managers are starting to face serious challenges of scale and efficiency.

    Constant code generation is the first obstacle. Imagine a team producing millions of lines of code, working for weeks and hoping for a smooth release. But testing time arrives, and there is a total data meltdown.

    Wasting valuable programming hours is not an option, but what if managers had the tool to turn back the time to a version before the mistake happened? Managers or developers may view logging as an incredibly boring activity, but it is vital. It serves like a black box of what went wrong—but also a way to roll back the bad code, and return to a more manageable situation.

    Logging and tracking habits ensure that a DevOps team will not see their code fail in production, and cause expensive breakdowns in service. Logging becomes more than a wishful addition, but a pivotal tool that may save a project.

    Data is proliferating, code is constantly being written to offer innovative products or support existing ones. There is a massive amount of data projected to be generated in 2025, only a few short years away, if the global IT production sector keeps increasing its activity. This amount of data to be kept and logged would be roughly 10 times higher compared to the data generated in 2016. This issue has plagued Silicon Valley, but also worldwide tech centers: how to keep track of what is going on. How do companies such as Instacart, Lime and OpenAI track their changes and ensure viability and awareness of errors?

    There is the new role for DevOps managers: ensure fast, modern and stringent log management. Dedication to logging means generating tight order out of the chaos of code creation. The right tool will be economical, intuitive and fast, ensuring the possibility for control in what could become a runaway data meltdown.

    Logging is one of the pillars in the modern DevOps department, and while now it is important, it will become indispensable, along with metrics and analytics.

    DevOps are already aware of the possibilities of logging, to keep track of multiple avenues in product creation, from databases to server side events, and even the effects and behavior of users.

    Stringent log management means making use of a valuable resource of knowledge and decision making, but also operating with a complex system of data. DevOps managers will need the tools to span the demands of stringent log management, for a multi-step process involving log generation, transmission, analytics, storage, archives and disposal.

    The right logging system can streamline those steps. Inexpensive logging tools and systems can unify log management across teams and simplify the process of logging. More services are appearing each day, either cloud-based or self-hosted, and with varying levels of security. With increasing complexity and the drive to create ahead of the competition, it would be the task of DevOps managers to make the team aware of the tools, and to provide tools intuitive and fast enough for the good habit of logging and log management.

    — Chris Nguyen

  • Log Management: What DevOps Teams Need to Know

    Log Management: What DevOps Teams Need to Know

    One of the most important shifts of this decade is the rise of interconnected data across distributed systems. This explosive growth not only sparked the need to rethink cloud and global IT strategies, but it also disrupted traditional development, DevOps and ITOps practices. These changes are pushing requirements for higher-velocity development of features and products—together with more responsive support requests—onto technology professionals. These new requirements have prompted teams to evaluate new tools and agile processes that help support the needs of a competitive modern business.

    Regardless of the pace of change within your group, basic functions such as infrastructure monitoring are crucial to operating production applications. At this new speed of business, concepts such as observability are important for enabling successful deployments with fewer user interruptions—not only in production, but also in the early stages of your continuous integration/continuous delivery (CI/CD) pipeline. As part of “shifting left,” log and event management in your CI/CD pipeline empowers developers to monitor and observe application behavior before releasing it to production. While some might say this adds additional layers of process, time, effort and due diligence in dev environments, it can help you evolve your organization’s fundamental software development behaviors while reducing preventable issues from post-deployment into production. It can create a smoother, more seamless user experience and reduce the need to fight fires or rearchitect your solution after the application is live in production and at scale.

    Setting the Testing and Dev Stage

    The rapidly evolving technology landscape has increased the need for log management and observability across distributed systems and containers. Changes to how applications and services are created, the ability to deploy applications across combinations of global logging-as-a-service (LaaS) providers and the pervasiveness and volatile lifespan of containers, along with the capacity to build services using various development languages, has increased the need to collect, monitor and trace data points across connected systems, which provide critical end user-facing value.

    Traditionally, many considered log management a tedious task because each instance of a new server required you to run a search command across the logs local to that server. That approach isn’t scalable when every cluster of more than 20 servers is built up in minutes, and each application has independent logging formats. In this scenario, you’d need to run 20 individual searches across multiple systems, compare timestamps across servers residing in different time zones and so forth. With the decreasing cost and increasing prevalence of containers and virtual systems, the number of systems running within a typical organization has grown exponentially as the business prospers.

    In addition to increased usage of virtual systems and containers, independent scrum teams of developers can build in parallel using technologies which are the best fit for the services they are producing, while also sharing environments for builds, staging and production. With these distributed services, aggregated log systems can quickly determine the root cause for debugging their application’s errors, exceptions or performance bottlenecks. A log aggregation system can reduce the element of “personalization” that can often happen when developers are using different technologies with various logging formats, and also reduces the noise by filtering out information that delays the team’s ability to find the source issue. This can cause confusion or slowed productivity when one developer is left to understand another’s non-standardized, yet shared, workload.

    The reduced cost, increased collaboration and prevalence of container services are collectively driving an uptick in virtual environments and containers, often requiring more logs to be created in testing and dev environments, which can force organizational shifts to enable full observability across their infrastructure.

    The 3 Pillars of Observability

    Businesses are in the midst of a shift toward adopting these principles—the three pillars of observability—and using a three-pronged method of monitoring that takes data from several vantage points to depict a more accurate and consumable view of overall health and stability:

    1. External Monitoring: An example of this is health checks run against your internal and external applications and websites to see the “digital experience” of your users.
    2. Metrics and Distributed Tracing: This enables you to trace communications between applications distributed across systems/containers to identify errors and exceptions from your applications and resolve latency quickly.
    3. Events and Logs: Data, which helps provide contextual information about events, enables you to identify issues in the code when combined with information from the first two pillars.

    The above types of information are critical to DevOps to provide elastic and resilient services. Bringing application, system and infrastructure events into logs together with external monitoring and metric collection is quickly becoming the standard. With powerful log-parsing features such as live-tail search accessible from a web browser and the ability to visually pair log insights with added health metrics, the real value of log data is made apparent with little effort.

    Site reliability engineering (SRE) is another area where significant evolution has occurred. ITOps has always been asked to react to fire and run from fire to fire. The evolution from ITOps to DevOps to SRE is forcing the alignment of different types of data to be used very early in a CI/CD pipeline, and enabling SRE teams to get involved in architectural discussion at project inception instead of in the latter stages or at release. The SRE is more involved in monitoring, understanding the development and architectural principles used to develop application services and build in automation, and high availability and circuit breaker patterns from inception.

    This approach has allowed teams to be proactive and responsive rather than reactive. It has equipped our teams (which keep our business running successfully and our customers loving what we do) with the ability to focus on being scalable, elastic and resilient. Instead of troubleshooting systems in production, we auto-scale and replace to minimize disruptions and investigate that incident offline when possible.

    Best Practices

    Given the increasing pace of application development required to support the needs of the business, the spread of containerized applications and virtual environments and the emergence of new disciplines for observability (due to the lowered costs and increased reliability), log and event management has become a critical aspect for all involved in building, supporting and even using mission-critical applications. It’s moving to a defined area where logs should be compartmentalized, and access control should be defined:

    1. Compartmentalizing Logs: Critical as a best practice so we can differentiate development, staging and production environments and segregate logs based on practical grouping. Using a log aggregation service provides you with a consistent experience and a common set of capabilities across all your environments.
    2. Access Control: Another best practice that should be implemented more broadly; for example, developers don’t need access to all of your production logs if approximately 95 percent of their work happens prior to reaching production environment. In other words, even though tech pros are working to “de-silo” log management, we still want to silo access control and manage visibility. This can also help to ensure personally identifiable data (PII) isn’t exposed to anyone it shouldn’t be.

    Looking Toward the Future of Log Management

    As the landscape evolves and best practices become more defined, we should expect revitalized strategies with elevated expectations for the traditional scenarios where basic/free monitoring is leveraged. While the pillars of observability are leveraged more broadly in the ITOps and DevOps communities, we’ll be able to move more efficiently from an end user-facing incident—for example, an application error traced across distributed systems—straight into the logs to find the specific error causing the issue. We’ll be able to see greater correlation between logs and other custom metrics, as well as much broader use of these techniques outside of production environments.

    DevOps teams need an intuitive event and log management system that is as de-siloed as we are. No longer can log management systems be relegated just to one on-premises storage system or one virtual environment; they need to fit the reality of how work is increasingly done today and will be in the future.

    — Keith Kuchler

  • We Still Haven’t Solved the Logging Problem

    We Still Haven’t Solved the Logging Problem

    At a company I previously worked at, I helped build out the initial infrastructure to support a large cloud security product. It was one of the first times I had the opportunity to build something from nearly the very beginning. Like many operations professionals, so much of my career prior to this experience had been spent cleaning up other people’s messes and technical debt that I was finally excited to “do things right” from the onset.

    Looking back, I wish I knew that building our monitoring platform would be hampered without the ability to retain and analyze both warm and cold log data. Here’s why.

    Time Series to the DevOps Rescue 

    Hindsight is almost always 20/20, and today I can proudly say that my team and I built a killer app monitoring platform. My goal was to make it easy for Dev and Ops folks to report time series metrics for their applications and build dashboards and alerts for app health. And we did just that.

    While many ops colleagues would say that monitoring journeys usually start by capturing application logs with open source tools such as Logstash/Elasticsearch, Graylog2, or even buying commercial services from Splunk or Sumo Logic, not all journeys start that way. My team and I took a different approach and optimized for time series metrics versus the more traditional structured log management approach, leveraging newer toolsets such as Librato for real-time operations monitoring and performance analytics. Librato enabled us to scale much faster than with traditional log solutions and gave us almost instant visibility across our containers, cloud servers and the apps themselves.

    Logging is Too Expensive 

    Using hosted services such as Librato to manage our time series metrics worked fine for a while. However, the cost of SaaS monitoring eventually grew, and, since we had the time and energy to bring metrics in house, we deployed Graphite, the open source enterprise-grade monitoring tool. Graphite allowed us to continue enabling our engineers with affordable access to granular time series metrics to help them manage their applications.

    While this allowed the business to respond quickly when an issue arose, time series metrics were only part of the solution. They allowed us to see what was happening, but not always why it happened. While time series metrics and even some of the more advanced tracing technologies including Jaeger and Opentracing are fantastic to help debug complex distributed systems, they fall short when investigating and analyzing causes for problems, issues and hacks.

    Digging Into Your Log History Gets Expensive, Quick

    Tools such as the ELK stack have been booming, with Elastic alone hitting more than 100 million downloads last year, which shows that DevOps teams want more from traditional log management solutions. They’re looking for simplicity, ease of use and affordability, among others. With JSON effectively the de facto standard for logging, most teams take log data with its predefined schema and then index it into Elasticsearch/Lucene. Unfortunately, the Lucene indexing mechanism results in an unintended consequence: a 5 to 10X increase in disk usage. And just like that, ops cloud budgets get blown away when they realize that 10GB of JSON data indexed into Elasticsearch could be 50GB of disk used. You wanted that index to be replicated for disaster recovery as well? Then add 100GB of storage for every 10GB of logs.

    DevOps Normal: Still the Tough Choice Between Retention and Cost

    Unfortunately, most people choose to simply purge data from our “hot” Elasticsearch clusters as soon as we possibly can to minimize cost issues. This is far from ideal.

    Companies moving to microservices-based architectures, deployed on containers such as Kubernetes, are going to be in for a rude awakening when they start building out their centralized logging platforms to keep up with the volume. Security teams are going to feel the burden as well, as they come under increasing pressure to keep data around for long periods of time due to compliance and audit reasons. When the next big web-facing vulnerability comes out, the ability to go back weeks or months will be key to see what IP address accessed a potentially malicious endpoint. With that data available, we could then take all those IPs and correlate them with other access logs.

    Is centralized logging any better than it was 10 years ago? Or has all this data generated exploded so much that the tools we have today are just barely keeping up? One thing is certain, this problem is far from solved, and many companies may find themselves drowning in their own data lakes.

    — Pete Cheslock

  • TMI: BigPanda Separates the Signal from the Noise

    TMI: BigPanda Separates the Signal from the Noise

    Organizations have struggled for years with the problem of too much information. IT professionals can only address so many issues at one time, but many are faced with an overwhelming volume of alerts to try and review and prioritize. BigPanda thinks it can solve that problem with its event correlation platform.

    If everything is an alert, then nothing is an alert. There has to be some way to separate issues and alerts that really matter so IT teams can prioritize efforts accordingly and address the concerns with the greatest business or security impact. At the same time, though, it’s crucial to minimize or eliminate both false positives and false negatives.

    I spoke with Assaf Resnick, CEO of BigPanda, about the company’s approach to event correlation and why he believes organizations need what his company has to offer. Resnick described the current state and trends in IT infrastructure and IT development, noting DevOps principles and tools have transformed the entire data landscape.

    Customers, he said, are asking themselves, “How do I move away from this monolithic application hierarchy that deploys once a quarter?” The goal, according to Resnick, is not just more reliable uptime, but delivering speed and agility at the same time.

    The BigPanda platform doesn’t try to reinvent the wheel, per se. In fact, it more or less relies on users having other wheels in place. BigPanda sits on top of what a company already has and pulls in data from other platforms such as New Relic or Splunk. It then normalizes and correlates that data so it can spit out results on a scale that mere mortals can comprehend. “Instead of 20,000 alerts from 15 different monitoring tools, I may only have 36 meaningful events,” Resnick said.

    Playtika is one customer taking advantage of the BigPanda solution. Playtika is a digital gaming company that develops social casino games. According to a recent case study, “With BigPanda, Playtika was able to centralize all of its fragmented monitoring tools into a single unified platform that provides instant insight into the health of their IT systems. Moreover, by automatically correlating alerts from multiple systems into high-level incidents, BigPanda enabled Playtika’s IT pros to cut through the noise and quickly detect problematic issues. And as Playtika has acquired companies in recent years, BigPanda has allowed them to consolidate the operations management of these newly acquired companies virtually overnight, with zero migration cost.”

    “We were struggling to work with many different tools in order to monitor the performance of our operations,” said Ella Vidra, director of IT Operations for Playtika. “Investigating alerts across our stack was tedious and time-consuming. We were at risk of missing critical errors that could impact production. BigPanda enables us to centralize our IT monitoring and quickly spot the issues that matter most.”

    The proof is in the proverbial pudding, of course. There have been plenty of solutions that promise to streamline or simplify logging and alerting. It seems as though we now have solutions that are designed to streamline and simplify the results from the solutions that streamline and simplify other logs and alerts. Ultimately, what IT departments need is something that brings attention to major or critical issues so they can be prioritized without ignoring smaller or less crucial alerts at the same time. BigPanda thinks it can be that solution that helps companies pull together input from all sources and boil it down to a prioritized, manageable list.

    — Tony Bradley

  • Q&A: Sematext’s Thies on Docker Monitoring, Part 2

    Q&A: Sematext’s Thies on Docker Monitoring, Part 2

    In part one of my conversation with Stefan Thies, Sematext DevOps expert. we spoke about Docker’s use of Sematext Logsene logging, anomaly detecting and alerting software and SPM monitoring software. Let’s continue the conversation.

    David Geer: What makes Logsene centralized? Where is it centralized a typical deployment? What are the advantages of it being centralized?

    Thies: Traditionally, systems store logs from multiple applications on every server. This makes it difficult to search logs for troubleshooting or auditing purposes. To find a relevant log entry, operators must log in to each server, find the right file on this server and search in this file for a relevant message.

    This can easily take several minutes. With systems storing logs on multiple servers, it is difficult to correlate data, so minutes can turn into frustrating multi-hour root cause-hunting sessions, often involving multiple team members. Logsene provides a central full text index for all log data coming from multiple servers and applications. There is no need to log in to multiple servers and search/grep multiple log files. All team members see all data at the same time and can collaborate much more efficiently.

    Having a central place for search makes it much easier to find logs quickly, run analytic queries or centralize alert rules in a single place.

    Geer: What is it about what Logsene does for Docker logging?

    Thies: Sematext Agent for Docker interacts with the Docker API to automate the collection, parsing and processing of logs, metrics and events. It detects containers automatically (no setup or configuration for a new container!) and supports many log formats for applications on Docker Hub such as Nginx, Apache, Elasticsearch, Solr, MongoDB, MySQL and many more.

    Geer: What does your statement mean, “Some DevOps engineers even think of Logsene as ‘ELK Stack on steroids’”?

    Thies: The ELK Stack is a combination of open source tools: Elasticsearch, Logstash and Kibana. These tools are very popular for log shipping (Logstash), indexing and search (Elasticsearch) and visualization (Kibana dashboards). However, these tools are missing many enterprise features like multiuser user interface, role-based access controls, secure transport of logs or out-of-the-box configurations and an easy query language to search logs. This is where Sematext Logsene adds the relevant enterprise features to these open source tools to make them easy to use and securely integrated into the enterprise information technology infrastructure.

    Maintaining Elasticsearch clusters is another problem for many people. It requires a lot of expertise. Using Logsene does not require this sort of expertise. The Logsene SaaS provides a managed solution where users don’t need to know anything about maintaining Elasticsearch clusters or the required infrastructure. Sematext takes care of that for them. Sematext customers can benefit from these managed services and pay a fraction of the cost compared to self-hosted ELK stacks, where storage, computing and networking infrastructure and manpower for highly skilled workers counts.

    Geer: What is Tutum? What is so important about ease of launching SPM and Logsene Docker on Tutum?

    Thies: Docker acquired Tutum in 2015. Tutum provides management of Docker containers as a cloud service. Tutum is a platform made to operate “Dockerized” application stacks. In the case of Sematext Agent for Docker, the Tutum integration deploys Sematext Docker Agent automatically to all cluster nodes in a few seconds. These nodes could run on different cloud platforms such as Amazon, Digitalocean, Rackspace or on premises. Again, a big time-saver for people running larger infrastructures or hybrid clouds. Supporting Docker products and having excellent integration with Docker products is important for Sematext.

    The Sematext Docker Agent runs on all solutions offered by Docker e.g., Docker Swarm and others such as Google Kubernetes, Google Container Engine, RancherOS, CoreOS, Amazon ECS or Hashicorp Nomad.

    Geer: Please discuss the Sematext Docker and Docker image?

    Thies: Sematext Docker image is available on Docker Hub, and we shared the Tutum Stackfile for Sematext Docker on Stackfiles.io. The easiest way to get it is to use Sematext, which generates the stackfiles for you, including tokens.

    Docker Hub is the central registry for public application images including related installation instructions. Docker users can find out quickly how to run a “Dockerized” application with the help of the instructions in Docker Hub. Typically, this requires a little configuration work before a container can be started with the right settings.

    Stackfile.io is the place where people share preconfigured application stacks (e.g., a web service together with the required database). Sematext provides a preconfigured setup for Sematext Docker Agent to monitor all containers in Tutum Cloud. This means one can launch Sematext Docker Agent and monitor all containers running in Tutum Cloud with just a single click for the chosen stack file and by entering a unique identifier (Token) for the application. That is really all it takes.

    When you log in to Sematext SPM or Logsene the general installation instructions have all required configurations displayed. Sematext users can simply copy/paste the relevant commands including unique identifiers (tokens) for their application to get the Sematext Docker Agent deployed. This, too, is a big time-saver for our users and it gets them up and running in just a few minutes.

  • Q&A: Sematext’s Thies on Docker Monitoring, Part 1

    Q&A: Sematext’s Thies on Docker Monitoring, Part 1

    Stefan Thies is DevOps evangelist Sematext Group. I recently spoke with him about Docker’s use of Sematext Logsene logging, anomaly detecting and alerting software and SPM monitoring software. Read on for part one of our conversation.

    David Geer: Technically speaking, for the purposes of this article, what is logging?

    Thies: Logging means to store the output of programs (applications, servers) running on a computer. Typically, log messages that software applications produce are stored in log files. However, software running in Docker containers writes its log messages to a virtual terminal (console). The Docker Daemon (the server process managing the containers) has several facilities to forward log messages via log drivers or the Docker API to external applications, which are responsible for storing the log messages. This is not the traditional way of working with log data. This changes the log collection and storage mechanisms and forces many organizations to change their workflows for log management by introducing new tools.

    Typically, you would use a service called syslog to receive and store log messages on servers, but this requires special configuration for all Docker containers. While Docker made it easier to deploy applications, the new technology made logging and monitoring much harder as you could not use the traditional tools apart from integration with Docker.

    This is where companies like Sematext provide added value, by enabling:

    • Integration into the Docker Ecosystem
    • Cloud storage for log files
    • Indexing for full text search, analytics, visualization and alerting on log data

    Geer: What has Sematext Logsene contributed to logging, technically?

    Thies: Logsene Log Management includes all required workflows and security procedures to handle log data. Logsene provides role-based access controls for access to log data. Logs are collected and shipped to Logsene servers (running in the cloud or on premises) where the data gets stored and analyzed.

    Other important features are:

    • Full text search
    • Analytics
    • Alerts
    • Anomaly detection

    Geer: In what way is Logsene/Sematext available to users and organizations that seek solutions to capture logging for monitoring their Dockerized distributed applications?

    Thies: Logsene is available on premises and as SaaS in the cloud. It fits in hybrid environments as well such as running the Dockerized application in-house on bare metal while shipping logs to Logsene SaaS. In this case, only Sematext Agent for Docker runs on premises.

    Sematext Docker Agent automatically detects running Docker containers and automatically collects logs from the containers. The Sematext Docker Agent can transform unstructured log messages into structured data for analytics. We do this with integrated parsers and a pattern library, which detects log formats to structure logs. The final task of Sematext Docker Agent is to ship the structured logs from all containers securely to Logsene, where indexing for full text search and analytics queries takes place. The Logsene user interface provides all tools for search, visualization, anomaly detection and alerts on log data.

    Geer: What is SPM for Docker? What role does it play?

    Thies: Both Logsene and SPM collect data via the Sematext Docker Agent. SPM is a performance monitoring, anomaly detection and alerting solution for many server applications like Web servers (e.g., Apache, Nginx), databases (e.g., Cassandra, MongoDB, HBase, MySQL, etc.), big data engines (e.g., Spark, Storm, Hadoop, etc.) and full text search engines (e.g., Elasticsearch, Solr). Sematext Agent for Docker collects not only container logs but also performance metrics of all containers at the same time, plus Docker events.

    SPM for Docker provides operational insights into application performance on the Docker platform. It is not limited to performance monitoring—it integrates with Logsene for log management in a single user interface. DevOps teams can find log messages in correlation with potential performance bottlenecks. Most organizations use different tools for performance monitoring and logging. Sematext has recognized that metrics and logs must be available in a single platform to provide detailed operational insights without having to switch between different tools and without overhead for things like integration of multiple tools, multiple user interfaces, logins managing access permissions for multiple tools, and billing or licensing from multiple vendors such as NewRelic or Splunk.

    Watch this space for part two of my conversation with Stefan Thies of Sematext.

  • DevOps Leadership Series: Monitoring Containers and Microservices

    DevOps Leadership Series: Monitoring Containers and Microservices

    Trevor Parsons (@trevparsons) is a Co-Founder and Chief Scientist at Logentries, a leading SaaS-delivered log management and analytics service. I caught up with Trevor at the Velocity Conference in Santa Clara and asked him what themes were resonating with attendees this year.

    For this episode of the DevOps Leadership Series, Trevor briefly illustrates some current problems with state of microservices and container-based architectures, while sharing a positive outlook for the future.

    https://www.youtube.com/watch?v=zHRE4011WVk

    Trevor says there’s been a lot of development in how to log and monitor containerized systems and he sees better support from Docker for logging, showing an increasing focus on this topic in 2015 and beyond. He anticipates the next year to have huge improvements in managing and monitoring these new systems and architectures.

    Next up in the DevOps Leadership Series, I headed to London for the DevOps Connect: Rugged DevOps event, where we’ll discuss more about Rugged DevOps practices.  In London, I was able to catch up with Helen Beal (Ranger4), Justin Arbuckle (Chef), and Gareth Rushgrove (Puppet Labs).

    If you have missed any of the other videos from this series, you can find them here.  (We’re up to 15 so far).

  • Five Areas Every e-Commerce Business Should Monitor Using Log Data

    Five Areas Every e-Commerce Business Should Monitor Using Log Data

    If you’re responsible for an e-commerce website, I guarantee you’ve been asked about, told about, or wondered about what kind of information and events you should be logging. Important things are happening all the time, and you need to know what’s going on! Right? Are they bad things, good things, or “nobody cares” things?

    So what do you do? Log all the things! Right?! If you’ve ever found yourself awake at 3:00am wondering why you decided to become a software engineer instead of [insert any job that lets you sleep during the night here], chances are it’s because you decided to log everything. Having said that, guess who has two thumbs and logs all the things? This guy.

    You need to focus. It’s okay to log all the things, but you need to focus your attention on the stuff that matters the most, and a simple tool for aggregating IIS logs isn’t enough. A scalable and easy-to-use service that will alert you and provide an interface for surfacing the events that matter the most is critical to ensuring a constant revenue stream and the best possible experience for your customers. Yes, there are lots of components and potential events that really matter when it comes to developing or hosting an e-commerce website. So where should you start? I’m going to break this down into the top five areas that should be the initial focus of your log data efforts. You should not only log these types of events but also put an alerting system in place so you’re aware about things that require your immediate attention. (We use Loggly for this.)

    1. Checkout. We’ll go with the lowest hanging fruit first. If you’re an e-commerce website, your job is to convert customers. Conversion means pushing your customers through the checkout funnel and completing an order. All checkout errors should be logged, and you should instantly be alerted if any part of the checkout process fails.

    This seems obvious, but it’s important you focus on each minute piece of the process including:

    • Address data collection
    • Inventory availability checks
    • Payment processing (Even if you use a third-party service, you need to log any events in addition to protecting yourself from service failures – more on this later.)
    • Authentication and authorization logic
    • Shipping estimation
    • Tax calculation

    …and the list goes on. Not all checkout failures will prevent your user from physically completing the checkout process if you’re doing it right, but many of them decrease the probability that your customer will complete their order. Any squeak in the cog of the checkout process can result in revenue loss.

    1. Add-to-cart. You can’t place an order if you can’t add products to your shopping cart or bag. This is a huge duh(!), but there are cases where shopping cart issues are intermittent and aren’t completely preventing people from adding products to their carts. The bigger danger lies in these occasional or even moderately frequent product adds that fail without your knowledge because orders continue to flow through your website. Causes might include bad inventory availability checks, bad data, or an intermittent bug that’s difficult to reproduce. Protect yourself by logging all add-to-cart failures when they occur, send out an alert, and investigate the problem ASAP. Honorable mention goes out here to logging Wish List and Favorites errors as well. These features are about customer experience, but they’re also proven winners when it comes to converting customers.
    1. Online Catalog/Product Page. Your customers can’t add items to their shopping cart if they can’t find your products. Similar to adding to their shopping carts, the danger here is the intermittent problem that doesn’t always rear its head in an obvious manner. The larger your online catalog, the greater the chance you’re going to miss errors with specific product lines, markets, or other logical groups of products. Bad data is a likely suspect here, especially if you’re dealing with very old data and/or integration with legacy software that has forced you to compromise a bit on data storage constraints. In addition to product and category pages themselves, consider logging issues with suggested, related, and recently viewed products. You’re missing out on impressions and potential revenue if any of these processes fail.
    1. Email Signup. Any marketing director worth their salt will stress the importance of collecting email addresses at every opportunity. Sometimes an email address is the only thing you know about a potential customer, and sometimes it’s all you need to turn a shopper into a buyer. Any break in this process should be identified and fixed immediately. If your website gets a lot of traffic, it doesn’t take much to lose out on collecting a significant number of email addresses and a significant amount of potential future revenue. The trick here is that much of your business logic for presenting email signup to customers may exist client-side. Don’t forget to catch and log those exceptions as well as server-side issues.
    1. Login & Registration. Well, you want to get to know your customers much better, right? Of course you do! All the better to turn wonderful customers into wonderful repeat customers by providing additional features and more relevant content. Failures here can lead to frustration and exits from your website as well as a degradation in your ability to provide a more personalized experience. Pay attention to your authentication and authorization logic as a whole, meaning you need to go beyond evaluating simple form data submission and validation issues. Social media login errors, out-of-sync authentication and authorization cookies, and errors generated by any additional authentication checks should be logged.

    I’ll wrap this up with a short discussion about third-party services. They’re not specifically called out in my top five areas listed above, but they can definitely impact the performance and user experience of your website. There’s a good chance you’re using at least one or more third-party services as a solution for different business problems your company needs solved, and it’s likely these third-party services are an integral component of critical website features. Whether you’re using a third-party service for calculating shipping, sending emails, or finding and displaying the cutest cat GIF of the day, the success of your site is depending on something that’s outside your control.

    Protect yourself from these services. You should know their unscheduled “maintenance” periods better than they do. Chances are you’re in the position to decide which third-party services your company uses or you have significant influence over that decision. Either way, be informed as to how they perform and have data to illustrate where their services are falling short. Are they reliable and worth the eleventy billion dollars they cost, or should your company be looking elsewhere for that cute animated kitty GIF service? It’s likely you’re already protecting yourself from failures here with some fallback business logic, but make sure you’re logging and alerting yourself when exceptions occur. They can have a serious impact on your customer’s experience and could prevent shoppers from becoming future buyers. That’s especially true if these services play in the five areas I describe above (e.g. shipping, tax, payment, personalization, etc.)

    And my last parting thought: The same considerations regarding third-party services apply when your e-commerce platform becomes more of a distributed solution where those distributed services and applications are your own. You will almost certainly have other performance monitoring and metrics in place for your own services, but you still need to log, aggregate, alert, and protect yourself from yourself!

    About the author/Aaron Remaklus:

    aaron rAaron Remaklus is chief architect and development manager for Speedway Motors, Inc. He leads agile development teams building high availability, extensible e-commerce platforms. A Nebraska Wesleyan alumnus, Remaklus taps into a diverse background in business, marketing, training, and software development in tackling large-scale problems.

  • Advanced Automation – Getting Your Systems to Work for You

    Advanced Automation – Getting Your Systems to Work for You

    In my previous post I discussed how to take your DevOps to the next level by taking it beyond infrastructure automation, to the automation of your deployments and code pushes, through patches and updates.

    And then I promised to make it interesting…so here is the next stage – actually using the extracted data to get your systems to work for you.

    Monitoring Like IT’S YOUR BUSINESS

    Let’s start by discussing what it means to monitor your application, and what kind of data you can extract from it.

    At the most basic level, you want to know whether your application is available for your users. It sounds very basic and simple, but setting it up properly is actually not an easy task. It is intended to give you an answer to the most important question to your business: can my users use the application? Think of it as a big red or green light on your dashboard. Although the answer may seem obvious and simple, but it is not at all. There are all kinds of parameters to take into account when answering this question, for example, what if the system is apparently up and running but the response time is very slow, or what if only parts of the system are actually running. At the end of day, you want to have that kind of an alert “traffic light” that tells you whether your system is functioning – yes/no, up/down.

    System, Application, and Business Metrics

    The green-red traffic light is an aggregation of multiple levels of monitoring. The most basic one is system related. It contains indicators like process availability, CPU and memory utilization, etc.

    The next level of monitoring is application related and is composed of application level KPIs (key performance indicators). KPIs can be anything from the average response time for a user for a certain request, to the number of concurrent database connections. In other words, anything that’s specific to the application and its architecture.

    And then above these, comes the highest level of metrics, which are business metrics. Ultimately these are what’s really important. If your business metrics are ok, then you can assume everything is running fine. Sample business metrics include the rate of failure to register to your website, how many users did a certain operation in the website or how many users executed a specific transaction. If we want to be really specific, then take Google hangout for example, a business metric of theirs would be how many people in a certain timeframe joined or started a hangout.

    Logs to the Rescue

    Logs are often used here as means to collect your metrics and alert about erroneous conditions. Logs usually contain a lot of useful information, but you need to be able to extract it and make sense of it. This means gathering all of the logs emitted from all of your servers, parsing them and looking for specific patterns that will help you generate the KPIs over time. For example, you look for a log message that says “User X signed in”, and by counting these messages you can produce a business level metric for how many users register over time. If you add some more data to the log message, such as the user’s location, or operating system, or anything else for that matter, you can slice and dice data and get deeper insights into how your application is behaving. When logs are emitted in a structured and consistent way (e.g. in a JSON format), it becomes very easy to analyze them and produce the relevant KPIs. There are many toolsthat can help with that.

    From Simple Automation to Orchestration

    That’s where orchestration comes into the picture. At the most basic level, orchestration is a higher form of automation, which helps you setup all the pieces that are related to your application, starting from the infrastructure (VMs, networks, block storage volumes, security groups, etc.), to the platforms your app runs on (database, web server, etc.), and all the way up to the application modules and code. This entire setup is often referred to as a topology. The role of an orchestration framework is to materialize a certain topology. More advanced orchestrators go beyond materializing the topology, and change it to meet the current workloads and needs of the application.

    As you probably gathered by now, monitoring and log gathering are an essential part of any running app, so they should be an integral part of the orchestrated application topology. Since the orchestration process is topology aware, it can wire and configure monitoring for your application components very easily, which is one of its greatest benefits. Going through these process without a global view of the topology can often be time consuming and error prone. Moreover, as the topology changes , you need to reconfigure and rewire your monitoring tools, and a good orchestrator will that for you as well.

    Next Up: Reactive and Proactive Orchestration, The Devops Holy Grail

    Hopefully by now you see the value of orchestration as a higher form of automation. In the next post I’ll dive deeper into the post-setup phase, and discuss how orchestration tools can react to events and monitoring data and adjust the application’s runtime topology to best fit the current workloads.

  • Logging Wins for Devops and Security

    Logging Wins for Devops and Security

    Its always great when companies can invest in tools that serve multiple groups and purposes.  Ensuring proper setup and use of logs is one toolset that can serve both DevOps and Security.

    Why logs are important to DevOps

    DevOps use  logs primarily  to understand what’s going on inside an application server, database system or the operating system.  For the most part, logs are a resource that helps DevOps diagnose problems and ensure proper working order of systems.  Sometimes logs provide a tracking mechanism that helps DevOps understand what customers are doing in their application.  DevOps folks are intimately familiar with newer and popular logging aggregation services like Splunk, SumoLogic and Loggly.

    Security Loves Logs Too

    The good news is that security teams love logs too.  The use, review and management of logs is so important to the security function of an organization, its even part of the SANS top 20 critical security controls.  From a security standpoint, logs provide a ton of important data for everything from forensic analysis to audit records.

    Recommendations

    Since logs are so important to both DevOps and Security teams, it pays to implement your logging tools in a way that will serve the needs of both teams. Here a few quick tips that will help you set up logging tools in a way that will create a better world for both DevOps and Security.

     

    Centralized Logging & Log Off Site

    Setup all your systems to log to the same centralized location.  If you have all your logs in one place it’s much easier to setup log reviews and alerts.  Its also a good idea to log to multiple locations for security reasons.  Consider logging to a centralized location within your network and also to a remote location.  For example, have all systems log to a central server and also log to a remote service such as SumoLogic, Loggly or Splunk.  That way you have multiple copies of your logs available for forensic purposes.

    Time Sync

    Make sure all your systems are in tight time synchronization.  There is nothing worse than reading your centralized logs to find that a few systems are not properly time synchronized.  NTP is your friend for setting up time sync.  Be sure to use two or more reliable upstream time servers and configure all your servers to peer with each other.

    Applications and Operating System Logs

    Start with the most common base configuration  so that all systems use syslog to capture and distribute logs.  In the beginning, the logs in syslog will probably only be from the operating system and few processes that  are already configured to use syslog.  However, there are a ton of other processes running that should also be logging to your centralized system.  The DevOps group will want to ensure that all applications, even custom developed code, logs important information.  Make sure that your custom apps log enough useful information that problems can be diagnosed when they occur.

    Setup Alerts & Reports

    Nobody wants to be trolling through tons and tons of logs. To prevent this, one of tne of the first things you should do is to set up a basic set of alerts and reports.  Security teams will want to make sure that logs record and alert when users gain administrative access to a system or an application, but there are lots of other events that should need logs and alerts. Spend some time thinking about events that could mean something is not working correctly and needs attention.  Ask the development team to specifically log non-critical events that should be reviewed and incorporate these into a daily report.

    Be Verbose, But With Care

    Crank up the logging levels on all systems but don’t go too far. You want to be careful you aren’t logging too much.  For example, you probably don’t need to run all your systems in debug mode all the time.  You also need to make sure you aren’t logging sensitive information like user passwords or authentication tokens. With logs, there is such a thing as too much.

    If logs are set up correctly, they will provide nearly endless benefits to both the DevOps and Security teams. Now, go forth and log!