devopsy.kinsta.cloud

Author: Pete Cheslock

  • We Still Haven’t Solved the Logging Problem

    We Still Haven’t Solved the Logging Problem

    At a company I previously worked at, I helped build out the initial infrastructure to support a large cloud security product. It was one of the first times I had the opportunity to build something from nearly the very beginning. Like many operations professionals, so much of my career prior to this experience had been spent cleaning up other people’s messes and technical debt that I was finally excited to “do things right” from the onset.

    Looking back, I wish I knew that building our monitoring platform would be hampered without the ability to retain and analyze both warm and cold log data. Here’s why.

    Time Series to the DevOps Rescue 

    Hindsight is almost always 20/20, and today I can proudly say that my team and I built a killer app monitoring platform. My goal was to make it easy for Dev and Ops folks to report time series metrics for their applications and build dashboards and alerts for app health. And we did just that.

    While many ops colleagues would say that monitoring journeys usually start by capturing application logs with open source tools such as Logstash/Elasticsearch, Graylog2, or even buying commercial services from Splunk or Sumo Logic, not all journeys start that way. My team and I took a different approach and optimized for time series metrics versus the more traditional structured log management approach, leveraging newer toolsets such as Librato for real-time operations monitoring and performance analytics. Librato enabled us to scale much faster than with traditional log solutions and gave us almost instant visibility across our containers, cloud servers and the apps themselves.

    Logging is Too Expensive 

    Using hosted services such as Librato to manage our time series metrics worked fine for a while. However, the cost of SaaS monitoring eventually grew, and, since we had the time and energy to bring metrics in house, we deployed Graphite, the open source enterprise-grade monitoring tool. Graphite allowed us to continue enabling our engineers with affordable access to granular time series metrics to help them manage their applications.

    While this allowed the business to respond quickly when an issue arose, time series metrics were only part of the solution. They allowed us to see what was happening, but not always why it happened. While time series metrics and even some of the more advanced tracing technologies including Jaeger and Opentracing are fantastic to help debug complex distributed systems, they fall short when investigating and analyzing causes for problems, issues and hacks.

    Digging Into Your Log History Gets Expensive, Quick

    Tools such as the ELK stack have been booming, with Elastic alone hitting more than 100 million downloads last year, which shows that DevOps teams want more from traditional log management solutions. They’re looking for simplicity, ease of use and affordability, among others. With JSON effectively the de facto standard for logging, most teams take log data with its predefined schema and then index it into Elasticsearch/Lucene. Unfortunately, the Lucene indexing mechanism results in an unintended consequence: a 5 to 10X increase in disk usage. And just like that, ops cloud budgets get blown away when they realize that 10GB of JSON data indexed into Elasticsearch could be 50GB of disk used. You wanted that index to be replicated for disaster recovery as well? Then add 100GB of storage for every 10GB of logs.

    DevOps Normal: Still the Tough Choice Between Retention and Cost

    Unfortunately, most people choose to simply purge data from our “hot” Elasticsearch clusters as soon as we possibly can to minimize cost issues. This is far from ideal.

    Companies moving to microservices-based architectures, deployed on containers such as Kubernetes, are going to be in for a rude awakening when they start building out their centralized logging platforms to keep up with the volume. Security teams are going to feel the burden as well, as they come under increasing pressure to keep data around for long periods of time due to compliance and audit reasons. When the next big web-facing vulnerability comes out, the ability to go back weeks or months will be key to see what IP address accessed a potentially malicious endpoint. With that data available, we could then take all those IPs and correlate them with other access logs.

    Is centralized logging any better than it was 10 years ago? Or has all this data generated exploded so much that the tools we have today are just barely keeping up? One thing is certain, this problem is far from solved, and many companies may find themselves drowning in their own data lakes.

    — Pete Cheslock

  • DevOps: Here’s How to Slack Securely

    DevOps: Here’s How to Slack Securely

    With 1.7 million daily active users, it’s clear Slack has come to dominate the team chat world, especially in the tech industry. This, however, means that a fair amount of security is required to ensure that strategic assets remain safe.

    From a security perspective, Slack has done a solid job of keeping its assets on lock, going so far as to score Geoff Belknap from Palantir in 2016 as its chief security officer. The company is also transparent about its approach to security and has dedicated a whole section of its website to it, including interviews with Belknap and others that delve into Slack’s precautions and philosophy around security.

    The company has also certified many of its products to meet strict compliance regulations including FINRA, HIPAA and SOC 2 and 3, which makes it a no-brainer for small teams and enterprises alike.

    So, it’s perfectly possible for companies of all shapes and sizes to lean on Slack for team chat and ops without worrying too much about security. However, no one’s perfect, and Slack’s ubiquity and popularity mean it will always be a target for cybercriminals looking to steal information—that’s basically the nature of being a cloud-based application.

    While, there’s no need to run scared, you do need to be smart about how you use this valuable tool. Here are a few tips for running Slack securely at your organization.

    Use Slack for SecOps

    Slack can be easily used for distributed security alerting, which means every member of your organization can play a role in keeping the business secure.

    Your security operations likely include members of the DevOps team as well as dedicated security folks, and Slack can help all team members integrate security into their workflows seamlessly. DevOps folks may not frequently log in to Threat Stack platform directly. However, any relevant alerts or notifications can be sent to them immediately via Slack, where they can be reviewed and the determination can be made whether further action is needed—all without having to step outside the daily workflow. This means security tasks don’t pose an extra headache, but are just a natural part of the way the company’s operations run.

    Require Two-Factor Authentication

    Slack’s built-in security precautions won’t do you much good unless you actually put them into practice. One great example is two-factor authentication (2FA). It’s up to team owners and admins to require this of their users (otherwise, it will be optional, and most users won’t bother). It is highly advisable you take advantage of this feature, which makes it much more difficult for hackers to tap into your organization’s Slack channels.

    Set Up User Provisioning and Deprovisioning

    Security-minded organizations need to be conscious of and work around insider threats. This also includes employees who have left the company, whether on their own terms or due to an incident.

    To ensure that you do not have any “lurkers” who might be able to take advantage of company information shared on Slack after they have left the company, it’s a good idea to plan ahead for user provisioning and deprovisioning.

    In fact, this should be built into your process for onboarding and offboarding employees, just like it would be for email or any other company asset. If possible, automate the process, so that the moment someone leaves the company, they no longer have access to Slack. To get started, here’s Slack’s guide to provisioning and deprovisioning users.

    Don’t Share Secrets

    Slack is a great place to have secure conversations, which unfortunately can lull some users into a false sense of complete security. “Secure” doesn’t mean you should treat it like it’s watertight; in other words, Slack should never be used to share secrets such as passwords, sensitive customer data or valuable corporate IP—basically, anything that could be considered highly confidential. A good rule of thumb is this: If a piece of information could be dangerous in a hacker’s hands, it doesn’t belong on Slack. Instead, it’s a good idea to use encrypted communication channels, like PGP-enabled email.

    Educate Users

    None of the tips above will do your organization any good if no one knows about them. So make sure that you regularly educate your users about steps they need to take to stay secure while using Slack (such as never sharing passwords there). You should hold user security training whenever new employees come on board, and also make sure to do a refresher now and again with the entire company.

    Additionally, if you change up security protocols around tools like Slack, make sure employees are given a heads up and reminded when new measures go into effect so they know what to expect. Remember: The best offense is a good defense.

    Don’t Slack on Security

    Slack is a great platform with all kinds of benefits for teams. As long as you take the right security precautions, there’s no reason why it can’t be used to its full potential on your team, whether you’re a small shop or a multinational enterprise. As with any other tool, the shared responsibility model is key. Take responsibility for your half of the security equation, and you should be well on your way to a secure Slack implementation.

    — Pete Cheslock

  • Having DevOps In Your Job Title Is Doing You Harm

    matrixThis post was originally published on my personal blog in May 2013.  But I think this topic is important and wanted it to be my first contribution to devopsy.kinsta.cloud

    I used to be Director of DevOps…twice. Both times, I changed my title later. I even ran a DevOps team…although the team was already called that when I took over.

    I have fallen victim to the the abuse of the DevOps title, and I see it all the time. In the DevOps twitter hashtag, people contacting about DevOps job opportunities…it’s got to be a real thing, right?

    I mean, we totally have people that are doing DevOps these days, right?  I know that I can find all kinds of job descriptions for DevOps Engineers from many types of companies.

    But, there is one very important reason why you might want to think twice before using “DevOps” as part of your job title.

    My friends (rightfully) give me grief for the fact that I self-assigned my title at Dyn to be the Director of DevOps (I still feel bad about it). I really hate job titles in general and this is what happens when you don’t care about what your title is.

    How the story goes is that when I received the offer from Dyn, my original title was Director of System Automation and Release Engineering. Since I was pretty sure that wasn’t going to fit on a business card, I took the job and asked that we change the title later. Fast forward a month, as after talking and trying to formulate my job description & title for about 20 min, I say simply enough, ”F this. Just call it DevOps, I’ve got other things to do.” I was far too busy to really care much about an arbitrary title.

    Fast forward a few weeks to Monitorama, and I was chatting with Mike Rembetsy. After talking for two minutes about how the new job was going, he said, “Oh, that reminds me, you need to change your job title. It’s terrible.” I tried to convince him for about three more minutes, mainly trying to express to him that it doesn’t matter. He then dropped the knowledge bomb that gave me the aha moment.

    When you are the Head of DevOps, you “own” DevOps. If DevOps fails, it is your failure, when it should be a failure of the entire company to change, adapt, and accept the cultural shift.

    That is the major problem with being the “head” of DevOps, or the “Director/Manager/VP/etc. of DevOps.” If DevOps fails inside your company, it’s a really easy way for the executives to say “Well, you are the head of DevOps so obviously it failed because you failed.”

    The reality is that DevOps is a culture that should be accepted by every facet of your organization. Everyone needs to embrace the changes of a DevOps culture, and there can not be a single owner of it.

    Soon after that conversation, I relaunched my team and title in general to be the “DevTools Team” (a brilliant Etsy recommendation). The more I thought about it, it accurately describes the work that we do. We build the tools needed for Developers (and Ops/QA/NetSec) to spend more of their time writing code, and less time dealing with distractions.

    I know some people want to have a DevOps team, or want to call their team the DevOps team. Jez Humble wrote a wonderful blog post last year that I highly recommend that you read. He summarizes that there are scenarios that a “DevOps” team label works, and I do agree with what he says. But don’t just label a team DevOps, assign a new Manager of DevOps and wonder why three or six months go by and things are still bad. Things are still terrible because all you did was change a team name, or give a brand new team a name, or in some cases , just created a new silo called the “DevOps Team Silo”.

    Well, what am I supposed to call myself?

    Here are some of the examples that came up during the Open Space:

    • Site Reliability Engineer
    • Automation Engineer
    • Release Engineer
    • DevTools Engineer

    I don’t know why recently there is a need to move away from the same job titles we’ve used in the past, as if there is some negative connotation towards them. It’s not that the jobs have really changed much; it’s that we’re just getting better and more efficient at doing them with the tools at our disposal. The tools have grown so quickly, it has enabled us to be much better at our jobs, but only if you are willing to learn something new.

    • System Administrator
    • Operations Engineer
    • Network Engineer

    In the end, titles really don’t matter very much, and they shouldn’t matter much. But they can actually have a more powerful (negative) impact internally by making you the owner of a culture that should be accepted by everyone. But there is one point that I believe is so important it needs to be repeated:

    If DevOps fails, it should be a failure of the entire company.

    Check out my talk “How to Keep the People You Need,” where I discuss recruitment and retention of your hard to fill positions. Also check out the panel on DevOps with Twitter’s Rich Paret from Dyn’s Geek Summer camp.