devopsy.kinsta.cloud

Author: Matthew Boeckman

  • From here to there: practical first DevOps steps

    From here to there: practical first DevOps steps

    I’m a bad programmer.

    There, I got that out of the way.

    The truth is, I’m a sysadmin that writes code infrequently, on an island, and out of disciplined requirement, release, or QA practices. I need to run an ETL that kicks a file transfer. I’ll write some code. I’ve got a workflow that needs a crappy but functional web UI. Now I’m a developer. But in between these kinds of projects are days, and sometimes weeks of meetings and planning and dozens of things that are not code, which means by the time I come back to actual programming, I’m starting fresh every time.

    I don’t think I’m alone. I think there are a lot of sysadmins in the world with functional perl chops, but because they don’t code daily, they don’t approach problems with an inherently code-focused approach. Systems are still running, the internet is still on, so this way of solving problems must be OK. It is, but there are drawbacks.

    Rolling changes out to 4 dozen of your 8 dozen systems and verifying each received a 4 line change to that logship script with scp? Good luck with that. The old SA just left, and now you need to find all his stuff? GLHF. Some report hasn’t been sending for 4 days, what changed?! Yeah… yeah.

    Where software development got it right

    Great systems need three things: consistent configuration, maintainability, auditability. Good dev & deploy practices get you all three. Version control systems like git provide a great mechanism to manage code, version, and collaborate in reliable and predictable ways. Issue trackers, like Jira or Rally that integrate with version control let you easily tie specific changes back to the original reason for the change. Deployment systems like Jenkins and Capistrano give you one stop interfaces to track what got deployed where and when and by whom.

    Take it as given that few SA teams are using all of these approaches for their work today. Probably you’ve got a ticket system, and you definitely get yelled at if you don’t comment your scripts. Maybe those comments link back to the work order number? Migrating what you do today to a more DevOps focused approach isn’t going to happen overnight. So how do you start?

    Version control or GTFO – Yeah. I know. It’s on the list! No. Stop what you are doing, and gather all of your nginx.conf’s, your limits.conf, pg_hba.conf, and commit them to a git repo. I can’t emphasize this enough – if you are not enforcing version control in your systems administration team you’ll reap the whirlwind.

    Change, Commit, Test. Welcome to the new DevOps discipline. You’re going to hate it at first, but 7 months from now when the webservers are flopping because somebody changed something, you’ll be grateful for a quick and reliable rollback.

    Puppet or Chef or Ansible – get a basic setup. One master, three nodes. My starting point here was user management, I setup a basic user-add class, and let it manage ssh keys. Let that soak for a couple weeks and slowly roll it out to the rest of your infrastructure. Ten bucks says you find a dozen hosts with inconsistent UID’s that you’ve been meaning to fix for years.

    Once that feels comfortable, move some basic scripts into your setup, roll those out slowly. Take the db configurations, commit them to the repo, and move ’em over to configuration mangaement too. The point is, moving to configuration management systems is not an atomic event across your entire infrastructure, iterate against it. Make small weekly changes that roll up to quarterly progress.

    Here’s where the rubber meets the road:

    “Git is going to suck because deploying is so much easier when you just vim a file on a server” – Every Sysadmin Ever

    If you’ve got a repo, and you’ve got say, puppet… it’s a pretty easy step to have commits to that repo pulled and auto-deployed to your puppet managed nodes. Which means rollbacks can be too.

    Pay off your tech debt – I don’t have to tell you about the 4 dozen scripts you’ve got out there, unmanaged, brittle, and needing constant tweaks to keep them lined up with system changes. These are the bread and butter of system administrators, and can provide a great opportunity to implement change. Next time you have to touch that script for a new SCP server, stop. Stop! Do this instead:

    • check it in to your git repo, with some meaningful comments about what it does
    • add it to your configuration management system (puppet/chef/ansible)
    • if it’s bash or perl, rewrite it as python. It’s going to take a bit more time to rewrite all your scripts, but spending this time gets you more comfortable writing code regularly, and gets you more familiar with a modern scripting language.

    Walk the path

    Whatever your system or your goals, the hardest part of adopting DevOps approaches is going to come down to time. Reading through this myself, I’m struck by what an insensitive jerk I am. “Hey guys I know you totally have tons of free time, just starting doing everything you regularly do but add a ton of extra work too. Kthxbai”. The reality is, I probably am a jerk, but you don’t have to think about moving to DevOps as a shining city on a hill that you can reach with enough late nights. It’s a journey, a path, a way, not a place to arrive.

    Find a way to start making small changes to your existing processes that line up with the best of the development world. Decide to add 10% more time to a proposed project to allow for it to be handled with these methods. If you begin to turn your steps onto the DevOps path, you’ll quickly realize you’ve already arrived at your destination.

     

  • Operational Overhead – or why hubris is bad.

    Operational Overhead – or why hubris is bad.

    Stop me if you’ve heard this one: A guy walks into a meeting and says “We need a system that does something”, and the engineer in the room replies “Oh, we can totally build that”. Most of the punchlines to this joke go “we can totally build it, and now we have to own its lifecycle and we have sacrificed valuable time and resources for something we should never have built.” ba-dum-ching

    I am guilty of playing my part in these conversations, as I’m sure many of you are. About two years ago, as our budding startup was really starting to take off, I was in one of those meetings. Some of the developers wanted to try a recommendation engine, and were going to use Mongo as part of it. Frankly, I thought they wanted to screw around with Mongo and needed an excuse as much as I thought they were going to really do something. At the time I was drowning – running our cloud systems, our LAN, facilities, purchasing, and I had just been tasked with bringing some sanity to the tech side of our video post-production process. I. Did. Not. Have. Time. For. A. Mongo. Setup.

    See, here’s the thing. Most of the time the system administrator in that conversation, trying to be a good devops partner, replies “oh sure I can stand up mongo”. But let’s be honest, it’s not just standing up Mongo. It’s handling upgrades, security, failover/HA, backups and restores. If this recommendation engine actually worked it was quickly going to become a key part of our infrastructure. I had just enough presence of mind to know that while I probably could find an hour to stand up mongo, I didn’t have the several other hours needed to do it well. And really, I didn’t have an hour. So I quipped a now-famous-and-often-used-to-troll-me response “Look guys, we need to reduce our operational overhead. Let’s find an outsource.”

    Ten minutes and a credit card later, we were setup with MongoLab. Two years later? The recommendation engine is awesome, and I have not cared one ounce about mongo. No upgrades. No middle-of-the-night pages. No architecture discussions. It just works. I’d like to spend a few words dealing with the standard objections:

     

    • OK we can try outsourcing, and if it works we’ll bring it in house. Right? Because you never run key pieces of the infrastructure out of your control. Right? Because, if you do… then… that’s bad. Right! Ye Olde World IT Wisdom is pretty clear on this point, you can only ever be sure that something is good if you’re running it in-house. In my experience however, you’re probably more likely to fumble a backup, fail an upgrade, or bungle a failover than an outfit whose sole job is to do those things.
    • But! it’s expensive! Right? Just about any *aaS provider has an entry level pricing structure that lets you screw around for less than the cost of that team lunch you’re buying on Friday. If you’re in production, and you are making money, or even better yet more money because of this infrastructure choice… justifying the cost is stupidly easy. As long as the cost of the service is less than the money you are bringing in, good job. If you aren’t actually making money, then what exactly are you doing?
    • We can’t trust Service Provider X. Guys, these companies are in business to provide 1 thing for you. They literally have 1 job. You probably have 14 jobs. Are you asking for a 15th? A 15th thing that you have to watch, and care for, and feed, and love, and hate? A thing that is not core to your actual business? Shouldn’t you make that 15th job something directly beneficial to your team/code/organization? Isn’t that what you’re here for?
    • OH GOD NO NOT VENDOR LOCK-IN. Hey genius, you know what? You have to write code, and whether you are writing code talking to a database in your environment, or a queuing system in someone else’s… it’s just code. I hope to spend most of the rest of my life writing about how utterly stupid this argument is. Yes! If you fire a vendor you have to refactor. You know what? If you upgrade your in-house stuff you probably have to refactor. If you change the in-house system you have to refactor. Anyone prescient enough to foresee all possible outcomes, and who can make perfect, refactor-free decisions can talk to me about vendor lock-in. The rest of you should move past this objection and get back to doing something productive.

    You know what my favorite operational overhead story is? Amazon SQS (Simple Queue Service). Different day, different developers, same idea. Wow, our little website is now processing enough orders that we need asynchronous fulfillment… I know! Let’s setup a queuing system! I used ActiveMQ once! Or RabbitMQ! Or… we can use the same queuing system that runs the largest ecommerce site on the planet. Where there is literally a horde of highly trained monkeys watching every teensy little thing that makes it work 24x7x365. Someone else has already thought of every imaginable permutation of scale/resiliency/security. And you know what? It costs 0.0000005 per request. I can process 14 million orders at a net cost of $7. Yeah, let’s totally spend time standing up something better and cheaper.

    Good engineers are good because they know they can build things. Even when they don’t know they can, they are pretty sure they can figure it out. DevOps should be about making smart decisions, and hubris is not smart. Use the best tool at hand. Rebuilding every wheel, administering every platform, doing everything yourself… is just adding to your operational overhead, and subtracting from your ability to be impactful to your business.

  • Can you appreciate DevOps, if you can’t exactly say what it is?

    I recently had a conversation with some SOX IT auditors and was asked what I meant when I said we were a DevOps shop. I stumbled a little, torn somewhere between an overly simplistic answer and a desire to baptize the unholy. I started by talking about collaboration between the developers and the admins. Mentioning that many of our developers are on call and have sudo access raised some eyebrows. Using Puppet made them pretty happy.

    Thinking back on it, what I realize is that for me devops isn’t a strictly defined thing, it’s a mix of operational changes that draw on the best of any tool, without prejudice. We do what works. We do not do what does not work. If we are doing something, and it ceases to work, we stop doing it. I don’t know if that’s devops, but it’s what we do. And it works. So we’re going to keep doing it.

    Configuration as code

    This is not news, central and key to successful devops is controlling system configuration and change management with code, not with disparate manually managed text files. Here’s the thing, you cannot operate at scale without it. You also cannot change direction, or iterate through rapid change without it. You can rapidly, accurately, and verifiably push changes, stand up environments, or demonstrate a fact to an auditor with just a few keystrokes.

    To be sure, it’s a helluva lot easier to start a business on these principles than to migrate after your business is a real thing. However it’s worth the effort.

    Sysadmins are our friends, Developers are on call

    Most places I’ve worked, neither of those statements were strictly true. At Craftsy, both are. The relationship we foster is one of open collaboration. I’ve never been somewhere where the sysadmins were jerks enough to really justify the animosity they received from dev, and I’ve never met a developer so hopelessly irresponsible to really justify the vitriolic mistrust dealt out by the SA team.

    Give your developers sudo and get them in the oncall rotation. Now, they share a burden with your SA group, and a responsibility to the systems their code runs on. Figure out a sensible rotation and escalation, but without that shared responsibility you will always struggle with trust between your dev and your ops.

    SaaS, Paas, IaaS, CotS, Cloud, OnPrem, Whatev

    The punchline is “all of the above”. Never let yourself get married to a particular way of doing things. We run stuff in the cloud. We run stuff locally. We buy software. We rent software. We rent services. We borrow code. We cloud. And whatever comes up next, we’ll try it one way, whichever way seems to make the most sense. If it works, we’ll keep doing it that way.

    I’ve heard some reasonable objections to Cloud/*aaS based on security concerns. Safely extending the trust relationship of your LAN isn’t easy. It isn’t impossible either. Tools like the AWS VPC offering, or SSO/IAM systems offer some very flexible approaches to open your environment and leverage the best tool, regardless of how it’s made available.

    Do what works

    I’m not sure exactly what the SOX guys thought of my ideas around devops. IT, much like accounting, is an area that traditionally is slow to adopt new approaches. It’s a counterintuitive truth that an industry so deeply engaged with technology is simultaneously so awkwardly regimented in its ways. The standard answer, that change has to come slowly to maintain stability is unsatisfying. Indeed, it belies the fundamental failings of the old methods.

    DevOps sets the table for you to achieve stability in any environment precisely by engaging deeply the technology around you.

  • So, you wanna DevOps?

    So, you wanna DevOps?

    How do you start your career in DevOps? How do I shift my team to use those approaches? This is a question I run into more and more frequently. Either while speaking to candidates, or among colleagues looking to move their IT operations into a more devops aligned culture.

    I want to look at that challenge from the perspective of a System Administrator. I’ve been an SA for 16 years now, and was able to fumble my way into being a pretty decent Devops Engineer. Much improved from fumbling, there are some clear objectives you can focus on that will get you there quicker.

    What isn’t devops?

    First, let’s talk about some of the differences between conventional SA and a DE. Many of these differences are direct results in the difference between conventional IT or Technical Operations and devops. A traditional SA role would involve

    • sole or shared on-call responsibilities only within the SA/NA teams

    • scripting for administration (bash, perl), very CLI based

    • silo’d from development, and possibly application support groups

    • heavy focus on stability and process

    • cultivating the walled garden

    Great DE’s share a similar skillset with SA’s – strong understanding of OS and networking concepts, deep familiarity with hardware or resource planning, diagnosis of performance bottlenecks, and instrumentation and tuning against servers or systems. In addition, DE’s

    • share on-call responsibilities with developers and overlap duties

    • utilize more robust languages (python, php, ruby), utilize web and database frameworks

    • embedded with day to day operations of development and application, actively contributing

    • heavy focus on innovation and agility

    • flat rejection of walled garden mentality

    There’s clearly a skillset component – in devops we’re writing code to admin and maintain systems, while implicitly acknowledge that systems are part of a broader ecosystem. I’m just as likely to address a file in an object storage system (s3, openstack) as I am on a file backed system. When I perform a system level action, I’m doing so because of a database flag, not a cronjob. I’m sending the result of a job to an API, not a filesystem log.

    But perhaps more importantly there is a mindset component – in devops we’re friends with the developers. I say that tongue in cheek, but am quite sure everyone reading this knows what I mean. The thing I’m most looking for in a devops engineer is their ability to relate to, and share responsibility with developers. I’m looking for developers who are excited to share responsibility with operations and contribute to the systems their code runs on. I think this is the most marked contrast between old world IT and devops – they integration of two traditionally warring tribes.

    So how do you start the transition?

    Code. Write code. Not shell, not perl. Perl is code, so is shell. Choose a language, it doesn’t matter much which, and write a basic CRUD app that you can interact with via a web browser. In my experience this kind of coding is rare in traditional SA focused folks, but absolutely critical for a DE. As part of your project, schedule at least 2 hours to feed the trolls about whether python is better than ruby.

    Puppet or chef. Stop everything you are doing and pick one of these up. Make yourself take an existing system and install apache on it using one of these tools. Configure apache using the tool. Make changes only in puppet or chef. As one recent AWS tips post said, disable ssh. In 2014 if you are not using puppet or chef you are terribly handicapped.

    Get into the cloud. Set up an EC2, Google Compute, or Azure environment. Start and stop instances only with API calls. Make sure that once started, the instances are applying a puppet recipe. You can have a functional playground for free, I’ve had a personal goof off area in ec2 for a couple years now. $18/month or so is a small investment in your career. Set yourself some basic tasks, and implement them in a cloud environment. 

    DevOps is here to stay, and is a massively impactful methodology for quick iteration against systems and code. Whether you’re at a startup, or an enterprise, or looking to shift… adding some basic devops tools to your bag o tricks will be time well spent!