devopsy.kinsta.cloud

Author: Ben Good

  • Scripts or Machine Images, What is the Best Way to bootstrap

    Scripts or Machine Images, What is the Best Way to bootstrap

    Between the advent of handy tools, like Chef and Puppet, and virtual machine infrastructures, like VMWare and AWS, I feel like there has been a great debate about how to bootstrap machines.   It seems like there are two extremes.  On one side is heavy reliance on machine images (AMIs, in AWS speak) or scripting (recipes, in Chef speak) on the other with a bunch of grey in the middle.  Naturally, there are pros and cons with each so let’s explore them.

    Machine Images

    The great thing about machine images are you get from zero to a fully functional machine in the time it takes to create the virtual machine.  This can be an extremely valuable thing when you need to spin up or down quickly.  However, they become dated with the rate at which your configuration changes.

    Scripts

    With the all scripts approach you are immune to the out of date issue encountered with machine images.  The big drawback is that it can take a long time to spin up a new machine off of scripts, especially when you have to compile one or many components.  In my opinion, this approach is most valuable when your system configuration changes quickly and there is some tolerance with regard to spin latency.

    Either approach is suitable for things like auto-scaling and unlocks the elasticity of the cloud, so which is the best?  To help you decide on which approach, or blend there of, best suits your situation. These may seem obvious, but I think you must weigh three major trade-offs or benefits afforded by scripting or imaging.

    • Time to being service ready

    How much time can you afford to spend from the time you start bootstrapping a new machine until handling live traffic?  To meet our on-demand scale requirements, at Clip, we have a target of 10 minutes from when the virtual machine is requested to handling live traffic.

    • Resilience to change

    How dynamic is your environment?  How often are new services deployed, dependencies added, or code deployed?  At Clip, we add new services on a fairly regular basis and our systems change frequently from both the code and dependencies stand point.

    • Cross platform support

    How many platforms do you need to support?  Platform, in this case, could be interpreted as operating systems, cloud providers and/or virtual or physical hardware.  To date we have been able to minimize any cross platform issues.

    Since we value flexibility and have the luxury of an acceptable spin up time of 10 minutes, we have chosen to lean heavily on Chef for our bootstrap process and only use basic machine images.  However, as the product and business requirements evolve, the time to being service ready is gaining more emphasis, so our process will also change.  At other companies, spin up time is so important, their CI system builds a new machine image for each version of the product that is deployed, so they can scale as quickly as possible.  Regardless of the technology or process used, it is most important to satisfy business needs in terms of service ready time, resilience to change and cross platform support and continue to adapt as the requirements change.

  • My DevOps Bookshelf

    My DevOps Bookshelf

    Aside from the many technical manuals on our bookshelves, I bet everyone has a few that are more process or philosophically oriented which hold a special place in our hearts.   To me, these are the ones we continue to go back to and that have helped develop us as engineers, software, operations, quality assurance or otherwise.  In no particular order, the following are the top four, not strictly technical, books in my library.

    Web Operations by John Allspaw and Jesse Robbins

    Allspaw and Robbins cover many topics in this text ranging from monitoring and metrics to databases and operations development relations.  I appreciate this one so much it is required reading material for anyone who joins my operations teams.  In my opinion, it illustrates many best practices for an operations organization.

    Continuous Delivery by Dave Farley and Jez Humble

    This book bridges the gap between good software development practices and getting to continuous delivery.  It made me really think about testing and quality assurance, which in my opinion is one of the most difficult parts of a continuous delivery world to wrap your head around.  While there isn’t one right answer, you should really think about the role testing plays in the overall process.

    The Phoenix Project by Gene Kim, Kevin Behr and George Spafford

    “The Pheonix Project” is a relatively new addition to my library.  While it may sound strange, this book was an absolute page-turner and I really related to it.  It presents many Agile and DevOps concepts via a fictional story that doesn’t seem fictional because we have all been there.  Despite the fact that it is fictional, the concepts presented and techniques are applicable to each of our real-world situations.

    Etsy – Moving Fast at Scale

    While this isn’t actually a book, but a video, to me this is a pivotal piece of information has continued to guide my pursuit of DevOps methodologies and practices, so it holds a special place on my figurative bookshelf.  In the video, the guys at Esty discuss a number of topics including deployment processes, application toggles, metrics and more.  Most importantly, they not only discuss the why but also the how and I find myself frequently referencing this video.

    For me each one of these books or videos has played an important role in pursuit and real-world implementations of DevOps practices.  What are the texts digital or otherwise that you hold dear and/or have influenced you?

  • Closing the Deltas Between Development and Production

    Who hasn’t heard the classic statements “It works on my machine!” or, “Man that would have been easier with real data!”?  In either case, had the development happened in production, problems would have been avoided!  Fortunately, in a DevOps world, the walls between production and non-production get broken down making the reality of developing and testing in production more tangible.  In no particular order, here are some of the things I’ve done to breakdown those walls.

    Develop on and deploy from a single code line.

    Many times we find ourselves in complex processes around deployments and production bug fixes.  For example, the code is branched for release candidates and deployments are done from the branches.  Branching before release by itself isn’t complicated, however, when there is a production bug, engineers find themselves repeating or merging the fix into at least two, if not several, different places.  Repeat that a few times over and, uh oh, now there is a problem because there are many moving parts and questions about what exactly is in production.  Unfortunately, the differences are most likely unintentional.

    Instead, simplify the process and have everyone work off the same code line and have that same code line deployed directly to production.  Now there is only one code base, therefore no question about what code is running in production.  Production isn’t static, so change will still happen after deployment.   Notation is required indicating what version was deployed, so you can still reproduce production.

     Let testing happen on the production database.

    Ok, even I’m not that crazy, but let developers use a copy of the production database.  There are two very important aspects to this.  First, refresh it often.  I suggest daily.  I have had very good luck using EBS snapshots to get a copy of the database on a different server, but if your database is small, good old SQL dumps also will work.  Second, make it safe.  Put some safe guards in place to make sure a change doesn’t accidently get made to the real production database.  Keep your ACLs tight, change the passwords and practice proper database user management.  Lastly, a bonus outcome is that you are constantly testing your database backups and recovery procedures.

    Minimize the steps between development and production.

    Each step, especially the manual ones, between the developer’s machine and production is another point where unexpected changes can be introduced.  It sounds obvious, but the gates code must pass through before entering production must be purposeful and prove their usefulness.  For example, if code must go through a regression, but no regressions are being found, then that step is not useful and just increases the probability of unintended change.  I have been a part of a process where code had to be deployed to “staging” where it was then hooked up to the live production database, but was only to be exercised in a read-only manner.  After doing that for many, many releases and very rarely finding defects before production, we decided to just get better at deploying to production and eliminated the deployment to staging.  In the end, we got faster and with no impacts to quality.  Less can be more.

    Depending on your situation, database size, regulatory compliance, etc. you might to be able to do all of them, but breaking down those walls will yield better code, fewer bugs, happier engineers and fewer sleepless nights to name a few.  So, grab that wrecking ball and break down a few walls.

  • Don’t Be Afraid of the “R” Word

    It seems that many people are afraid of the word refactor.  I think in many cases, if someone admits they need to refactor, it means they either didn’t do their job right.  They either missed requirements, wrote inefficient or buggy code, etc.  Alternatively, there is something fundamentally wrong with the product or system.  The fact of the matter is that sometimes there are big problems and a major refactor/rewrite is required.  I think that is rarely the case, but if embraced by all and done judicially, the outcome can be significantly better and the major refactor avoided.

    It doesn’t matter if it is a feature, deployment script or architectural decision, at some point, everyone must admit they are not clairvoyant and cannot predict every way end-users or systems will behave.  Therefore, there will be missed/new requirements and unforeseen problems.  As such, I very much embrace the philosophy of build for what you know now and adapt as you go.  Typically, that means the output is simpler, it got done faster and enhancements and bugs are easier to address.  That philosophy also means you are constantly in a state of refactoring, or said differently, incorporating new learnings, making individual methods more efficient, improving the readability of the code, filling in missing tests, etc.

    Now, just because we have embraced refactoring and we have noble intentions of, in general, leaving the code better than we find it, that doesn’t give us the license to go crazy and start rewriting everything we see. I think there are right and wrong times to refactor.

    First and foremost, don’t let those yucky bits of code distract you from your real purpose which is to deliver business value by building those new systems, features and fixing bugs.  So instead of just jumping right into refactoring, quickly note the yuck as a TODO in the code, story or in your notepad so you don’t forget about it, but also stay on task.  That said, bad code that is directly in your way is fair game and I even think that bad code near the edges of your task are also fair game.

    Secondly, follow good engineering practices.  Make changes in small, bite size chunks, test and deploy your changes often and communicate them with the team.  In the end, you will end up with a better product, happier end-users and engineers which should all translate into a more successful business.

    If you are one of those clairvoyants, then email me and let’s talk over a beer because I have lots of questions.  Otherwise, if you are a mere mortal like myself, then embrace the “R” word and seek out companies and teams that also embrace it.  At the end of the day or after a few releases, it is pretty impressive to reflect back on how adaptable the team and code can be while solving real problems very quickly.

  • Improving Communication Through Autotmation

    Improving Communication Through Autotmation

    One of the great traits of an organization that adopts DevOps principles or even just an Agile methodology is that everything, and I mean everything, is more dynamic.  While each company will have a different level of fluidity in their processes, Clip Interactive’s source control and deployment processes are very fluid.  As such, we attempt to over communicate when it comes to these parts of the engineering process.

    The primary methods of communication the engineering team and rest of the company have adopted are chat/instant messaging and email.   There are several chat clients out there, all of which have their pros and cons and there is no one right answer except what works best for your team.  At Clip, we have chosen to use HipChat.  Other options include IRC, Flowdock, GTalk, Skype and the list goes on. HipChat was a good option for us because it is cross-platform, supports custom plugins and integrates with a wide variety of tools. 

    In HipChat, each time an engineer pushes code into source control, a build is started, finishes or deployment runs on a QA system a message gets posted to the “Engineering Feed”.  The messages for code pushes include the commit summary and link to the code diffs.  Messages regarding builds and deployments contain a link to their respective outputs.  More significant events, such as deployments to staging and production, get posted to the general chat room where more conversation happens and other non-engineers are more likely to be.  Those notifications are pulled “out of the noise” of the feed.  Additionally, these activities are typically chatted about more before they happened, so they fit more in the flow of conversation.  Last but not least, system notifications, including warnings, problems, recoveries and acknowledgements from Nagios, are sent into the “Operations” room.  To keep everyone informed and spread the knowledge, we post updates and resolutions to issues in the chat room.  All of these messages enable everyone to keep a pulse on the state of the system without being intrusive and should problems arise, to very quickly put together a chronology of events.  Aside from the automated items, there are additional rooms for in-depth feature discussion, production problems and sometimes just for fun.

    Another key form of communication is good old email.  Our build and monitoring systems sends the standard emails and text messages directly to the engineering team.  Each production deployment sends out the list of commit messages, naturally with links to the code diffs.  Since our deployment process is very fluid, this extra notification has gone a long way toward keeping everyone informed on the features being deployed into production.

    Going forward, we intend to add more automated communication around the product management process and system performance metrics.  These automated communications, when added to our normal cadence of meetings, have had a significant impact on getting us closer to the goal of improving transparency in and out of the engineering team, while not making people go out of their way gain value from the tools.