devopsy.kinsta.cloud

Tag: devops

  • DevOps Dirty Little Secrets

    DevOps Dirty Little Secrets

    By all outward appearances DevOps is gaining real velocity within IT shops worldwide, Duh tell you something you don’t know. I have read much on this topic as I am sure you have, but I never seem to read much with hard data results, other than we are doing great with some very broad  metrics. What are the hard numbers as compared to the legacy IT shop?  So my search begins; where in the world are devops metrics in Fortune 500/Enterprise organizations?

    What got me thinking about this were a couple of recent AppDynamics blogs “Measuring your devops success” and   “Quantifying the value of devops”.  Both blogs do an excellent job at breaking down DevOps to the implementation level. Hard data creates metrics that set a baseline with the legacy system that is the critical measurement component, without a baseline there is nothing to compare or measure against. This approach creates a meaningful way to measure the success of executing a DevOps strategy within an IT shop.  The blogs also point other factors not quite as easy to measure, particularly human elements. Many others have written on measuring DevOps, you need to adopt the approach that makes sense for you and your organization.

    At that point I figured it was time to visit Puppet Labs, when you think DevOps Puppet Labs and their survey’s comes to mind. I utilized the 2013 State of DevOps Report; the 2014 report is not available yet. I believe that in order for any new philosophy to be adopted as a true standardized practice, the C-level in Fortune 500 organizations need to embrace it. In my long experience, typically the first sign of acceptance in traditional Fortune 500 organizations is when the financial vertical embraces it.  As an example, Wall Street traders will pay good money to save two milliseconds on a trade.

    The Puppet Labs report numbers demonstrate real velocity in the adoption of DevOps as an accepted business practice. The first takeaways I had from the report were that organizations ship code 30x times faster with 50% fewer failures. The numbers paint a glowing picture for DevOps. But looking beyond the glowing numbers, I see something else. DevOps is at a critical fork in the road and needs to cross the chasm before it loses its luster and becomes just another application development tool, as opposed to a practice.

    Only 17% of the respondents were Managers or C-level, with only 22% of respondent companies having 10,000 employees or more.  It is essential to look at individual Fortune 500 companies IT shops and measure them with hard data before and after the implementation of DevOps.  I would love to read some in depth case studies of the Fortune 500 with hard numbers and metrics; I did not find anything with the legacy baseline data. A huge benefit of DevOps to the Fortune 500 is it will enable them to fail fast as opposed to continuing an implementation that will inevitably fail anyway and consume a huge amount resource in the process. Anyone who has worked for a Fortune company I am sure has witnessed this.

    In truly understanding what DevOps is and is not, certain questions need real answers; Are Enterprise/Fortune 500 companies really doing the same old thing and just need to write better code utilizing some new application development tools, does that make it DevOps or just continuous improvement? Will some IT execs change the organizations title from sysadmin to Devops Admin to protect their silo and make them seem kinder gentler Ops? Are the DevOps tools available today ready for prime time in the Fortune 500 IT environment? Is Devops about technology and metrics or really about culture? We can speculate, but time will tell us the real answers. I am looking for good metrics on what does DevOps do to a Fortune 500 company Not some so called Unicorn that was born with a DevOps spoon in its mouth, but a good old fashioned Wall Street firm who adopts DevOps. Until then call me the DevOps skeptic.

    If you would like say something; email parker@devops.com and don’t forget to comment here.

  • Hacking Your Auditor

    Hacking Your Auditor

    No, not that kind of hacking, give me a little credit…

    Recently I was having a conversation with a consulting friend about DevOps, and he found himself in a bit of a tough situation. The organization he was working with was a pure cloud/DevOps shop, but one that was stepping into having the meet compliance obligations. Needless to say, things weren’t going well with the auditor.

    For those of you who haven’t had to deal with the miasma of audits, one of the strongest requirements, for nearly any regulation/contract/agreement, is strong change management. This is especially true after those Enron guys screwed everything up for the rest of us.

    Change management makes a ton of sense from both an operational and compliance standpoint. I, myself, have blown things up in production but not being smart about my change management. For compliance, it is absolutely essential. If you are dealing with anything with financial or privacy implications, you sure as heck want to get it into a stable, secure, compliant state and ensure that state is kept over time.

    There is a big misconception out there that DevOps-style organizations don’t have change management. That when you push multiple changes per developer per day into production, there is no accountability or control. Now, while you can certainly build things this way, that sure isn’t how I see successful teams doing it. Not even close.

    The problem isn’t that DevOps necessarily skips change management, it’s that it looks different. Much of change management becomes automated, and the fundamental nature of changes themselves is different. But think about how you describe DevOps to your traditional IT-focused friends. Now imagine trying to explain this to an auditor working off set standards that are still figuring out if cloud includes a co-hosted Mac Mini duct taped to a rack.

    Here are some tips on how to both adjust your process, and explain it to the auditors:

    • First, figure out which pieces of your application and operations are in-scope for the regulation. Odds are only financial and privacy related bits are, and you can then apply more-granular controls or logging to these if you need.
    • Make sure your continuous integration tool (if you have one) includes some security checks and logs all updates/changes. If you push right from your code repository to production, get a CI tool.
    • Keep records of all changes as they flow from dev to test to prod (or whatever you use). Again, this should be trivial to automate and dump the records to a secure log repository someplace.
    • Unless your cloud provider tracks all API calls (like CloudTrail does on AWS, although that only works for some services and regions) use an API or web UI proxy like Netflix’s Asgaard. This really is a good security practice anyway, and can actually speed up infrastructure and platform management processes anyway. Keep the logs from this to make the auditor happy.
    • If you SSH into instances, you need to log that activity and dump the logs to a central repository.
    • Make sure you keep revisions of both all code changes and infrastructure automation changes (e.g. your CloudFormation templates, Chef/Puppet scripts/etc.). This is often built-in, but make sure you configure it correctly.
    • Those suggestions now give you logs of all application changes, infrastructure changes, and direct access to instances. These are the things the auditor wants to know exist.
    • Filter them based on your in-scope compliance requirements. Such as payment systems components only.
    • Explain to your auditor that all changes go through an automated approval system that checks the code, tests it (detail the tests), tracks the change, and then moves it into production. That all server logins and infrastructure changes are logged, as is the complete history of the structure of your entire environment. And that you can provide all the documentation, filtered for the pieces within compliance scope.

    This might look like a lot, but odds are very high you are already doing most of this. You keep code revisions, run changes through test suites before they go into production, and track all infrastructure changes. You might not have your own Ops portal/proxy, but you really should for a lot of reasons that have nothing to do with compliance.

    The trick is explaining this to your auditor in their terms. They expect manual checks and records. You need to show them that you still have these checks and balances, but they are automated. You still might need to add some pieces, such as an alert if certain parts of your code are changed (e.g. the file with payment processing or account reconciliation), but these will hopefully be minimal.

    A good DevOps environment actually has a much better compliance infrastructure in place than traditional infrastructure. The automation and template-driven models give us much better control and insight as compared to manually fixing everything, even when we push more day to day changes. The trick is to document this, add a couple pieces if you need it, and then explain it all to someone with… less than refined technical skills.

     

     

  • Logging Wins for Devops and Security

    Logging Wins for Devops and Security

    Its always great when companies can invest in tools that serve multiple groups and purposes.  Ensuring proper setup and use of logs is one toolset that can serve both DevOps and Security.

    Why logs are important to DevOps

    DevOps use  logs primarily  to understand what’s going on inside an application server, database system or the operating system.  For the most part, logs are a resource that helps DevOps diagnose problems and ensure proper working order of systems.  Sometimes logs provide a tracking mechanism that helps DevOps understand what customers are doing in their application.  DevOps folks are intimately familiar with newer and popular logging aggregation services like Splunk, SumoLogic and Loggly.

    Security Loves Logs Too

    The good news is that security teams love logs too.  The use, review and management of logs is so important to the security function of an organization, its even part of the SANS top 20 critical security controls.  From a security standpoint, logs provide a ton of important data for everything from forensic analysis to audit records.

    Recommendations

    Since logs are so important to both DevOps and Security teams, it pays to implement your logging tools in a way that will serve the needs of both teams. Here a few quick tips that will help you set up logging tools in a way that will create a better world for both DevOps and Security.

     

    Centralized Logging & Log Off Site

    Setup all your systems to log to the same centralized location.  If you have all your logs in one place it’s much easier to setup log reviews and alerts.  Its also a good idea to log to multiple locations for security reasons.  Consider logging to a centralized location within your network and also to a remote location.  For example, have all systems log to a central server and also log to a remote service such as SumoLogic, Loggly or Splunk.  That way you have multiple copies of your logs available for forensic purposes.

    Time Sync

    Make sure all your systems are in tight time synchronization.  There is nothing worse than reading your centralized logs to find that a few systems are not properly time synchronized.  NTP is your friend for setting up time sync.  Be sure to use two or more reliable upstream time servers and configure all your servers to peer with each other.

    Applications and Operating System Logs

    Start with the most common base configuration  so that all systems use syslog to capture and distribute logs.  In the beginning, the logs in syslog will probably only be from the operating system and few processes that  are already configured to use syslog.  However, there are a ton of other processes running that should also be logging to your centralized system.  The DevOps group will want to ensure that all applications, even custom developed code, logs important information.  Make sure that your custom apps log enough useful information that problems can be diagnosed when they occur.

    Setup Alerts & Reports

    Nobody wants to be trolling through tons and tons of logs. To prevent this, one of tne of the first things you should do is to set up a basic set of alerts and reports.  Security teams will want to make sure that logs record and alert when users gain administrative access to a system or an application, but there are lots of other events that should need logs and alerts. Spend some time thinking about events that could mean something is not working correctly and needs attention.  Ask the development team to specifically log non-critical events that should be reviewed and incorporate these into a daily report.

    Be Verbose, But With Care

    Crank up the logging levels on all systems but don’t go too far. You want to be careful you aren’t logging too much.  For example, you probably don’t need to run all your systems in debug mode all the time.  You also need to make sure you aren’t logging sensitive information like user passwords or authentication tokens. With logs, there is such a thing as too much.

    If logs are set up correctly, they will provide nearly endless benefits to both the DevOps and Security teams. Now, go forth and log!

  • Config Management & F***ing Shell Scripts

    Config Management & F***ing Shell Scripts

    Joke tech projects pop up all the time. Some, like the lamentable titstare, should never have seen the light of day. Others, like Mark Zuckerberg’s Facemash start out as a joke and grow into something that is anything but. Most come and go without anyone really noticing.

    Whilst their lives are typically short we often learn more from people’s reactions to these projects than we do from their use of them. An excellent example of this popped up recently when a couple of frustrated Chef and Ansible users came up with a simple configuration management alternative, the regrettably named F***ing Shell Scripts (FSS). As they explain it, as in “Why can’t we just use f***ing shell scripts?”.

    Whilst usability issues with configuration management tools – particularly in the Enterprise – is a topic close to my heart, I would not normally have had much more than a passing interest in FSS. What grabbed my attention was the reaction it elicited when posted to Hacker News. With over 150 comments FSS clearly touched a nerve.

    Now to clarify, the authors of the project were quite definitive in assuring the mob that they had created FSS as a joke:

    We wrote it literally in an afternoon, and were giggling like schoolgirls the whole time.

    People either missed this, or chose to ignore it:

    “This is simply brilliant! I’ve dreamt about such tools.”
    – artellectual

    “I’d love to see a tool like yours be brought to full maturity, then I think I’d love to use it.”
    – sdegutis

    It provided a perfect platform for many to voice their frustrations with current tools:

    “Even for an experienced sysadmin, using Puppet and Chef to do even a trivial project (replacing a ~10 line shell script) took a painful couple of days.”
    – mattjaynes

    “In my experience, there’s a huge amount of work involved in just getting Ansible (and other CM systems) to work right in the first place”
    – s_kilk

    “We’re using Chef right now, and evaluating other options. They seem to either be between way too over-engineered, or not flexible enough, but nowhere in between.”
    – sdegutis

    “Keep it simple. puppet/chef have horrendous DSL’s that make it really complicated to reason about.”
    – sp1982

    In the interest of balance I should point out that there were some voices of support amongst the crowd. Ansible in particular won praise for the effort its team has put into simplifying things.

    Overall though I was still blown away with the level of frustration being expressed. Automation tools have been around for years now; CFEngine, the grandfather of the modern tools was created in 1993. Despite this, a tongue in cheek project suggesting ditching them all in favor of the technology they set out to replace (or at least abstract away), shell scripts, is considered credible. For me this is akin to people ditching virtualization for physical boxes. It just doesn’t make sense.

    Unless of course something is still fundamentally broken in the approach the current set of tools is taking; that they are still too hard; that they still fail to address key prerequisites like visibility and accountability. I believe that this is the case because I’ve experienced it first hand; I’ve seen it a hundred times in businesses I’ve talked to, and threads like the Hacker News one mentioned above leave little room for alternative interpretations. Heck, we’ve based our product GuardRail on this entire premise; these are not empty words on my part.

    Incremental improvements such as those offered by Ansible are a good thing, but what people seem to be searching for is a step change. As things stand this is less likely to come from the current batch of tools, and far more likely to come in the form of a new approach. Docker anyone?

  • What “Programmable” Means

    What “Programmable” Means

    One of the most often used words these days to describe emerging architectures and technology is (drum roll please) “programmable.”

    Now, from my used-to-be-a-developer-and-architect-turned-infrastructure-geek perspective, I know what that means. It means I get to write code. But to what purpose that code is put to is a question that’s rarely answered. So today we’re going to answer it because, well, my used-to-be-a-developer-and-architect-turned-infrastructure-geek persona likes to talk about code.

    When you hear the term programmable used in the context of Software-Defined Networks (SDN) they’re talking about the same things that “programmable” means in the context of Devops: programmatic control of both the data and the control path.

    The easiest way to visualize the difference between these impressive sounding terms is to recognize that the data path is inside and the control path is outside. 

    programmabilty-illustratedWhat that means is if you’re interacting with the data path, you are participating in it. You can inspect it, evaluate it, and transform it and, perhaps more importantly, you can make decisions based on data in flight. That’s powerful for the network, because a variety of Devops patterns are based on this very concept and require the network for implementation. Consider, for example, the implementation of an A/B testing pattern. The basic premise of the pattern is that somehow the identity of a user is known and, based on their membership in a defined group (beta testing customers, roles within the organization, volunteers from the community, etc…) they are routed to different versions of the same application to determine how differences in the UI impact behavior.

    Notice the term routing, which implies it is a function of the network. The reason programmability is critical is two-fold: one, you must be able to identify the user as belonging to a specific group and two, you must be able to dynamically route based on that information. The former, identification, could ostensibly be accomplished using fixed lists IFF (If and Only If) it is possible to map a user to some variable readily accessible to the network, which traditionally means things like IP and MAC addresses and TCP ports. Most folks will immediately recognize that today this isn’t enough (and actually it hasn’t been enough since the days of the mega-proxy, but I digress). The information that’s needed is much higher in the networking stack at the application layer, aka in the OSI model as Layer 7. Which means you’re probably going to have to dig into HTTP headers and maybe the payload to get the data you want. That requires programmability in the data path, inside. 

    To really make the pattern scale (and to make it flexible enough to support a phased approach to beta testing) you probably don’t want group membership  to be a static, fixed thing. So now you’re going to want to extend your implementation to include making a call to a database or identity store (LDAP, AD, etc…) to get current information. That requires programmability in the data path, inside. 

    So what programmability in the data path means is the ability to use code to direct the behavior of some “thing” in the network, be it an application proxy or a load balancer or what have you. You can’t easily do this through configuration-based proxies because they don’t allow for the execution of the domain logic  necessary. There are a limited number of interactions you can achieve with configuration-based systems because of the logic they’re hard-coded to execute,and most of them don’t include calling out to a data store to look-up arbitrary information.

    Data path programmability = Code in Flight. So you now you know where the name of the blog came from. This developer-and-architect-turned-infrastructure-geek finds the intersection between programmability and networking irresistible.

    Okay, so that’s data path programmability. What about control path programmability?

    Well, that’s the easy part. Control path programmability is on the outside. It’s the APIs that enable automation and orchestration of provisioning and configuration. Control path programmability is how you integrate infrastructure into command control systems like OpenStack, Puppet and Chef. Infrastructure comes with varying degrees of API-enablement, some extremely granular and others less so, depending on the particular function and maturity of the vendor.

    So what you can do with the control path depends on the API, but in general it enables things like provisioning new services – think configuring a new virtual server in Apache and all the settings that accompany it – or start up and shut down functions that enable rolling maintenance across a set of devices or servers to avoid downtime.

    But all control path programmability is on the outside, enabling integration and management without having to learn a CLI or configuration file syntax.

    SDN and Devops use both programmability to reach relatively the same goals: scale of operations, rapid provisioning of services, to decrease the possibility of human error (the fat finger factor) and to introduce flexibility into systems that were previously bound to fixed processing and configuration-based paradigms.

    Programmability in the network is going to enable a variety of new patterns, processes and services and as traditional infrastructure becomes more and more programmable through the adoption of architectural approaches like SDN, Devops is going to see more and more opportunity to apply its maturing expertise in leveraging programmability to a much wider audience.

     

  • Follow Friday, DevOps Twitter Accounts to Follow

    Follow Friday, DevOps Twitter Accounts to Follow

    twitterdevopsAs part of doing my research for this blog I have been looking around for a good list of DevOps people to follow. I have found a small list here and there, but unlike in the infosec community where we had the @securitytwits, there is no analogue to that in DevOps.

    So with a tip of the hat to Bill Brenner of Akamai (and formerly of CSO.com) I am going to start a new tradition here on devopsy.kinsta.cloud.  We will have our own Follow Friday: DevOps Peeps to follow on Twitter.

    I am starting with a small list that I actually found on a Puppet Labs blog. I will add to this list every week and also start a Twitter list of them.  You are already following most of these folks, but you have to start somewhere.

    If you would like to be added to the list or have a suggestion for the list please leave it in a comment or email me at parker@devops.com.

     @patrickdebois: @RealGeneKim: @botchagalupe: @jezhumble; @garethr; @kartar; @KrisBuytaert; @jordansissel; @agentdero; @allspaw; @monkchips: @DEVOPS_BORAT; @lnxchk: @damonedwards;  @lusis; @obfuscurity; @shanley: @adrianco; @scriptrock; @jumpcloud; @sd_architect; @govictorops @SoftLayer @docker; @chef;

  • Don’t let automation be the tail wagging the dog

    Don’t let automation be the tail wagging the dog

    There is quite a bit of talk right now about DevOps and how important automation is.  This could be compared to GM in the 1980s when the company tried to transform the way they built cars through automation and robot.  In the early 1980’s GM spent $90B dollars in an attempt to speed up the way cars were delivered to the market.  This failed with famous stories of robots painting each other and others welding doors shut.  Ultimately this project failed and the robots were scraped.  It was later reported that the money lost on the project could have been spent to acquire both Toyota and Nissan at the time.

    Automation dominates the conversation when it comes to DevOps.  If you do not learn how to build cookbooks you are not doing DevOps.  I fear this is going to set companies trying to transform IT off on the wrong track, focused on the outcome and not on the scientific process to get there.    Alan Sharp-Paul recently wrote that you cannot automate what you don’t understand.  The best way to understand it is in two key ways, change and performance.

    Let Science set you free

    This was the secret that Toyota used to dominate the automobile industry.  They were able to transform all of their employees into little scientist.  Constantly tinkering to squeeze our inefficiencies in the system.  Every worker was empowered to think how things could be done better.

    At the heart of the agile and DevOps processes is the practical application of the scientific method.  Think back to your 8th grade science fairs.  No not the volcanos but the ones that won best of show.  These had a clear hypothesis that stated what they were going to try and prove.  They were able to understand what they were going to change and measure the difference between the before and after.  They could clearly state what changed and if it was a successful experiment or a failure.

    This thinking is key for companies trying to transform to a DevOps IT organization.  If a company does not understand the experiments that are running and what the outcomes should be then automation will just make the problem worse. That is why change management and performance management are foundational for transforming to a DevOps culture.

    Tracking your experiments

    Any good scientist knows exactly what he is working on.  He knows every experiment that is running anticipating the outcome.  He knows all the details about each change that has been made to see how they impact the outcome.  This is the same way that IT should be tracking every change happening.

    Every change done in IT is a scientific experiment running. Tracking every change allows for understanding of the outcome of changes within the application.  Developers and Operations will be more informed about the cause and effect of their changes.  There are multiple different ways that you can track changes but the key is that you are doing it very well before you start to apply automation.

    Measuring the outcome

    There is no need to track the experiments if there are not being measured.  A scientist is very deliberate in how they measure the outcomes.  Good enough just does not get this done.  This is beyond just monitoring.  Monitoring in a lot of cases is after the fact, in production.  This is about measuring the difference for every experiment regardless of where it is being tests.  This creates the need to test changes all the way to when the first functional tests are done, measuring the changes of every code update.  Following it through all of testing and to production.  Understanding how each experiment impacts the system as a whole.

    Efficiency through automation  

    Once there is good understanding of all the experiments and a standard platform for measuring the outcome then automation starts to return exponential benefit.  Because every experiment it known the lower priority ones can be identified and automated.  Risk is better assessed.  Processes can start to speed up because there is visibility and accountability.  Applications are no longer classified as high risk or low risk only change is classified that way.  Determining what can be automated just becomes another experiment that is tried and measured for success and and failure.  Automation becomes just another outcome of DevOps to the business and not the route to achieve it.

  • Can you appreciate DevOps, if you can’t exactly say what it is?

    I recently had a conversation with some SOX IT auditors and was asked what I meant when I said we were a DevOps shop. I stumbled a little, torn somewhere between an overly simplistic answer and a desire to baptize the unholy. I started by talking about collaboration between the developers and the admins. Mentioning that many of our developers are on call and have sudo access raised some eyebrows. Using Puppet made them pretty happy.

    Thinking back on it, what I realize is that for me devops isn’t a strictly defined thing, it’s a mix of operational changes that draw on the best of any tool, without prejudice. We do what works. We do not do what does not work. If we are doing something, and it ceases to work, we stop doing it. I don’t know if that’s devops, but it’s what we do. And it works. So we’re going to keep doing it.

    Configuration as code

    This is not news, central and key to successful devops is controlling system configuration and change management with code, not with disparate manually managed text files. Here’s the thing, you cannot operate at scale without it. You also cannot change direction, or iterate through rapid change without it. You can rapidly, accurately, and verifiably push changes, stand up environments, or demonstrate a fact to an auditor with just a few keystrokes.

    To be sure, it’s a helluva lot easier to start a business on these principles than to migrate after your business is a real thing. However it’s worth the effort.

    Sysadmins are our friends, Developers are on call

    Most places I’ve worked, neither of those statements were strictly true. At Craftsy, both are. The relationship we foster is one of open collaboration. I’ve never been somewhere where the sysadmins were jerks enough to really justify the animosity they received from dev, and I’ve never met a developer so hopelessly irresponsible to really justify the vitriolic mistrust dealt out by the SA team.

    Give your developers sudo and get them in the oncall rotation. Now, they share a burden with your SA group, and a responsibility to the systems their code runs on. Figure out a sensible rotation and escalation, but without that shared responsibility you will always struggle with trust between your dev and your ops.

    SaaS, Paas, IaaS, CotS, Cloud, OnPrem, Whatev

    The punchline is “all of the above”. Never let yourself get married to a particular way of doing things. We run stuff in the cloud. We run stuff locally. We buy software. We rent software. We rent services. We borrow code. We cloud. And whatever comes up next, we’ll try it one way, whichever way seems to make the most sense. If it works, we’ll keep doing it that way.

    I’ve heard some reasonable objections to Cloud/*aaS based on security concerns. Safely extending the trust relationship of your LAN isn’t easy. It isn’t impossible either. Tools like the AWS VPC offering, or SSO/IAM systems offer some very flexible approaches to open your environment and leverage the best tool, regardless of how it’s made available.

    Do what works

    I’m not sure exactly what the SOX guys thought of my ideas around devops. IT, much like accounting, is an area that traditionally is slow to adopt new approaches. It’s a counterintuitive truth that an industry so deeply engaged with technology is simultaneously so awkwardly regimented in its ways. The standard answer, that change has to come slowly to maintain stability is unsatisfying. Indeed, it belies the fundamental failings of the old methods.

    DevOps sets the table for you to achieve stability in any environment precisely by engaging deeply the technology around you.

  • PEBKAC Avoidance

    PEBKAC Avoidance

    We’ve all said it. We’ve all done it. We’ve all shook our heads at it. PEBKAC. Problem exists between keyboard and chair.

    User error.

    While generally applied to the end-user community – those folks who are considered technical neophytes by IT professionals – it can and should also be applied to those of us who have, at least once (admit it, come on, I know you’ve done it) fat fingered a configuration on a web server, a switch, a router, or some other network or application service.

    It’s okay. We’ve all been there – head down on the keyboard, a litany of words we wouldn’t use in front of our mothers streaming from our lips between enumerations of how long we’ve sat at our desk looking for the problem. Mine was a misconfiguration of route metrics in the now long gone Network Computing lab that sent traffic from one side of the lab over a simulated T1 and back over the 100 Mbps link. When you’re trying to simulate a WAN environment and put WAN optimization solutions on either end, this is highly problematic and tends to, well, not work. Not work at all. If it wasn’t for the astute observation of my lab partner,  I’d probably still be at my desk hunting for the problem.

    The point is, we’ve all had our PEBKAC moments. We know they happen, and avoiding them is a very real goal of devops in general. We want consistent, repeatable and successful deployment processes in part to avoid the time and effort it takes to troubleshoot some minute configuration error in the lengthy and often complicated data path over which application traffic must traverse in our domain.

    And it’s no longer just about downtime, which is costly and frustrating for everyone involved. It’s also about security; security of the network, the applications, and data.

    Tufin Technologies, in a 2010 survey of 100 registered DEF CON 18 attendees, found that “73 percent of hackers came across a misconfigured network more than three quarters of the time – which, according to 76 percent of the sample, was the easiest IT resource to exploit.” [1]

    With more and more layers of complexity added to the network in an attempt to simplify the network (yes, how’s THAT for irony?), there are more and more opportunities to experience the unique pain of PEBKAC every time a new service is provisioned. Consider the math:

    A typical data center has over 500 servers, each running a hypervisor. Each hypervisor carries an average of 20 virtual workloads. Each workload requires at least 5 separate network attributes to be configured. That’s a total of 250,000 opportunities to make a mistake. At even an error rate of less than 1 percent, that’s still 250 problems you’ll need to track down.

    This is, in part, why Devops exists. To enable the development and use of reliable processes and methods for provisioning services. That means configuration is happening under the covers, unless you’re already living in rainbow unicorn land where services configure themselves automatically. I didn’t think so.

    There’s a reason why typists are measured based not just on speed but on the number of errors made while typing. It’s not just about the speed, it’s about the accuracy because eventually, you’re going to miss the mistake you made and it’s going to be sent out in a corporate wide e-mail and … well, you can peruse the Internet at your leisure to see how well those turn out. The same is true for deployments and provisioning. It’s not just about how fast you can get a service up and running, it’s about how fast you can get that service up and running correctly.

    Devops should focus as much on accuracy as it does speed. You’ll save more time avoiding misconfiguration and the associated long hours of troubleshooting by simply slowing down a few MPH (or WPM) and ensuring that you’ve got it right the first time.

     

    [1] http://www.continuitycentral.com/news05331.html, http://www.darkreading.com/perimeter/misconfigured-networks-are-easiest-prey/227200159

  • Zen and The Art Of Infrastructure Maintenance

    Zen and The Art Of Infrastructure Maintenance

    There’s a famous passage I’ve quoted from time to time when I see someone “stuck” with a problem, flailing potential fixes and solutions wildly in hope that one of them does the trick.

    “Assembly of Japanese bicycle require great peace of mind.”

    Just like that. Improper grammar and all, quoted verbatim from Robert Pirsig’s philosophical classic Zen and the Art of Motorcycle Maintenance. The quote is in reference to the first printed words on a set of technical instructions, which the narrator discusses with friends and colleagues on a long-distance motorcycle trip.

    And the response I get is the same every time, likely the same response you’re having right now…

    What in the world are you talking about?

    Well good, now I have your attention. And I’d be willing to bet that attention is on 1) the ludicrous loss of translation from Japanese to English and 2) debating whether or not something as seemingly superficial as “peace of mind” is really required to assemble a bicycle.

    Language barrier aside, let’s abstract away from modes of bipedal transport into a more generic “machine”. Do we really require “peace of mind” to assemble or create a machine?

    cory2

    Reliable? Maintainable? Usable?

    We certainly all want to receive peace of mind from our machines. We want them to do the job they were created to do without fail so we can move our attention onto other things.

    What words do we use to describe such machines that give us this peace of mind? Reliable? Maintainable? Usable? When we think of the reliability, maintainability or usability of a machine, we’re really just describing a machine that can produce peace of mind. I’d argue if we want that machine to produce peace of mind, then it must be created under peace of mind.

    Why is that? If we, the creators of machines, are in a state of peace of mind when we start our endeavor, we’re highly motivated to do that which will continue our state of peace of mind. We don’t want to lose this state. Since if the machine does break, we the creators will be forced to lose our peace of mind and fix it, we’re motivated to create a machine that’s resilient against failure and resilient against our loss of peace of mind.

    Without this key precondition to creation, we’re likely to build our existing personal problems right into the machine itself! Under time pressure? We’ll cut corners to make deadlines. Dealing with a different machine breaking? Likely to gloss over a key failure condition in the machine we’re actively building so we can fight a fire elsewhere. Worried about the problems of today? Then I can guarantee there’s no thought going into a machine that will be resilient to the problems of tomorrow.

    Quality creations produce serene creators as much as serene creators produce quality creations.

    Those in a state of serenity will continue to create that which maintains the state of serenity or peace of mind; those that are not in a state of serenity are unable to escape a vicious cycle of build-fix-build-fix, never achieving peace of mind. Ultimately, the serenity of the creator is the key test for the quality of creation.

    And ultimately, this passion for peace of mind from our machines is what you’ll find when you meet the folks in the DevOps community. We’re guided by a desire to maintain our peace of mind — for ourselves, for our families and for the folks that count on us the most: our customers.

    If the machine produces tranquility, it’s right. If it disturbs you, it’s wrong until either the machine or your mind is changed.

    The material object of observation, the bicycle or [in this case, the global infrastructure], can’t be right or wrong. Molecules are molecules. They don’t have any ethical codes to follow except those people give them. The test of the machine is the satisfaction it gives you. There isn’t any other test. If the machine produces tranquility it’s right. If it disturbs you it’s wrong until either the machine or your mind is changed. The test of the machine is always your own mind. There isn’t any other test.
    – Robert Pirsig

    So the next time you find yourself struggling with a problem, take a lesson from Robert Pirsig: step back, achieve peace of mind, combat that which threatens peace of mind and strive to maintain peace of mind.

    You’ll be a better (and happier) creator for it.