devopsy.kinsta.cloud

Tag: devops

  • Technology’s effect on religion

    Technology’s effect on religion

    Today being Good Friday, with Easter being Sunday and in the midst of Passover; it is a time for reflection. Instead of thinking about DevOps, today I ponder life and religion. What comes to mind is the future of religion in these technological times. The number of Americans with a religious affiliation continues to decline, less and less of us make an appearance at a Synagogue, Church or any house of worship for that matter. In my childhood we would at the very least make an appearance a few holidays a year, notably at the annual pancake breakfast.

    From the calculator to the computer, from mobile devices and the Internet, advances in technology are creating monumental changes in our lives and in turn the way religion is practiced.  The universal acceptance of mobile devices, like the iPhone and iPad, is provoking a profound social change. The dependence on these devices is eliminating the individuals need for traditional religion and going to traditional houses of worship. The problem is, without the traditional religious brick and mortar upbringing, the youth of today forgo a key element of the religious experience. Religion was never just about a supreme being, the social gathering and interaction of those with similar beliefs and values is also a critical part of the experience. Religion is losing the youth.

    It seems there are now thousands of apps which allow parishioners to bring Bible quotes, Torah-chanting practice and Buddhist prayer wheels anywhere, anytime day or night. My concern is how many of our youth are downloading religious apps as opposed to angry birds. The challenge is how to mobilize religion while still achieving all the religious personal support and socialization benefits that are realized from real life interaction in attending a place of worship. Is the virtual congregation a realistic way to gather people and practice religion? I have a hard time thinking the young will embrace religion without the real life experience which creates the feeling of being at peace with oneself.

    The virtual congregation will be effective fill in for those who have already experienced religion in the traditional sense. Worshippers who cannot find their way to their house of worship for services will pray online with their fellow believers via the latest technology, confident that the other worshipers are doing the same on their devices. How powerful that will be is the question, it has been relatively effective via Television. But those are already followers seeking religious celebrities, not new believers.

    There is no denying religious leaders who conduct services via technology to a geographically wide spread audience will notice a significant difference in how worshippers experience the event. Religion is very much an emotional event and technology can be a non starter for the holy rollers. Gathering around technology in smaller groups may be an answer.

    The religious apathy of the young produces a serious problem for religious leaders looking to grow shrinking congregations. The lack of formal religious upbringing has lost many of the young and younger worshippers that do exist expect any religious event  to embrace technology use. Device multitasking has become such a pervasive part of their life that quiet,  hard cover book based religious ceremonies seem awkward and therefore not compelling.

    Technology may represent the end of traditional religious worship, and the beginning of a new age of worship, the form of which no one can reasonably predict or control. The bottom line is if you look at technology and say how does this change people going to house of worship? You miss the point, because religion is a much deeper set of beliefs. The question is without new young followers, what does the future hold for organized religions?

    If you would like say something; email parker@devops.com and don’t forget to comment here.

     

  • DevOps adoption can succeed where ITIL failed, if we let it

    DevOps adoption can succeed where ITIL failed, if we let it

    When I look at the future of DevOps adoption and acceleration I contrast it with what I feel was the failure of ITIL to really move beyond large enterprise IT shops. I am of the firm belief that DevOps has the legs to succeed and soar where ITIL sort of crashed and burned. OK if not crashed and burned, at least sputtered out. To understand why I feel this way, let me share a little personal history with you.

    ITIL demands you to come to IT

    I remember my first ITIL project. I was working at a large retailer and I was asked to help our IT department understand what this ITIL thing was and to help drive adoption of some of ITIL’s basic concepts. First on the list was to get a service catalog up and running. In the space of just a few weeks, I realized the main barriers to adoption and helped our IT executive team understand that getting IT consumers to understand, much less embrace an IT-centric taxonomy such as ITIL was not going to be an easy task.

    This scenario points directly at one of the big reasons ITIL failed to achieve critical mass (i.e., adoption of the model outside of large enterprise IT) and also reveals why DevOps has a much better chance of success of achieving critical mass (as long as we ourselves don’t mess it up).

    Mind you I’m not saying that ITIL is dead or that it has nothing to teach IT shops. I am saying that ITIL alone has not really changed much of anything inside of IT shops, let alone improved the reputation of IT with it’s consumers and customers.

    When you contrast ITIL and DevOps several things really set the two approaches apart:

    • ITIL makes it easy to blame the business for not talking IT’s language while DevOps continuously seeks to meet people on whatever terms they are capable of communicating in. DevOps inspires, while ITIL deflates
    • DevOps is centered around improving the lives of IT workers everywhere while also improving bottom lines at the companies they work for. ITIL is centered around improving stability in the most efficient manner possible. I don’t know about you, but I just don’t see a room full of people, much less an entire organization getting excited or inspired by the concept of efficient stability. The only people likely to get sustainably excited by “efficient stability” are the accountants who can now start to push cost (i.e., people) out the door.
    • ITIL at its core is reductive (or at least allows for reductive methods and approaches to grow unchecked). For those uninitiated in philosophy or how language influences our ideas and behavior, reductionism is when people essentially believe that a complex system is nothing more than the sum of its parts. This reductive approach at the core of the ITIL movement is what reinforced the division between Dev and Ops – because the concept of team chemistry has no place in ITIL given that the organization is nothing more than a collection of faceless workers who play out a prescribed role.

    Reductive goals and processes, essentially reduce people and their efforts to numbers. The act of using quantitative targets as goals in large initiatives is a dangerous practice in general and grows more dangerous in larger groups and organizations – case in point the recent GM ignition failures that resulted in 13 deaths in an effort to save .57 cents per car affected by a defect.

    Before you get excited, I’m not proposing that ITIL will kill people. I am stating that allowing an organization to be led by reductive philosophy allows a whole bunch of unintended consequences to come into play and often leads to holding dollars as singularly more important than anything else; sometimes even human life.

    Whether or not you agree with my point that efficiency driven quantitative metrics are not inspiring, it’s kind of hard to argue with 40 years of scientifically proven fact that when “a measure becomes a target, it ceases to be a good measure”.

    DevOps shops “live to serve”, while ITIL shops “serve to live”

    DevOps contains a concept of craft that extends beyond efficiency. When a problem arrives in a DevOps shop (or at least one that really embraces the cultural ethos), the staff looks at the real resolution of that defect as a matter of personal pride. This pride is not a statement of “pride in self” however, but a statement in “pride in craft”. To hammer this contrast home, does ITIL by itself even have a deep cultural bias at all, much less an ethos?

    This perspective is what allows DevOps shops to act in a manner that transcends the checkbox approaches that pervade formalist driven shops that believe in “adherence to process” above all rather than the “results the process is intended to achieve”. If you think I’m wrong, go ask your business clients how they feel about the DMV-like experiences they get when trying to get a problem solved.

    This perspective is also why DevOps forms strategic pillars in automation and performance. Performance and automation efforts are not only about the manufacturing of time. Automation isn’t only a method of driving towards error free processes, or even a liberator of individuals from mundane tedium. Driving highly performant code and infrastructure isn’t only about conservation of compute resources.

    Well implemented automation routines and highly performant applications are also a deeply creative expression of both development and operations practitioners. The same sense of artistry and elegance that was once reserved for just the software developer has been opened up to all IT practitioners. With the arrival of DevOps, being a great IT professional is not out of reach of any IT practitioner no matter their discipline.

    We just can’t afford to mess this one up. In order not to, here is some practical advice:

    As you go forth in creating or playing your part in your organization’s DevOps journey, look out for these pitfalls and pivot back to the center.

    1. Reductionism – Don’t let numbers be the mission. This doesn’t mean don’t have numbers in your goals. It only means to make sure that your number oriented goals have a basis in making things better for people. When you hear someone say “We need to get this deployment routine under 4 hours!”, make sure that someone reminds everyone that the reason to get it under 4 hours is to allow employees to have a reasonably well balanced life outside of work. Off hours deployments that last four hours, when things go right, should not be a target that anyone is proud of. Getting people home to their families and allowing people to be able to keep a healthy sleep schedule is goal worth shooting for.
    2.  Formalism – Worshipping the shovel rather than the hole it digs is an easy trap to fall into for people whose jobs revolve around buiding and using tools. Remember that processes and tools are mechanisms intended to guide novices and intermediates through decisions. When you see someone turning a process or a tool into a sword and shield for working with others, make sure you remind them that results are better than processes.

    People Over Process

    Neither processes nor tools solve problems; People do. Processes and tools don’t create business opportunities by themselves; People do. Isn’t that why we chose this profession in the first place? To be useful in solving problems and to create opportunities to make things better while allowing people (ourselves included) to have a good standard of living.

    Now go DevOps practitioners! Go kick some ass and make the internet awesome!

  • Programmability in the Network: Stop a Bleeding Heart…

    Programmability in the Network: Stop a Bleeding Heart…

    heartbleedIt is not often the case that a security vulnerability can get the entire Internet talking. And not just the security community on the Internet, but everyone. End-users and IT alike are looking for answers and trying to mitigate Heartbleed. It has its own web site and logo. It’s that big of a deal.

    Many service providers have already patched their systems, but there are a whole lot of sites on the Internet and it’s estimated that a significant number of them are vulnerable.

    Netcraft notes that Heartbleed, based on OpenSSL, “affects around 17% of SSL web servers which use certificates issued by trusted certificate authorities.”  One of the most trusted sources of data regarding web server software in use today, Netcraft’s “most recent SSL Survey found that the heartbeat extension was enabled on 17.5% of SSL sites, accounting for around half a million certificates issued by trusted certificate authorities.”

    But it’s not just server software that may be vulnerable. Any device in the data path that terminates SSL and is based on OpenSSL could be vulnerable. Any “thing” that might use OpenSSL to enable secure communication and accepts connections could be vulnerable. In other words, hearts are bleeding all over the Internet right now, and it could take quite a while for the Internet in general to plug the holes.

    The hole itself is fairly simple – it takes advantage of the mishandling of what seems like an innocuous meta-protocol that runs atop TLS and DTLS that consists of two message types: HeartbeatRequest and HeartbeatResponse. It’s used to “ping” the other party in a secure connection to make sure the peer is alive. It can be used to offset idle times that might otherwise cause the connection to time out.

    It can also, thanks to a missing bounds check on the message structure, arbitrarily and completely silently grab 64 kilobytes of the server’s memory. That memory might include sensitive things like the private key used to encrypt communications or passwords or pieces of critical documents.

    heartbeat-memory

    Whatever might be in memory when the vulnerable software goes to grab it is leaving the building faster than Elvis – and without a trace. There’s no logs, no evidence of the theft, nothing. It’s clean and remotely executable by anyone on the Internet. That may explain the attention it’s currently getting.

    The answer to any vulnerability is ultimately to patch it. Depending on how many servers or systems are impacted, that could take some organizations days – assuming patches are available immediately. In the meantime, the heart is still bleeding and the only way to prevent more information from leaking is to shut off the hose.

    Except it’s not the only way.

    Programmability in the data path can mitigate this, now. Right now. Like in the next five minutes.

    Let’s review programmability in the network for a brief moment: you have, in the data path, a device. That device (or software) is capable of programmatically intercepting, inspecting, and transforming the data that’s traversing it. That means such a system can intercept a request to initiate SSL/TLS traffic, inspect it and, if it finds a request to include Heartbeat support it can reject the connection.

    before-after-heartbleed

    Interestingly, one of the primary motivators of early SDN architectures was the ability to support new or experimental protocols by inserting them into the data path in the network. While that’s a driver for academics, a more realistic driver for businesses is the ability to interact with protocols in flight and “fix” them – especially when the “fix” involves preventing the exploitation of a vulnerability.

    Programmability in the network enables this kind of rapid response to threats. Being able to deploy a solution very quickly to prevent further data leakage provides protection for both the organization and those who interact with its properties (employees, partners and customers). It also offers time for IT to patch and update all impacted systems without remaining vulnerable or taking the drastic step of going completely dark until the situation is resolved.

    By enabling a view of “infrastructure as code”, programmability not only enables a more software-oriented approach to deploying services, but also a software-oriented approach to quickly creating and deploying an appropriate response to emerging threats.

    A devops approach can facilitate this capability, and push protection into the network as code.

     

  • Automation versus Orchestration

    Automation versus Orchestration

    Yes, Virginia, there is a difference.

    One of the things devops practitioners are tasked with is the provisioning and configuration of all sorts of infrastructure.  Application servers, web servers, load balancers, proxies and database servers are among the lengthy (and no doubt growing) list of “boxes” devops needs to get up and running to support just about any given application today. One of the key value propositions of a devops approach to operations is that it can reduce the time it takes to get applications to market by getting them up and running in production faster. That’s increasingly important as we move into the Era of Things, driven by extreme connectivity of everything, because all those “things” need are going to need to be talking to applications on the back side.

    So speed is of the essence, but not at the cost of accuracy. We’re encouraged, then, to automate tasks and orchestrate processes to achieve that speed. The side effect of codification of tasks is higher accuracy (aka fewer mistakes).

    So far so good.

    But you may notice that I have (and will continue to) clearly delineate between “automate” and “orchestrate”, between “task” and “process”.

    The reason for that is that, simply, they are not the same thing. We should strive not to confuse these two because while the two share some benefits, there are other benefits that are achievable only by one or the other – or the combination thereof.

    Process optimization, for example, can’t be achieved by simple automation. Automation is concerned with a single task – launching a web server, configuring a web server, stopping a service. Orchestration, however, is concerned with automating, if you will, the execution of a workflow – of a process. A provisioning process may be comprised of multiple tasks and involve multiple systems. An “application” is not just a single server, after all, it’s likely several servers – web, app and database in a traditional three-tier architecture.

    The goal of orchestration is not just to automatically execute a service, which brings speed to the deployment process and gets applications into production faster. It also affords an opportunity to streamline – to optimize – those processes for even greater gains in deployment velocity.

    One of the simplest ways to optimize a process is to eliminate repetitive steps. This entry on Vagrant explains it fairly well :

    “We could just SSH in and install a webserver and be on our way, but then every person who used Vagrant would have to do the same thing. Instead, Vagrant has built-in support for automated provisioning. Using this feature, Vagrant will automatically install software when you vagrant up so that the guest machine can be repeatably created and ready-to-use.” [emphasis mine]

    — Vagrant Docs “Getting Started | Provisioning”

    Now certainly the step of installing a web server on a new box with vagrant is hardly all that time consuming, it’s just a single line of text. But every line of text must be manually typed, and that introduces the possibility of fat fingering the command, which either consumes time in fixing it or introduces an error into the system. Neither are desirable results.

    automate-orchestrate

    Ensuring that the process is repeatable and orchestrated by automating all the tasks that comprise that process an optimize it by eliminating redundancy. That results in greater deployment velocity.

    Recognizing that processes should be comprised of composable tasks offers a higher degree of reuse across operations. The task of provisioning and configuration a web server with the appropriate base software means consistency, aka standardization, across all applications. Processes that take advantage of standardization result in greater infrastructure stability, another benefit often cited by organizations that have adopted a devops approach to operations.

    iweek-devops-stability

    The key here is to recognize the difference between automation and orchestration and how they relate to one another. Automation is about codifying tasks and orchestration is about codifying processes. Orchestration takes advantage of automation by reusing these basic building blocks.

    There is a difference, and done right, the two play off one another to result in streamlined, more accurate execution of operational processes that speed up successful application deployments.

  • DevOps Needs a Tsunami To Jump The Chasm

    DevOps Needs a Tsunami To Jump The Chasm

    All start up business endeavors must go through the phase of crossing the chasm.  Most of those business endeavors usually fall prey to the chasm. The chasm is this black hole that everyone claims to understand but no one truly does. We have all seen businesses cross the chasm or fail to cross the chasm and fall into the black hole. But no one seems to understand why and how that crossing happens. The bestselling book by Geoffrey A. Moore titled “Crossing the Chasm,” is all about the heart and soul required to get early stage technology across the chasm, from early adopters to mainstream customers.

    There is a big difference between people who are enthusiastic to try leading edge technologies and the rest of the inhabitants, who tend to be much more conservative. The reasons that people spend time trying to figure out how to cross or jump, if you will, the chasm is because that black hole is what stands between them and making their fortune. The early adopters are just a small sample of the market and there is no money in building a business or technology around them. To make real money, you need to cross over into the mainstream.

    In Geoff’s model, the innovators are all about technology.  You know those people, when a new iphone is released, they spend days standing on line to get latest and greatest new technology.  The early adopters are the visionaries that are always on the bleeding edge. The mainstream are the pragmatists, the people who practice risk avoidance. These pragmatists are the key to jumping the black hole, they move the needle.

    DevOps first order of business was getting on the curve and to that end DevOps has evangelists. But, understand that what worked in converting these early adopters probably won’t work for the pragmatists. These DevOps evangelists and visionaries want DevOps power tools, as opposed to the pragmatists who simply want to push the button and say “that was easy”.

    By all appearances DevOps has been accepted by the early adoption set. Just how does DevOps cross the chasm? Jumping the black hole after all can be quite a daunting task. To do so DevOps has to create what I refer to as the tsunami effect.

    The Tsunami effect is where DevOps gains acceptance in multiple industries or markets in what seems like a gold rush, it’s here that fortunes are made. More accurately, it’s where cult communities transform overnight into mainstream communities.  Many technologies simply create a single tornado which remains in a silo forever never gaining enough momentum to really emerge. Another common path what I call an earthquake, there is a lot of hype but it tends to fizzle out quickly, if not tended to properly. Right now DevOps has all the appearances of an earthquake, Devops evangelists have created a lot of buzz, but acceptance beyond a set of power tools is still a huge question.

    Considering DevOps is in the earthquake phase, as opposed to the tornado phase, DevOps cannot be successful converting one group at a time. The strategy in the tornado phase is convert industries and markets one at a time, via tornados forming within industries or markets. The tactical execution is to get smaller industries or markets committed and therefore the larger industries or markets will then embrace the movement. That is simply crossing the chasm. DevOps has too many cultural implications to convert believers over a long period of time; DevOps requires reorganization, not a subtle shift.

    Since I consider DevOps to be in the earthquake phase, crossing the chasm is not an option. DevOps must jump the chasm in a surge of overwhelming momentum, hence creating a tsunami. DevOps has created a very closely knit culture, DevOps needs to push hard to create a tsunami and flood the pragmatic population. But, be advised, DevOps has to be all but universally accepted, there is no almost in bringing together Dev and Ops. But if DevOps wins, it wins big, DevOps will go from being a niche culture to a truly mainstream practice.

    So what’s the strategy to create a tsunami? And more importantly, how do you move beyond the earthquake phase? Here’s how I see it according to Geoff, with my DevOps spin: First, DevOps must clearly establish that the mission-critical software development process is broken and is causing management enough pain that it would gamble on unproven methodology that is populated with a bunch of prima donnas. I am not convinced the DevOps cult has clearly established this as fact. Is the development and operations process broken or have just new power tools been created to improve speed and accuracy? Second, DevOps needs to present the total solution, not just tools. Overwhelm the community; each new DevOps tool should leverage the success and technology of the last hence creating a methodology. Even more importantly, because the DevOps tools are closely related, credibility can cross industries and markets and create the momentum required for a tsunami.  Once Devops has reached critical mass by providing the total solution in a few industries or markets simultaneously the tsunami is created and DevOps has jumped the chasm.

    The tsunami will emerge suddenly with everyone seeming to adopt DevOps all at once.  This will result in explosive growth and will pull, willingly or unwillingly, everyone into its path. It’s driven and sustained by the pragmatists.  Here, everybody has to get the latest thing all at once; the masses have been converted to DevOps. DevOps needs mass-market success, DevOps needs to become a tsunami and everyone will want to play DevOps.

    Now that you know the formula, go forward and create the perfect storm.

    If you would like to say something; email parker@devops.com and don’t forget to comment here.

     

  • Using AWS CloudWatch for Anomaly Detection

    Using AWS CloudWatch for Anomaly Detection

    Based on my unscientific poll of friends, one of the least used and most overlooked features of AWS is CloudWatch.  Not only can CloudWatch be used to monitor the availability of your AWS services, but it can also be used as anomaly an detection tool. Did I mention that these feature are free?

    Since the holy grail of DevOps is to bring  the security and the operations teams together to meet the same goals in an automated fashion; Cloud Watch is a perfect DevOps tool. In fact, once it’s implemented for your environment  you are one step closer to the exalted state of SecDevOps.

    If you aren’t familiar with AWS CloudWatch then head on over to https://aws.amazon.com/cloudwatch/. As a quick introduction, here’s how AWS website describes the service:

    “Amazon CloudWatch provides monitoring for AWS cloud resources and the applications customers run on AWS. Developers and system administrators can use it to collect and track metrics, gain insight, and react immediately to keep their applications and businesses running smoothly. Amazon CloudWatch monitors AWS resources such as Amazon EC2 and Amazon RDS DB instances, and can also monitor custom metrics generated by a customer’s applications and services. With Amazon CloudWatch, you gain system-wide visibility into resource utilization, application performance, and operational health.”

    Let’s assume that both security and Ops teams need to be aware when weird things are happening with their servers and applications.  Events outside the norm (aka weird things) can be either be related to security or operational events; either way they may merit investigation. Here’s a simple CloudWatch  monitoring process you can use  to give you a heads up on events that could indicate someone is trying to exfiltrate data from your systems.

    Opening Scene

    Think about what kind of information attackers are after.  They want your intellectual property and your customer data because it can be converted to cold, hard cash.  Next, consider the architecture of your systems and where that data is stored. Most likely your treasures are stored in databases or Git repositories.

    Guess what? Attackers know that so that’s exactly what they are looking for.  Take a few minutes and think like an attacker bent on exfiltrating your data.  Obviously, the first thing you should do is carefully monitor your egress traffic for suspicious activity.  In addition, look for anomalies in traffic flows – they can also help determine if suspicious events are happening.  Amazon CloudWatch can help you do  both of these things.(Note: This process assumes you haven’t already been breached.)

     1. Enable CloudWatch

    2. Establish a baseline

    Review the historical data on a daily, weekly and monthly basis.  Soon you will start to see a trend in your egress bandwidth usage.  Start with eyeballing the min, max and average usage over time.  Put these in your favorite tracking tool (text file, spreadsheet, abacus, whatever).

    3. Configure alerts

    When you feel comfortable with your baseline, configure your first alert to send an email if traffic usage goes over the baseline.  Consider how “chatty” you want this event to be.  For example, alerting over the average baseline will often generate more alerts than alerts over maximum usage.  This is a tricky balance and you’ll need to make trade offs between missing problems and the time spent responding to potential false positives.

    4. Respond

    However you choose to respond is up to you.  I suggest you start with automatically opening a ticket in your tracking system so that someone will be sure to review the alert and make notes on their findings.

    5. Tune as needed

    Make a plan to review the alerts and the data on a regular basis.  Architecture and other changes to the system are going to change the baseline and the alerts may need to be reconfigured.

    Building and automating security into DevOps is a requirement that everyone should be working toward. It’s a theme I’m going to be returning to in future posts because it’s really important.

    AWS Cloudwatch is a free tool that you can help you get started automating security right now, so what are you waiting for?

  • No DevOps Roadmap? No Problem.

    No DevOps Roadmap? No Problem.

    I’m a huge fan of The Phoenix Project and I’m honored to count Gene Kim both as one of my professional inspirations and also as a personal friend. There is, however, a dangerous precipice inside of the Phoenix Project that, if misunderstood, could cause many DevOps practitioners and enterprise IT shops a daunting problem in adopting and progressing in the ways of DevOps.

    Roadmap or Roadblock?

    One of the main concepts discussed in both Kim’s The Phoenix Project and Goldratt’s The Goal is “the bottleneck” and what what to do about it. The basic premise explained is that all actions to increase throughput before a bottleneck, pile work up at the bottleneck and all actions to increase throughput after a bottleneck create a void of work to address after the bottleneck. The logical conclusion from this idea is that: in order to have a net positive effect on throughput, action must be taken at the, holistically viewed, tightest constraint on throughput within the operational system.

    This concept model is both accurate and valuable as the IT shop and the manufacturing floor have a lot in common. The power of this concept model, also happens to be where the aforementioned dangerous precipice is revealed – getting alignment on which bottlenecks to focus on can turn into a multi-month effort in many enterprises which can require consultants, workshops, deep-dives, process studies and other activities that are at once highly valuable and not always likely to be approved by a shop that has not fully bought in to the three ways of DevOps.

    One of the lesser understood secrets of corporate america is that even when you have formal authority, permission is required to do almost anything of substance. One “no” from the wrong party can kill even the most promising initiatives. This “veto effect” doesn’t require an absolute “no” either. Much like a parent saying “let me think about it”, the veto effect only requires the absence of a “yes”. This absence can take the form of “maybe”, “let me think about it”, “let’s wait and see”, or even a “run this by the other department heads and see what they think”.

    Given that any reasonably sophisticated roadmap takes more than a few weeks to develop, vett and roadshow around a medium to large enterprise, a mature DevOps roadmap suffers from a classic chicken/egg paradox for many organizations. I extend my deepest sympathies to those who have found that a clear and easily understood roadmap can somehow both provide and require the catalyst to consensus and permission to proceed in the ways of DevOps. You can take heart, however, in the fact that neither you, nor DevOps, are alone in this conundrum as this is what can be referred to as “the corporate condition” that we must all bear.

    While company may not always provide solace to the miserable, there is however, another way.

    Just Follow the Sun

    What is this one side road still available to you that has the power to relieve almost any non-alignment traffic jam? Start heading west! Just like the American pioneers, you can start following the sun to the wide open spaces of DevOps land where automation is plentiful and provable performance gains via instrumentation are gold. All you need is one or two comrades with some skills worthy of the road ahead and you can be off!

    Just because you can’t get the unix team to automate environment creation, doesn’t mean you can’t head west and automate your build processes. Just because you can’t automate your entire production deployment doesn’t mean you can’t head west and automate your performance testing and benchmarking processes. Just because you can’t get everyone to agree on the tightest constraint, doesn’t mean you can’t head west and alleviate the ones you already know about. To head west on any of these things, you don’t need full concensus, you only need a traveling partner or two.

    The Trail Has Already Been Marked

    For mid to large size enterprises, a true DevOps transformation happens over the course of three or more years. Allowing a lack of consensus to become a total standstill is not the right answer. The tool at your disposal to overcome the veto affect is public consciousness.

    Take UX design principles for example. It was less than a decade ago that design activities were considered a luxury item that produced pretty pictures; and then Apple happened and tipped public consciousness over. How many people are willing to casually veto design now?

    When public consciousness to the value of DevOps tips over, it will no longer be you going to the powers that be to get the “OK to proceed”. It will be them coming to you and asking “what is taking so long?”. In that one moment, you’ll be able to fire back at them: “Here’s my roadmapping proposal. I’d love to have you be my business owner!”

  • Parallel Evolution of DevOps Enterprise Patterns

    Parallel Evolution of DevOps Enterprise Patterns

    The DevOps evolution (Revolution!) has a great story; a story of winning the hearts and minds of people of all experiences who all anxiously strive to achieve their goals in amazing new and opportunistic ways. This story isn’t that of throwing everything away and starting over, nor is it a story of always having to start from scratch and do things from the ground up. It’s a story that reflects the hard work, critical thinking, scientific processes and deep analysis of your business needs with your goal in mind.  The concepts of the classic DevOps cloud business and the classic Enterprise all evolve in different ways but end up with the same evolutionary cognitive artifact, the focus on improvement and adding value through technology.

    When it comes to talking about Enterprise patterns vs the more spoken about “Cloud” patterns, we sort of speak to them as if their exclusively different, almost denying them an existence by choice.  One sees an “Enterprise” shop and almost cringes at the thought of them being DevOps out of an ideological view more than an evolutionary view.  However, when studying a “DevOps Enterprise” and “Cloud” company  from the ten thousand foot view we can see them as being much more similar.

    Cloud Patterns

    Develop -> Build -> Test -> Deploy

    Enterprise Patterns

    Develop -> Build -> Test -> Deploy

    Visually, this pattern while overly simplified is the same but how people get from develop to deploy may be entirely different.   In a developer centric “cloud pattern” you may be developing a web application to offer self service functionality to your end users but in a corporate environment you may be less developer centric as a business analyst adding features in a corporate resource planning application.  The end result is still creating value for your customer, developing functionality, building that functionality, testing that functionality and deploying that functionality and doing so in a very “DevOps Way”. Taking a closer look at the Enterprise through a slightly stronger magnifying glass, we would see something more like this for a typical enterprise deployment pattern:

    Storage -> Physical Host > Hypervisor ->  Virtual Host -> Database  <-> Application -> Customer

    Everything that composes this pattern could be made of of all sorts of parts from many vendors, open source projects, COTS systems and value add resellers. The components aren’t really what matters.  It’s the technology that an organization has acquired in doing business and the processes they have developed to meet the demands of their organization over their years of growing. It’s grown organically just as our concepts of IT have grown over there years and its done so creating and growing value too (at least, that’s the expectation). It’s evolved.  For better or worse, there is something we’ve encoded in these concepts and transmitted in an artifact we call “Enterprise”.

    The end result of applying DevOps to an Enterprise “Develop -> Build -> Test -> Deploy” pattern is to add value to this pattern, align this pattern to the business and to optimize the flow of this pattern. It’s really no different than applying these ideas to a “Cloud” business. One could optimize the Enterprise pattern of their Oracle Financials application by doing continuous integration just as cloud operations strive for continuous integration, but the CI you see in Oracle world  may be drastically different than that of typical Cloud Ops CI. Enterprise CI may be more of a “validation” experiment, a “Safe to fail” experiment. It may be more business process oriented – you may see people experimenting with BPEL or other tools to wrap business requirements and compliance around tasks/alerts/metrics/services/data.  The more we talk about Enterprise DevOps patterns the more we realize that we can benefit from these experiences in all DevOps patterns! There is a lot we can parallel in our views and see value from. Just because the “Enterprise” and the “Cloud” may appear differently, doesn’t mean they’re exclusively different by mandate or incapable of converging ideas.

    Looking more closely at the Cloud, we see this a mode of operations:

    Virtual Host -> Database  <-> Application -> Customer

    More frequently, depending on which type of cloud you seek out, you could simply see:

    Database  <-> Application -> Customer or simply just  Application -> Customer

    I feel it’s important to look at this evolution as parallel or even somewhat convergent (the differences being arbitrary if you ask me) because the end results are the same, that all-powerful “Goal”.    How one gets there, is how you tell your own story, but that story is your own evolution of services and what worked for you and your survival. The common hereditary artifacts of this separate but in the end parallel evolution of services is that they appear to build upon each other; sharing similar services, using APIs, using abstractions when needed, using automation when needed, building up “self-service” type systems/frameworks and doing so with the end user in mind and creating value (surviving!). To the customers in an Enterprise business, all they see is “application” or “service” just as in the cloud services organization your customer  sees “application” or “service”.  DevOps is the patterns that create/add value/optimize flow regardless of the service to the customer.

    Some people call the Enterprise applications and services we all know so well as the “Dinosaurs” of the computing age. Let me remind you, dinosaurs ruled the earth for 230 million years;  that’s 229+ million more years than humans! Dinosaurs weren’t stagnant by any means, they continually evolved and grew with staggering variations and abilities to survive.  If you stopped evolving, you went extinct.  Beyond the extinction of the dinosaurs, many species from the same era did survive and they survived not just because they had the winning combination of hereditary traits for survival but because they evolved through the changes that continued to happen regardless of their will to survive. Dinosaurs had an evolutionary dead end not because they did the wrong things or had the wrong design, but because couldn’t survive against the external forces that happened upon them.  The Cloud business could just as well be a dinosaur, soon to be extinct if that business fails to evolve for its own survival. Same for the Enterprise business.  We’re all evolved from dinosaurs in the end and if some have figured out how to survive through their cosmic collisions and still appear to outsiders as a metaphorical dinosaur, then that survival should be seen as having value in and of itself. The cognitive ability of reason should win, not an ideological opinion thereof.

    DevOps at “Big Giant and often cumbersome Corporation” may be different in scale/ideas, but the goal is just as much the same  and the experiences we can learn from each other, are just as much the same. Join my efforts in documenting DevOps in the enterprise, and help work on Enterprise DevOps patterns or DevOps Audit Defense concepts.  Both of these are open to participation and aren’t in any way specific to any technology or service other than sharing experiences that are common in the arbitrary “Classic Enterprise” or “Classic Corporate IT” shop.

    Through the parallel evolution of Cloud and Enterprise systems, convergence may be the end result, but it certainly isn’t the mandate. Keep evolving! Mother nature has done it for billions of years and she isn’t stopping anytime soon.

  • An Introduction to DevOps

    An Introduction to DevOps

    What is DevOps

    You can ask 10 people for a definition of DevOps and likely get 10 different answers.

    If you want to find a definition of your own, your research will probably begin by asking Google, “what is DevOps”. Naturally, Wikipedia is one of the first result so that is where we will begin. The first sentence on Wikipedia defines DevOps as “a software development method that stresses communication, collaboration and integration between software developers and information technology (IT) professionals.” Well, that’s a fairly dense definition, yet still pretty vague. I think DevOps can be explained simply as operations working together with engineers to get things done faster in an automated and repeatable way.

    So let’s delve into how DevOps works and can better adjoin two traditionally opposing departments.

    Growing pains

    When you are responsible for large distributed applications the operations complexity grows quickly.

    • How do you provision virtual machines?

    • How do you configure network devices and servers?

    • How do you deploy applications?

    • How do you collect and aggregate logs?

    • How do you monitor services?

    • How do you monitor network performance?

    • How do you monitor application performance?

    • How do you alert and remediate when there are problems?

    Combining the power of developers and operations

    The focus on the developer/operations collaboration enables a new approach to managing the complexity of real world operations. I believe the operations complexity breaks down into a few main categories: infrastructure automation, configuration management, deployment automation, log management, performance management, and monitoring. Below are some tools I have used to help solve these tasks.

    Infrastructure Automation

    You no longer have to build a server from scratch, buy power and connectivity in a data center, and manually plug a machine into the network. After wearing the operations hat for a few years I have learned many operations tasks are mundane, manual, and often have to be done at two in the morning once something has gone wrong. DevOps is predicated on the idea that all elements of technology infrastructure can be controlled through code. With the rise of the cloud it can all be done in real-time via a web service. Infrastructure automation solves the problem of having to be physically present in a data center to provision hardware and make network changes. The benefits of using cloud services is that costs scale linearly with demand and you can provision automatically as needed without having to pay for hardware up front.

    Configuration Management

    Configuration management solves the problem of having to manually install and configure packages once the hardware is in place. The benefit of using configuration automation solutions is that servers are deployed exactly the same way every time. If you need to make a change across ten thousand servers you only need to make the change in one place.

    There are other vendor-specific DevOps tools as well:

    In the operations environments I have worked in there were always strict controls on who could access production environments, who could make changes, when changes could be made, who could physically touch hardware, and who could access what data centers. In these highly regulated and process oriented enterprises the thought of blurring the lines between development and operations seems like a non-starter. There is so much process and tradition standing in the way of using a DevOps approach that it seems nearly impossible. From an operational perspective, my first instinct is to understand the application architecture so I can start thinking about the proper deployment model for the infrastructure components. From a development perspective, my first milestone is to make sure the ops team fully understands the application and what it takes to deploy it to a pre-production environment.

    DevOps isn’t just a set of tools, but a philosophical shift that needs that requires buy-in from all folks involved to truly succeed. It’s only through a high-level of collaboration that things will change for the better.

  • A Critical First Step in Bringing Dev and Ops Together

    A Critical First Step in Bringing Dev and Ops Together

    We all want to deliver excellent software to our customers. We now have a greater awareness than ever before of the interconnectedness of all the activities required to do so. The agile movement started this but it was not until the advent of DevOps that Ops was considered part of the full picture. In this article I will explain an effective first step that agile organizations can take to complete the picture and bring ops more completely into the product development process.

    Delivering excellent software starts with product definition and flows from those initial stories and designs, into dev, verification and testing, into production and then back again in the form of customer feedback and bugs. In reality the feedback paths represent a transitive closure of all activities just mentioned as you can see in the diagram below. Agile recognized the importance of feedback loops in good product development. Agile also realized that for the right information to flow, walls had to be broken down. The motto “Done means tested” arose and indicated that product owners were not done with a story, could not consider it closed, until it had been tested and they had verified it. This represented the integration of PM, development and QA functions. A huge step, but one step too short. Operations was still not viewed as a key part of building excellent software and there was not much insight into the fact that the feedback loop did not stop at testing. DevOps has recognized this shortfall and taken it to its logical conclusion – “Done means in production.”

    Operational Excellence Feedback Loop

    Product development is a cycle, a cycle that operations is a critical part of. In describing how all parts of the cycle can be integrated lets start at the most natural beginning; product definition. Product definition activities produce stories, designs, functional acceptance criteria (A.C.) and prioritization. In this phase we don’t produce all we need for the finished product but just enough to get us going. With popular agile methodologies like Scrum this is ideally not done solely by a product owner but instead in conjunction with dev and test at story grooming meetings, design reviews and sprint planning. In these meetings dev, QA and product communicate and negotiate stories, comment on designs, and flesh out acceptance criteria. In this way the stories, designs and acceptance criteria are improved and everyone gains an understanding of what is being built; they also become invested in it. A story may look something like this (abbreviated):

    Story: Blog administrator can change user roles SO THAT users can be managed effectively

    A.C: Given blog administrator is logged in
    And on a user profile
    When he selects a permission scheme from a permissions drop-down menu
    Then the users permissions are updated and stored in the DB

    This is a great start but it is not enough. If we are to be excellent we must modernize this just a bit and add in some DevOps spice.

    A first step towards bringing ops effectively into the cycle is to include someone responsible for actually operating the software in the story and acceptance criteria grooming sessions, design reviews, and sprint planning meetings. The critical aspect is that when the team negotiates the acceptance criteria associated with a story it will be ops responsibility to ensure we for each story or feature the team adds success and failure metrics, initial alarm values and any operational functionality such as a kill switch that will be needed in order to operate the functionality effectively. So a story will now contain things like the following in the A.C. (acceptance criteria).

    Given operation X fails
    Then the metric operation-x:failed will be sent through the metrics feed

    We will also alter our definition of done to make sure that a story will not be closed until we meet the ops driven acceptance criteria. For example: we can see the metrics we have defined in our A.C. show up on an ops dashboard. There are so many benefits to this such as folks that are unlucky enough to be on on-call will have participated in this story grooming and will know exactly what these metrics mean when they encounter an issue with them for the first time in production. Furthermore, they will know what operational levers they can pull or request to be pulled.

    This first step represents a connection from product definition to dev, to QA, to operations. This connection is perfectly natural and intuitive when you think about it. Being a true DevOps organization and developing operationally excellent products requires more but having connected product definition to operations as we have provides a nice toe hold from which to flesh things out further. Best of luck. If you have any questions leave them as comments and I will respond as best I can.