devopsy.kinsta.cloud

Category: Enterprise DevOps

Parallel Evolution of DevOps Enterprise Patterns

The DevOps evolution (Revolution!) has a great story; a story of winning the hearts and minds of people of all experiences who all anxiously strive to achieve their goals in amazing new and opportunistic ways. This story isn’t that of throwing everything away and starting over, nor is it a story of always having to start from scratch and do things from the ground up. It’s a story that reflects the hard work, critical thinking, scientific processes and deep analysis of your business needs with your goal in mind. The concepts of the classic DevOps cloud business and the classic Enterprise all evolve in different ways but end up with the same evolutionary cognitive artifact, the focus on improvement and adding value through technology.

When it comes to talking about Enterprise patterns vs the more spoken about “Cloud” patterns, we sort of speak to them as if their exclusively different, almost denying them an existence by choice. One sees an “Enterprise” shop and almost cringes at the thought of them being DevOps out of an ideological view more than an evolutionary view. However, when studying a “DevOps Enterprise” and “Cloud” company from the ten thousand foot view we can see them as being much more similar.

Cloud Patterns

Develop -> Build -> Test -> Deploy

Enterprise Patterns

Develop -> Build -> Test -> Deploy

Visually, this pattern while overly simplified is the same but how people get from develop to deploy may be entirely different. In a developer centric “cloud pattern” you may be developing a web application to offer self service functionality to your end users but in a corporate environment you may be less developer centric as a business analyst adding features in a corporate resource planning application. The end result is still creating value for your customer, developing functionality, building that functionality, testing that functionality and deploying that functionality and doing so in a very “DevOps Way”. Taking a closer look at the Enterprise through a slightly stronger magnifying glass, we would see something more like this for a typical enterprise deployment pattern:

Storage -> Physical Host > Hypervisor -> Virtual Host -> Database <-> Application -> Customer

Everything that composes this pattern could be made of of all sorts of parts from many vendors, open source projects, COTS systems and value add resellers. The components aren’t really what matters. It’s the technology that an organization has acquired in doing business and the processes they have developed to meet the demands of their organization over their years of growing. It’s grown organically just as our concepts of IT have grown over there years and its done so creating and growing value too (at least, that’s the expectation). It’s evolved. For better or worse, there is something we’ve encoded in these concepts and transmitted in an artifact we call “Enterprise”.

The end result of applying DevOps to an Enterprise “Develop -> Build -> Test -> Deploy” pattern is to add value to this pattern, align this pattern to the business and to optimize the flow of this pattern. It’s really no different than applying these ideas to a “Cloud” business. One could optimize the Enterprise pattern of their Oracle Financials application by doing continuous integration just as cloud operations strive for continuous integration, but the CI you see in Oracle world may be drastically different than that of typical Cloud Ops CI. Enterprise CI may be more of a “validation” experiment, a “Safe to fail” experiment. It may be more business process oriented – you may see people experimenting with BPEL or other tools to wrap business requirements and compliance around tasks/alerts/metrics/services/data. The more we talk about Enterprise DevOps patterns the more we realize that we can benefit from these experiences in all DevOps patterns! There is a lot we can parallel in our views and see value from. Just because the “Enterprise” and the “Cloud” may appear differently, doesn’t mean they’re exclusively different by mandate or incapable of converging ideas.

Looking more closely at the Cloud, we see this a mode of operations:

Virtual Host -> Database <-> Application -> Customer

More frequently, depending on which type of cloud you seek out, you could simply see:

Database <-> Application -> Customer or simply just Application -> Customer

I feel it’s important to look at this evolution as parallel or even somewhat convergent (the differences being arbitrary if you ask me) because the end results are the same, that all-powerful “Goal”. How one gets there, is how you tell your own story, but that story is your own evolution of services and what worked for you and your survival. The common hereditary artifacts of this separate but in the end parallel evolution of services is that they appear to build upon each other; sharing similar services, using APIs, using abstractions when needed, using automation when needed, building up “self-service” type systems/frameworks and doing so with the end user in mind and creating value (surviving!). To the customers in an Enterprise business, all they see is “application” or “service” just as in the cloud services organization your customer sees “application” or “service”. DevOps is the patterns that create/add value/optimize flow regardless of the service to the customer.

Some people call the Enterprise applications and services we all know so well as the “Dinosaurs” of the computing age. Let me remind you, dinosaurs ruled the earth for 230 million years; that’s 229+ million more years than humans! Dinosaurs weren’t stagnant by any means, they continually evolved and grew with staggering variations and abilities to survive. If you stopped evolving, you went extinct. Beyond the extinction of the dinosaurs, many species from the same era did survive and they survived not just because they had the winning combination of hereditary traits for survival but because they evolved through the changes that continued to happen regardless of their will to survive. Dinosaurs had an evolutionary dead end not because they did the wrong things or had the wrong design, but because couldn’t survive against the external forces that happened upon them. The Cloud business could just as well be a dinosaur, soon to be extinct if that business fails to evolve for its own survival. Same for the Enterprise business. We’re all evolved from dinosaurs in the end and if some have figured out how to survive through their cosmic collisions and still appear to outsiders as a metaphorical dinosaur, then that survival should be seen as having value in and of itself. The cognitive ability of reason should win, not an ideological opinion thereof.

DevOps at “Big Giant and often cumbersome Corporation” may be different in scale/ideas, but the goal is just as much the same and the experiences we can learn from each other, are just as much the same. Join my efforts in documenting DevOps in the enterprise, and help work on Enterprise DevOps patterns or DevOps Audit Defense concepts. Both of these are open to participation and aren’t in any way specific to any technology or service other than sharing experiences that are common in the arbitrary “Classic Enterprise” or “Classic Corporate IT” shop.

Through the parallel evolution of Cloud and Enterprise systems, convergence may be the end result, but it certainly isn’t the mandate. Keep evolving! Mother nature has done it for billions of years and she isn’t stopping anytime soon.

April 7, 2014
How to Evolve to DevOps

What is DevOps, and how do we do it?

It’s a question I’m often asked, with confusion around job titles, team structures, tooling, personnel responsibilities, office layout and more as potential ingredients for a successful devops recipe. The first step in evolving an existing organization to a devops mentality is to take a step back, and level up your thinking, far from the tactics mentioned above.

The first thing to recognize is that this is a cultural and professional movement of technology workers wanting to be more impactful for their businesses and be personally happier themselves, and this movement has three key strategic pillars of success: cultural, structural, and tooling. When we get all three right, we enable a devops mentality, the benefit of which is a business that can respond quickly to customer or market feedback (i.e., agility).

The three pillars are foundational to the collaboration among technologists to enable this business agility.

Evolving to DevOps

You have a status quo of culture, structure and tools. Regardless of whether you explicitly defined each, or allowed some to more implicitly evolve than others, the fact remains that you have a status quo. How do you evolve? You need to tackle each pillar in turn, understanding your status quo, and what helps or hinders a devops mentality for your organization. The first pillar is the most critical: culture. Let’s speak to that first.

A Culture to Enable DevOps

What happens at your company when things go wrong? Be really honest. Do folks become defensive about their role in contributing to a problem? Do they blame others for problems? Do fingers point outward before they point inward? Do you ever hear, “It’s not my servers, it’s the change you made to the code!”, or the converse, “It’s not the code, something’s wrong with the servers!”.

If any of the above holds true, stop everything right now, get everyone together, and talk about what you see in how you interact with each other. Your goal is to enable a shared outcome mentality across functions and jobs and roles (i.e., we all succeed or fail together), and a culture of self-improvement and introspection where the first thought when things go wrong is “what could I have done to prevent this from happening, and how do I want to change what I do in the future to make it better?”. When you get it right, people will gravitate TO problems and collaborate on solutions, not distance themselves AWAY from them to avoid blame.

So what do you say? How do you say it? What’s necessary to evolve the culture? Every situation is unique, and you’re going to need to chart your own personal and organizational evolution to foster a culture that can enable devops. If you’re the leader of the team or business, start surrounding yourself with mentors and peers in other companies that have achieved this, and ask to learn from them.

If you’re not the leader, again, surround yourself with mentors and peers that have built this culture, and ask to learn from them. Then be the change you seek in your organization, and lead a cultural shift from within. Either way, you’ll begin to discover your own unique path of learning, supported by many readings and mentoring conversations, that will empower you with the skills and conviction to bring about cultural change.

If you’re the current leader of the team or business, and you’re not willing to undergo this difficult process of personal and organizational evolution of your culture, I have a rude awakening for you: you won’t be the leader for the long term. Over a period of years, either your business will succumb to competitors who are able to get this right, or a member of your team will take over for you as the rightful cultural steward. Choose wisely.

A Structure to Enable DevOps

Within your company, look at your organization chart, and look at the people that you are physically near or collaborate with the most, and answer the question: what are we responsible for? Now look at other parts of the organizational chart or office layout for departments or functions of the organization that you are dependent on for collective success. What you’re looking for is a physical or proverbial wall in the organization that draws boundaries on what one group of people are responsible for, and what the next group of people are responsible for; these walls create a “that’s someone else’s problem” mentality. In the most classic devops sense, we’re talking organizational (i.e., you report in to different parts of the organization from a managerial perspective, or you are accountable to metrics that put you at odds with another part of the company) and physical (i.e., you sit in different areas of the office) silos separating developers from operators. If you’ve ever heard the phrase “developers toss the code over the wall for operators to run,” you know what I’m talking about.

Some of these physical and proverbial walls will make sense in the organization, while others you’re going to need to question why they are there. You may even find some walls that don’t make sense with other parts of your organization (say product management, or security, but we’re going to stick to the classic developers and operators example for brevity). Physical and proverbial walls that hinder collaboration toward shared outcomes must be evaluated for removal.

The classic example from a proverbial perspective is a development function of an organization being held accountable for performance metrics such as number of features released, while an operational function of an organization being held accountable for reliability metrics. Clearly, there will be tension and angst as these metrics can easily pit two groups of people against one another. In this example, a common solution is to focus on shared outcomes, and hold everyone accountable for both feature velocity and reliability. Another common solution is to have both developers and operators report in to the same person. Another common solution is to put developers “on call”. The tactics will vary on the situation, the strategy remains the same: break down the physical and proverbial barriers to collaboration and embracing shared outcomes. If there is a wall that prevents people from working together to improve the business, tear it down.

Tooling to Enable DevOps

How effective are you on 3 hours sleep after being paged all night? How strategic is your thinking while typing in a command for the Nth repetitive time? How demotivating is it to put all of your energy into treading water without signs of visible forward progress? Even the greatest collaborative culture with a structure reinforcing shared outcomes will fail if everyone is burning out debugging problems and performing the same mundane tasks over and over. This is where you need to evaluate your tooling.

Start with the repetition. Every time some task comes up a second time, make a commitment organization wide to automate it and track changes in revision control. Use this as a foundation to eventually drive to fully automated configuration management from revision control, but you have to start somewhere, and you will free up significant time and frustration over the long haul with this simple mentality shift. We’re going to invest this newly found time into the followup tooling activities.

With that executing over time, turn your attention to telemetry. Are you collecting the right data from the right places to let you understand where problems are? Data debunks all blame, and properly aligns people toward investigating problems in a collaborative manner. Consider “monitoring” as the definition of thresholds on the right telemetry data, and “notifications” as the definition of who needs to know and when. Be diligent on getting the right threshold and right notifications; set thresholds too aggressive or notifications too broad, and folks will become numb to the system and lose trust, while setting thresholds too passive or notifications too narrow, and important issues affecting customers could get missed. Over time, improved telemetry will enable you to predict problems before they happen, and the right thresholds and notifications will ensure the right people know what to do to fix what problems right away, freeing up more time to go into the final phase of tooling.

With common tasks automated, and a good, clear understanding of how well your systems are performing and where problems lie, you can focus your energy on improvements that will enable resilience for the future, both in your technical systems and in your teams. On technical systems, you’ll investigate refactored architectures that are more scalable and fault tolerant (e.g., horizontal scale, multi-datacenter load balancing). For your teams, you’ll invest in better collaboration systems (e.g., wiki, chat, ticketing, etc.).

You’re Never Done

While I’ve provided a framework to systematically tackle common barriers to enabling the business agility that comes from a devops mentality, truth be told, there will always be more to do, and there will always be more personal, organizational and technical improvement to be had. Embrace this journey, be proud for each new milestone along the evolution, celebrate every victory, and deliver wonderful things for your customers.

April 7, 2014
Collaboration – A DevOps Tenet
You can’t turn around in the tech space without hearing about collaboration. Whether it has to do with the new economy being built around collaboration or the platforms designed to improve it or the various strategies one might implement to spur more of it on your team, collaboration is the hot new term, especially when it comes to DevOps.

All that being said, what is collaboration exactly? And the deeper question…can you get to an answer to a problem faster if more viewpoints are involved? Can you cut through space and time to overcome physical barriers and find the specific knowledge someone on the team has that others do not? Can you make the practices of DevOps, which are already more nimble, even faster by getting more eyes on the situation?

Here’s what we talk about when we talk about collaboration internally at VictorOps:
- remote engagement – In order for collaboration to work, you need to be able to get folks involved no matter their physical location. Collaboration doesn’t happen just between 9-5 and doesn’t take place just in the office setting. Team members want to help out (especially during a firefight) but they may not want to give up everything in their life in order to do it.
- real-time troubleshooting – As anyone who has used Salesforce or JIRA can attest, effective collaboration doesn’t happen in a ticketing system. If you want to work together, you need a communication channel that allows for synchronous discussion that keeps everyone on the same page, at the same time. Immediacy is a crucial factor in figuring out the cause of a problem and synchronous messaging, which closely resembles face-to-face conversation, is one of the best means of expressing that.
- the right tools – These days, if you want your team to adopt a collaboration tool, you have to keep two things in mind: the tool must be mobile and the tool must feel natural to use. Without these two things, you’re going to have a hard time getting everyone on board. If a tool is clunky or requires a steep learning curve, no one will actually use it, making for very ineffective collaboration.
- solving problems faster – The end result of effective collaboration is coming to a solution in a quicker manner. Whether it’s because you’re getting the right eyes on the problem, noticing data patterns or reducing new participant spool-up time, collaboration should help the entire team do their jobs better while achieving company objectives.
Collaboration can become just another marketing term or vague business concept without any concrete examples of why it’s important. In our next post, we’ll talk about different types of collaboration (vertical vs. horizontal, asynchronous vs. synchronous) and why collaboration works best when it’s vertical.
April 4, 2014
DevOps Is Free? Sure I’ll Support It
One of the biggest misnomers about DevOps is that since it’s all about process and communications there aren’t any costs associated with adoption. Sure, you can probably gain some improvement by simply modifying your current methods for delivering IT services in your organization, however, DevOps is about more than just modest improvements.

Let’s explore why organizations that are succeeding with DevOps are succeeding over traditional IT approaches. From the article, “Fresh Stats Comparing Traditional IT and DevOps-Oriented Productivity,” also on devopsy.kinsta.cloud, we see that DevOps is driving down time spent on tasks where there is no value being added to the business, such as support, deploying changes, communication and firefighting. Time spent on these tasks reduce opportunities from innovation and responding to business needs, leading the business to view IT in an unfavorable manner unable to help them meet current business demands.

Where DevOps-oriented teams do spend time is on automating repetitive tasks so that they don’t continually consume time, self-improvement and innovation. Instead of treating IT like assembly line workers capable of only completing the task at their station, which causes friction and increases turnover, DevOps-oriented organizations foster an environment where workers are rewarded for ensuring quality and reducing overhead.

So, where are the costs associated with becoming a DevOps-oriented organization?
- Training / Education – DevOps transformational activities takes funds and requires outside training and, possibly, consulting. For businesses that desire to move forward but are stuck in the Catch-22 that is keeping the bus rolling while trying to paint it, it can be daunting to make the necessary changes. Getting past this point often requires groups trained in helping businesses make these types of transformations.
- Tooling – There’s some great Open Source tools that can help drive continuous integration, automation and orchestration. However, the worst thing that can happen is that getting these tools to work the way you need them to ends up consuming resources and time that you’re trying to give back to driving business value. The supported versions of these tools will provide software support and deliver tested versions, which will minimize unnecessary disruptions.
- Updates / Refresh – Older versions of some infrastructure components or software packages may not support automation efforts. Since this is a key element of DevOps, it may be a requirement to do a technical refresh or software update.
- Platform Changes – Alas, the big kahuna, the platform change may be a necessary action to enable your IT team to become more agile and alleviate some operational challenges. The cloud is a perfect example of a platform change that supports DevOps. A platform change can be very expensive, but the long-term benefits need to be weighed; especially if it speeds transformation of your IT team into one that is agile and more quickly addressing the needs of the business.
So, can you do DevOps without any financial outlay? Probably so, but the benefit received will be nominal, your team will most likely feel shorted and look to work for a business that has fully committed to a DevOps approach and the real time consuming sinks in your business will continue to exist eating your IT budget like a bear eating salmon in preparation for winter hibernation.
April 3, 2014
Don’t Let the (April Fools Day) Joke Be on You

As much as seasonal increases in demand for on-line resources are discussed, the one day that’s often overlooked is April Fools Day.

It’s the day the Internet loses its mind and your best bet is take everything not just with a grain of salt, but with the entire shaker.

The use of April Fools’ Day to promote brand awareness and bring some laughter into the world as everyone tries to outdo everyone else in a bid to be named the King of Fools is well known. This is no more a surprise today than is Black Friday, Cyber Monday or Super Bowl Sunday.

So it behooves those participating in the tom foolery to be prepared. That means being ready to handle an increased demand for whatever it is you’re offering up for consumption. This is where devops shines; where automation and orchestration and a demand-driven scalability strategy is your best answer. Because you can’t be sure your offering is going to go viral, but you can be sure it won’t if any piece of the application delivery chain breaks under the pressure.

In other words, this is unacceptable:

What might have gone viral may end up languishing as just another overlooked and forgotten attempt because of a lack of database resources and a failed scaling strategy.

One of the key aspects of devops is that it should approach scalability with a comprehensive view of how each piece of the delivery chain needs to scale – and when. Thresholds and metrics collected from each domain of an application architecture must be correlated and understood in the context of how it impacts capacity so that it can be automatically scaled – or at least addressed – on demand.

Maybe that means devops must advise a cloud-based approach to handling demand for such days. Maybe that means a bit more preparation and integration across the entire application delivery chain to ensure the load balancing service is basing its decisions on the right variables such that resources are maximized and enable greater scale across the application. Maybe that means throwing up a cache to reduce the pressure on web and application servers, or taking advantage of a CDN.

Regardless of the approach to how you’re going to scale, devops practitioners should be driving the process to ensure that on April Fools Day – or any well-known high demand generation day – the joke is not on them.

April 1, 2014
DevOps Lessons Learned – The “DevOps Engineer”

Back in 2008 I was part of a startup that had success and sold to large company three years later. During the startup years, there was three of us in IT plus an offshore team that did our web development. The three core resources owned everything from the API down while the offshore team owned the front end. Since we were small, we each owned our own part of the architecture and managed the deployments and operations of the system with few issues. Once we were purchased, our world changed drastically.

Our team quickly ramped up to over 20 people while my original two architects soon left for greener pastures. Now we had a lot more people involved in the process and none were domain experts. Things quickly started spiraling out of control as server sprawl started creeping in and deployments were becoming increasing challenging and error prone. We hired our first “DevOps” person to help us reign all of this back in. Then things got really ugly. For those who have followed my DevOps writings and my rant called “No you are not a DevOps Engineer“, what I am about to describe is the basis for my position on DevOps.

DevOps as a Role: Fail Pattern

We never intended to create a DevOps silo or position. We just wanted someone with expertise with cloud and automation since everyone else on the team was a developer and many of them were new to both the cloud and the platform. Unintentionally, this person became labelled as DevOps and he was soon joined with three other people which became an official DevOps team. We now were the proud owner of a new silo which became the new bottleneck.

The DevOps team had their hands full to regain control over the environment. Permissions had to be pulled back to prevent server sprawl from continuing. Development teams now had to request environments from the DevOps team. When we were a startup, this was easy and manageable. As the team got larger and more concurrent projects were being run, this became challenging. The DevOps guys were so busy keeping up with the demand that they were not always in tune with the goals and the deliverables of the different projects. The app dev teams were so focused on “agile” that they were not including the DevOps guys early enough in the projects.

The DevOps team became drastically overworked. They were monitoring and operating the systems, they were being pulled from their deliverables to provision environments, and they were trying to put controls around a platform that experienced sudden growth from a 3 person team of founders to a 20+ person team of new people. To make matters worse, we knew what we wanted to do to fix these problems but there were not enough hours in the day to get there. We had mapped out an entire continuous integration and delivery strategy, but we just could not get enough time to focus on it.

DevOps started to get a bad rap from the app dev team because there was often delays in getting environments provisioned in time. We experienced environmental issues between dev, QA, and stage because of the lack of timely communication of the environment requirements from the app dev team and because the different app dev teams were not being consistent (e.g. different version of Python and different libraries).

All of this led to frustration from the business, especially from the business people who came from the startup and was used to frequent deployments with high quality.

Lesson Learned

Eventually the team turned this around. Although they still have an official DevOps team, the team works much closer with the app dev teams. A high level of standardization is now in place for the environments. The teams share operational requirements early in the project life-cycle and developers assist in the environment creation. The lesson learned here is that creating a DevOps role or organization is not the correct approach. Creating a new silo just creates new bottlenecks. DevOps is not a role, it is a way of getting quality, reliable software to market faster. The sooner an organization realizes this the sooner they can get to a successful state.

March 27, 2014
How to laugh at DevOps
Talking about DevOps is all the rage these days. But with that conversation, it’s easy to become inundated with the amount of serious discussion around the topic. I do agree that DevOps is not something we should joke around about, but since humor is an inherent part of our culture at VictorOps, it makes sense that we would be drawn to the humor of the situation.

Below are the ten of the funniest Twitter accounts that make light of the DevOps movement. Some are obviously one-joke wonders while others continue to make us laugh. Either way, the list below is proof that you can’t take DevOps too seriously:
- @DevOpsBane – The superhero of faster deploys.
- @DevOpsYakov – In Soviet Russia, apps deploy you.
- @UncleDevOps – Everyone’s favorite continuously-deploying uncle. Also a little crazy.
- @DevOps_Jesus – Continuously delivering from evil since 4 BC.
- @DevOpsGrouch – I complain. I swear. I automate. Deal with it.
- @DevOpsTourettes – Development operaRIAK CHEF JUJU FUCK.
- @DevOpsBorat – Cultural Learnings of DevOps for Make Benefit Glorious Teams of Devs and Ops.
- @DevOpsX – Always fighting the good fight for DevOps equality.
- @DothrakiDevOps – Master of War, Horse and Large Scale Deployments.
- @DevTylerDurden – May the build never be complete.
Feel free to add any that I might have missed to the comments below. And remember to laugh…it’s just DevOps after all.
March 26, 2014
The Automation Continuum
The speed of innovation these days is often influenced by the speed of adoption of new and exciting tools, and specifically open source tools. Open source projects have been growing at an exponential speed over the course of the past few years; however the process for exploring, testing, configuring, and ultimately integrating these in production often times is still quite complex and time consuming. In many cases, this process is still done manually. Like with all DevOps processes by automating this, we are then able to adopt new tooling much more quickly, and with less human error in the process.

The Open Source Revolution

The friction in this process is due, in large part, to the fact that each of these stages from exploration through POC and testing, integration and then management, many times requires the use of very different tools for each stage, on top of a completely different setup and environment. This means that every time you want to move from exploration to testing, for example, you have to basically rebuild the entire system. This can become quite frustrating and, as mentioned above, time consuming.

What if we could streamline this process and explore tens and even hundreds of projects quickly, thereby enabling us to seamlessly increase our innovation?

How this could work – the flow:
1. Exploration: Explore the product without having to install any software at all. This could be through a click of a button on Github, and seeing the product in action. This would enable you to play with it without having to download, configure, or set up any machines.
2. Testing: When we move to trying it out, we want to use the same model as in the exploration phase. We’d like to try it out also on a SaaS basis, but this time with more control. By this I mean testing with real clusters & machines, in a more high performance, and production-like environment.
3. Development: At this stage, we want to be able to test it with our actual stack, and plug it into our project quickly. Then, eventually at a later stage download it and run it in our own environment.
What’s Needed

With a unified tool for all of these stages, the majority of the friction in the process can be diminished. One way to achieve this is by running the project as a service or in our environment, with an orchestration engine that can basically be used to take us through all these stages.

By wrapping the entire project into an uber-launcher [which with Cloudify we call it “the widget”] that any open source project can plug into – like an app store or repository, and any user can start using immediately without necessarily having to know how exactly it runs or works.

The nice thing is that we can eventually take the software from that app catalog or store, since it’s not packaged as a black box but rather as a blueprint or script. We can then fork it and customize it to our needs. Because of this, we can actually use the same tools through the entire cycle and break the friction completely.

So what it all boils down to, is the basic principle of combining the power of SaaS and cloud orchestration to avoid complexity of installation, use a script-based model to make the packaging of software generic – but still customizable, and use a cloud abstraction to move environments without having to change recipe. This is what we call the automation continuum, and we’ve been using to demonstrate how to take the power of any open source project to any cloud.
March 25, 2014
Config Management & F***ing Shell Scripts

Joke tech projects pop up all the time. Some, like the lamentable titstare, should never have seen the light of day. Others, like Mark Zuckerberg’s Facemash start out as a joke and grow into something that is anything but. Most come and go without anyone really noticing.

Whilst their lives are typically short we often learn more from people’s reactions to these projects than we do from their use of them. An excellent example of this popped up recently when a couple of frustrated Chef and Ansible users came up with a simple configuration management alternative, the regrettably named F***ing Shell Scripts (FSS). As they explain it, as in “Why can’t we just use f***ing shell scripts?”.

Whilst usability issues with configuration management tools – particularly in the Enterprise – is a topic close to my heart, I would not normally have had much more than a passing interest in FSS. What grabbed my attention was the reaction it elicited when posted to Hacker News. With over 150 comments FSS clearly touched a nerve.

Now to clarify, the authors of the project were quite definitive in assuring the mob that they had created FSS as a joke:

We wrote it literally in an afternoon, and were giggling like schoolgirls the whole time.

People either missed this, or chose to ignore it:

“This is simply brilliant! I’ve dreamt about such tools.”
– artellectual

“I’d love to see a tool like yours be brought to full maturity, then I think I’d love to use it.”
– sdegutis

It provided a perfect platform for many to voice their frustrations with current tools:

“Even for an experienced sysadmin, using Puppet and Chef to do even a trivial project (replacing a ~10 line shell script) took a painful couple of days.”
– mattjaynes

“In my experience, there’s a huge amount of work involved in just getting Ansible (and other CM systems) to work right in the first place”
– s_kilk

“We’re using Chef right now, and evaluating other options. They seem to either be between way too over-engineered, or not flexible enough, but nowhere in between.”
– sdegutis

“Keep it simple. puppet/chef have horrendous DSL’s that make it really complicated to reason about.”
– sp1982

In the interest of balance I should point out that there were some voices of support amongst the crowd. Ansible in particular won praise for the effort its team has put into simplifying things.

Overall though I was still blown away with the level of frustration being expressed. Automation tools have been around for years now; CFEngine, the grandfather of the modern tools was created in 1993. Despite this, a tongue in cheek project suggesting ditching them all in favor of the technology they set out to replace (or at least abstract away), shell scripts, is considered credible. For me this is akin to people ditching virtualization for physical boxes. It just doesn’t make sense.

Unless of course something is still fundamentally broken in the approach the current set of tools is taking; that they are still too hard; that they still fail to address key prerequisites like visibility and accountability. I believe that this is the case because I’ve experienced it first hand; I’ve seen it a hundred times in businesses I’ve talked to, and threads like the Hacker News one mentioned above leave little room for alternative interpretations. Heck, we’ve based our product GuardRail on this entire premise; these are not empty words on my part.

Incremental improvements such as those offered by Ansible are a good thing, but what people seem to be searching for is a step change. As things stand this is less likely to come from the current batch of tools, and far more likely to come in the form of a new approach. Docker anyone?

March 24, 2014
The Dawning of Continuous Support

Development teams have a tendency to think that once they push the deploy button, the release is done. But with Devops, it’s quite the opposite now. More software changes, and more deploys, make every production release a new piece of code with unique idiosyncrasies. This has turned what has historically been a more static production environment now highly dynamic.

Teams have responded by layering in a lot of monitoring and performance technologies, which has caused even more information overload. Continuous Deployment makes it easier than ever to push code to production but it has also created an additional priority within the company – the need to provide Continuous Support.

Continuous support addresses the human side of how to harness your team in more effective ways, using new methodologies to adapt to a much different world, and extends out to include other areas of the business besides IT. Sales, marketing and customer support are all going to be affected by this way of doing business.

Imagine that it’s 5 a.m and there’s an intense firefight with all the right Dev and Ops guys working to get the system back up. It’s quickly approaching prime time for the users of your app. How will you get Support up to speed so they can intelligently talk to the issues? Where is Marketing to manage the Twitter-storm and blog post? Or Sales, who is about to demo to a big East Coast prospect? Continuous support aims to help answer those questions.

If we’re going to break down the silos with DevOps, we have to extend that out to *all* silos in the company in order for our end-users to benefit. The ability to solve smaller problems faster means quicker turnaround and quicker fixes. There’s talk about getting customer features out faster but all conversations virtually ignore the support team…even though they’re the ones talking to customers on the front line, listening to feature requests and helping to resolve user issues.

When you extend situational awareness across the company, it becomes easier to see patterns and solve for them. Here are five tenets of customer support:

1. The team is always-on. This means that people are not just alerted when problems are happening but rather, get updated all the time. VictorOps has a Salesforce integration that provides the perfect example. If you’re a company promising 24/7 support, the old-school way to go about doing that is to hire eight people, which is the bare minimum to insure you are single threaded on most shifts. Now, we’re making it work with just a few people and the SF integration. There are tools available that enable your support team, reduce its size and make it more likely that you’ll get to the right person faster. You don’t need to overstaff, or have someone actually sitting there, in order to be situationally aware of your customer experience and efficiently solve their issues.

2. Collaboration is king. Getting different members of the team involved in quickly solving a firefight is a necessary but difficult task in the world of Continuous Support. If you can reduce the spool-up time for looping new participants in, then you’re on your way to finding the right answers. Allowing for scalable “lean-in” means that you have more eyes on the problem and can resolve that critical alert faster.

3. The feedback loop is crucial. All the different departments in the company need to be involved in completing the product feedback loop. This means that when scheduling postmortems, in addition to inviting the Support and Marketing teams to join in these discussions, Product, Development and Operations should all be involved. There’s always something to be learned by internal failures and those lessons should be shared with everyone in the company who might benefit from them. The feedback loop is essential for getting customer feedback all the way back up the chain for Product.

4. Automate, automate, automate. The single biggest driver of high performance, automation increases the overall speed & quality of code. But don’t stop there when talking about automating. You can also automate systems to help detect potential user issues, which is important if you don’t want your customer to be the first one to find the problem. Automating components of the support chain is a vital part of making Continuous Support work because it isn’t about removing the humans but instead making them more efficient through automation.

5. Company culture is inherent to success. If all employees in the business know what the company goals are and how continuous support can help to achieve them, then it becomes an easier sell to get everyone on board. Leadership is essential in making the culture of the company a priority and keeping it there. When Continuous Support is done right, it provides the entire product chain with the proper information and empowers your employees to make better decisions.

March 19, 2014