devopsy.kinsta.cloud

Tag: devops culture

Enterprise DevOps is Real and Here to Stay

A recent column in the Wall Street Journal’s CIO Journal argued that while DevOps is gaining inroads in small and mid-sized businesses, it’s facing a much less rosy future in large enterprises. The story cited organizational challenges, many of which we covered, such as large siloes, regulatory compliance, and resistance to change as great impediments enterprises embracing DevOps.

The column also spoke of the buzzword “DevOps,” how it’s a software development methodology, and spoke about “holistic” DevOps tools. Interestingly, that’s not how we view DevOps. Not at all. DevOps is more a movement about organizational collaboration and driving improvement.

Still, the author wasn’t completely wrong. There are significant hurdles to overcome when moving to a DevOps culture, and those that the author cited are the primary obstacles for both enterprise and SMBs – mainly cultural changes the processes to support it – and they are being overcome. Just, typically, at a slower pace among larger organizations.

“I think the SMB vs. enterprise is the wrong axis to use,” says Donnie Berkholz, an analyst at Redmonk. “Like any change, it’s going to be a lot easier for greenfield teams and projects, so new startups will have an easier time starting from scratch with a DevOps mindset. And there’s also no guarantee that SMBs are new — in fact many of them have been around for a long time and have similar inertia, relative to their sizes, as large enterprises do,” says Berkholz.

And, as Berkholz explains, one of the big challenges for companies considering DevOps is whether the benefits of the move exceed the switching costs. “Fortunately we’ve got growing data, such as the Puppet Labs survey, showing that DevOps greatly accelerates time to delivery, although quantifying “first to market” in terms of dollars can be challenging,” he says.

Bigger ships turn more slowly, but they still turn
Jeff Sussna, founder of the IT service innovation consultancy Ingineering.IT, concurs and says change always happens more slowly in enterprises than SMB’s, simply because they’re larger and more complex. And just because enterprises are bigger ships that turn more slowly, they’re still turning, though it may be more difficult to perceive immediately. “I think people are confused and [they] think DevOps requires either smashing org charts and mushing everyone into shared teams, or else rolling out some big expensive overarching tool across the entire organization,” he says.

The reality is DevOps is manifesting when teams collaborate and partner much more closely – no matter their size within the enterprise or if such change is occurring at a sweeping pace, at first. It’s when operations teams seek developers’ input, and when developers reach out proactively to operations. When teams are working to improve continuous QA, they are performing DevOps. “I think the fundamental problem with DevOps at the moment is we think it’s a destination, when really it’s a path,” says Sussna.

And Berkholz adds that just because it’s a slower and more cautious in the enterprise doesn’t mean it’s not transformational. “Like any other organizational transformation, the path of least resistance tends to be one of incremental change rather than a wholesale shift. Switching one tool at a time to a tool chain that encourages DevOps-style behavior and discourages silos can help catalyze that transformation. Continuous integration and continuous delivery/deployment pipelines are great examples of this,” he says.

And those are exactly the kinds of transformations underway at the enterprises we speak with every day.

Big rivers often begin as small streams
As for the business benefits, many organizations view it as a matter of survival. “I think over the years folks have been amazed how long (not to mention how expensive) it has taken just to bring basic levels of good, let alone best practice in to IT. There have been more than enough failed Bataan Death March style best “implementations” to make any executive wary of the next new thing. In my opinion this is the real opportunity cost of old scientifically disproven mechanical metaphors for human organizations,” says Kevin Behr, chief science officer, at enterprise opsflow consultancy Praxis Flow, and co-author of The Phoenix Project and Visable Ops.

“I believe if your org can not improve at the individual level it is incapable of getting much from DevOps, or anything else for that matter,” Behr says. That is why I have focused on first evolving some sort of continuous improvement form or kata using the scientific method as its base. This capability starts day one helping to create resilience, which is the ability to handle change. With that you can “improve into DevOps” iteratively and within the guardrails of your organizations overall direction and strategy. We call this approach Opsflow, as you’re flowing Ops in to Dev, then to test, then to infosec, and then on to the rest of the business,” says Behr.

And the data increasingly show that the move to DevOps is in fact well underway. And happening it already is, at least according to numerous and recent studies. One is the Vanson Bourne study (funded by CA Technologies) of 1,300 senior IT decision-makers, at enterprises with $100 million in revenue, and within financial services, healthcare, manufacturing, public sector and telecommunications industries from 21 countries around the world. In that study, a decisive 99% see a greater need to move foreword with DevOps.
Additionally, the 2013 State of DevOps Report by Puppet Labs found that the majority of those that have implemented DevOps have found significant benefits including improved quality of software deployments, more frequent software releases, improved visibility into IT processes, cultural change, and IT being more responsive to business needs.

May 28, 2014
DevOps developers; don’t be a DevGoof

There was a lot of brush back last week over Jeff Knupp’s post about how DevOps is killing the developer. Frankly I wasn’t shocked by Knupp’s opinion that developers “are the dentists in the dental office”. In my 30+ years of involvement in the IT industry my experience is that developers often think of themselves as the only smart person in the room. Often times that is the reason they are are the only person in the room, lol. The fact is many developers relate better to code than to people. For me all this talk about DevOps killing the developer is akin to saying power steering killed the driver. Just because you can do more with new technology, does not mean one dies, one just has to pivot and in this case become business social.

Now the sociable part may be the problem. Sometimes I feel that DevOps is trying to create a love fest between Ops, Developers, QA, security and the rest of the IT stack. While they may not have to love each other, I do think they all need to be nice to each other as opposed to ignoring or just going around each other, as was all too often done in the past. A common scenario I have seen for example is for developers to rollover Ops by stating this is mission critical for the business. These tactics will no longer work to let developers have their way.

Developers need to realize they need to become a normal part of the business social structure. They will be required to talk and work with people, not throw tantrums when they don’t get their way. Left unchecked, certain developer personality types can sabotage your project, or worse, make themselves no fun to work with. I am sure we have all met and unfortunately had to work with people like that. In my career, I’ve encountered virtually every personality type. In fact, I’ve also knowingly hired all of them. They know who they are; they are the developer that if challenged or worse accurately taken to task for lack of being part of the team; quit cause they can’t manage the confrontation, thus interpersonal stress. In the old days they would not come out of their house for weeks continually playing Leisure Suit Larry, if your not familiar with it Google it

I remember one of my good friends telling me this story that sums this whole developer persona issue perfectly. This company, we will call DevGoof, was a well funded start up. Every quarter they held a meeting with all the employees to let them know how the company was doing. Most employees had some kind of equity play in the company, so everyone was incented and theoretically interested in company performance. Several quarterly meetings came and went and not one developer showed up. Eventually DevGoof management got concerned and called a mandatory meeting for all developers’.

DevGoof management put out some pastries, juice and coffee for the brain trust. Then proceeded to tell them why these meetings were important to them and their fellow employees. Notably pointing out how it is not just about them but also about the rest of the DevGoof team. This meeting went on for about one hour and not one developer asked a question. So eventually this meeting was over and everyone went back to developing. The next morning, about 50% of the developers quit and some in a not so professional fashion.

The moral of this blog is the DevOps is going to force the development community to become more interpersonal within the business environment. This will lead to personal success within the new world of DevOps. The DevOps Developer/Engineer that can demonstrate leadership and compassion for team will be the one leading the DevOps way within the organization. Don’t be a Devgoof!

If you would like say something; email parker@devops.com and don’t forget to comment here.

April 23, 2014
Technology’s effect on religion

Today being Good Friday, with Easter being Sunday and in the midst of Passover; it is a time for reflection. Instead of thinking about DevOps, today I ponder life and religion. What comes to mind is the future of religion in these technological times. The number of Americans with a religious affiliation continues to decline, less and less of us make an appearance at a Synagogue, Church or any house of worship for that matter. In my childhood we would at the very least make an appearance a few holidays a year, notably at the annual pancake breakfast.

From the calculator to the computer, from mobile devices and the Internet, advances in technology are creating monumental changes in our lives and in turn the way religion is practiced. The universal acceptance of mobile devices, like the iPhone and iPad, is provoking a profound social change. The dependence on these devices is eliminating the individuals need for traditional religion and going to traditional houses of worship. The problem is, without the traditional religious brick and mortar upbringing, the youth of today forgo a key element of the religious experience. Religion was never just about a supreme being, the social gathering and interaction of those with similar beliefs and values is also a critical part of the experience. Religion is losing the youth.

It seems there are now thousands of apps which allow parishioners to bring Bible quotes, Torah-chanting practice and Buddhist prayer wheels anywhere, anytime day or night. My concern is how many of our youth are downloading religious apps as opposed to angry birds. The challenge is how to mobilize religion while still achieving all the religious personal support and socialization benefits that are realized from real life interaction in attending a place of worship. Is the virtual congregation a realistic way to gather people and practice religion? I have a hard time thinking the young will embrace religion without the real life experience which creates the feeling of being at peace with oneself.

The virtual congregation will be effective fill in for those who have already experienced religion in the traditional sense. Worshippers who cannot find their way to their house of worship for services will pray online with their fellow believers via the latest technology, confident that the other worshipers are doing the same on their devices. How powerful that will be is the question, it has been relatively effective via Television. But those are already followers seeking religious celebrities, not new believers.

There is no denying religious leaders who conduct services via technology to a geographically wide spread audience will notice a significant difference in how worshippers experience the event. Religion is very much an emotional event and technology can be a non starter for the holy rollers. Gathering around technology in smaller groups may be an answer.

The religious apathy of the young produces a serious problem for religious leaders looking to grow shrinking congregations. The lack of formal religious upbringing has lost many of the young and younger worshippers that do exist expect any religious event to embrace technology use. Device multitasking has become such a pervasive part of their life that quiet, hard cover book based religious ceremonies seem awkward and therefore not compelling.

Technology may represent the end of traditional religious worship, and the beginning of a new age of worship, the form of which no one can reasonably predict or control. The bottom line is if you look at technology and say how does this change people going to house of worship? You miss the point, because religion is a much deeper set of beliefs. The question is without new young followers, what does the future hold for organized religions?

If you would like say something; email parker@devops.com and don’t forget to comment here.

April 18, 2014
An outage war room primer

One aspect of the DevOps movement I’ve seen adopted at numerous companies is the idea that everyone supports their products by being on-call for any incidents that occur in the production environment. This responsibility often leads to participation in the outage war room. For those of you who may be new to this experience, I’m offering this primer from a grey-hair ops guy who’s served countless hours in every manner of war room you could imagine, some that stretch the bounds of that imagination, and a few that I just still can’t talk about. Been there. Done that. Still doing it.

In all that time, however, I’ve observed four distinct war room dynamics that can cause conflict. Understanding how that conflict arises is key to mitigating it.

The war room is a stressful place

This statement sounds obvious, but it’s fundamental to understanding the tensions that can arise in a war room. The clock is ticking. In fact, the clock has already been ticking for some time when you join the room. These things don’t happen instantaneously. When you get the call to join a war room, the problem has already manifested and is causing an impact. Even with automation, it takes time to notify people and have them connect to the war room, in whatever form that takes. In environments with a very high SLA target, the set up time for the war room alone can exceed the monthly allowance for downtime. So once you get into the war room, it’s serious business. The clock keeps ticking.

The war room is often a stressful place as a result. Tensions, and even tempers, can run high. But each and every person in the war room has an obligation to keep themselves under control. Tension and stress feed on each other, and if you succumb, it makes it worse for everyone. Even more important, if you get angry, emotional, or display some other decidedly not cool way of coping with the situation, you are likely to lose the confidence of your peers. That really sucks, so it’s important to force yourself to project calm confidence and be deliberate and measured in your actions. If you have the opportunity (and the ability) and can help to disperse some of that tension, do it. Those are the kinds of moments that build credibility.

And there is another very personal connection to the outage that can be stressful. A professional, worst-case scenario. In the worst case, you are actually responsible for the whole mess. When that happens, you are going to be in your own special hell in the war room. But you should know exactly what is expected of you – the one thing you can do that will really make a difference. You need to tell everyone exactly what you did, without skipping anything, and without regard to how stupid it might sound in hindsight. As in, “I continued to run through all the commands in the procedure even though they were failing with errors and even though I had guidance to stop and seek help if they failed.” To help prop you up in this most difficult of tasks, consider this: everyone else in the room has done the same thing. We’ve all been there. If we haven’t, it’s probably just a matter of time. So you have to swallow your pride, tell the whole story and then get actively involved in fixing things, because that’s what the war room is all about.

The goal of the war room is to restore service as quickly as possible

A war room is bad. If you are in one, it probably means that your customers are impacted. You have an obligation, working with the rest of the team, to recover the service in the most effective and least risky way. In a perfect world, that method is also fast, but life doesn’t always work out that way. If there is no obvious and simple solution, though, things can slow down a bit and we see different approaches forming across groups of people in the war room. Some, traditionally the operations staff, want a way to restore service quickly by a reboot, reset or other “big red button” kind of operation. Others feel it’s important to fully understand the nature of the problem before taking action. These are clearly often opposing goals.

One way this manifests in war rooms is when, in order to further analyze the problem, someone asks to run some experiments, or capture data, or otherwise use the production environment to debug the problem. If there is no method available to recover service with a reasonable level of risk, such experiments may be justified. Otherwise, the team has an obligation to execute whatever procedure they have to bring the system back up, even if that destroys information that might be helpful to the root cause investigation.

Another thing I see happen frequently is when the someone jumps the gun in trying to fix things. It plays out like this:

Jane: Do you think we should reboot the server?

Dick: I’ll get set up for it.

Jane: I think we should do it.

Dick: Done!

Jane: Wait…

A good war room manager will easily spot and work through these kinds of conflicts. If there is such a person on your war room, listen to them and obey them. The war rooms I’ve participated in are not democracies. They are run by a strong leader who is empowered to make extremely critical decisions. It’s crucial to respect that authority in the heat of the moment rather than try to be the hero. Not so coincidentally, this is my next point.

We need solutions, not heroes

In a war room, you need to separate your ego from your idea. The war room does not need heroes. The war room needs solutions. Be advised that no matter what part of the organization you come from, your idea is probably going to go up against some alternative ideas. They will not be judged, generally speaking, on how clever or elegant they are. They will be judged on how effectively they can recover the service and how much risk there is. And if yours is not the chosen idea, it does not mean that you suck. The only people who really suck in a war room are those who interfere with the process of service recovery. You know, the people who interrupt the flow of a critical conversation to ask for a status update, or who won’t stop debating a relatively minor point when the team really needs to focus on a larger issue. That kind of thing. But I digress.

Sometimes, the right path forward does not actually address the root cause of the problem.

Sometimes, the right path forward precludes ever figuring out what the root cause is.

Heroes have a hard time coping with that.

You must accept that you may never know why

The incident and problem management processes in most reasonably mature organizations are designed to recover the service, find the root cause and then ensure that it never happens again. But reality is often different than these goals. Incidents sometimes last a lot longer than you like, as when you discover that something has been broken for a long time and you did not know about it. And we should all know by this point that sometimes you never really know why it happened. You just can’t figure it out. It sucks, but again, it doesn’t mean that you suck. It means that the cost of determining the root cause has crossed the point at which it is feasible to continue, given the impact of the problem when it manifests. That’s just common business acumen, but it’s worth stating. Because when you realize that this is a possible outcome, it can give you a better perspective on evaluating your options. Suddenly, restarting the system and walking away from any run-time state information that might be helpful in root cause determination seems more viable. And those are the kinds of trade-offs that everyone in the war room has to come to terms with.

Conclusion

It sucks being called into a war room due to an outage. It is the very definition of failure in the SaaS world. But you can make a bad situation into a great and positive experience if you handle it properly. You can take an event that should mean that you suck and turn it into something that demonstrates your excellence. Being an effective member of the war room is a way for you, as an individual contributor, to demonstrate that awesomeness. You do it through calm, reasoned action and sound leadership. That builds confidence and trust. I guarantee you that your peers in the war room will notice and respect you for it and your customers will benefit from it. Let’s not let any of them down.

April 15, 2014
DevOps Needs a Tsunami To Jump The Chasm

All start up business endeavors must go through the phase of crossing the chasm. Most of those business endeavors usually fall prey to the chasm. The chasm is this black hole that everyone claims to understand but no one truly does. We have all seen businesses cross the chasm or fail to cross the chasm and fall into the black hole. But no one seems to understand why and how that crossing happens. The bestselling book by Geoffrey A. Moore titled “Crossing the Chasm,” is all about the heart and soul required to get early stage technology across the chasm, from early adopters to mainstream customers.

There is a big difference between people who are enthusiastic to try leading edge technologies and the rest of the inhabitants, who tend to be much more conservative. The reasons that people spend time trying to figure out how to cross or jump, if you will, the chasm is because that black hole is what stands between them and making their fortune. The early adopters are just a small sample of the market and there is no money in building a business or technology around them. To make real money, you need to cross over into the mainstream.

In Geoff’s model, the innovators are all about technology. You know those people, when a new iphone is released, they spend days standing on line to get latest and greatest new technology. The early adopters are the visionaries that are always on the bleeding edge. The mainstream are the pragmatists, the people who practice risk avoidance. These pragmatists are the key to jumping the black hole, they move the needle.

DevOps first order of business was getting on the curve and to that end DevOps has evangelists. But, understand that what worked in converting these early adopters probably won’t work for the pragmatists. These DevOps evangelists and visionaries want DevOps power tools, as opposed to the pragmatists who simply want to push the button and say “that was easy”.

By all appearances DevOps has been accepted by the early adoption set. Just how does DevOps cross the chasm? Jumping the black hole after all can be quite a daunting task. To do so DevOps has to create what I refer to as the tsunami effect.

The Tsunami effect is where DevOps gains acceptance in multiple industries or markets in what seems like a gold rush, it’s here that fortunes are made. More accurately, it’s where cult communities transform overnight into mainstream communities. Many technologies simply create a single tornado which remains in a silo forever never gaining enough momentum to really emerge. Another common path what I call an earthquake, there is a lot of hype but it tends to fizzle out quickly, if not tended to properly. Right now DevOps has all the appearances of an earthquake, Devops evangelists have created a lot of buzz, but acceptance beyond a set of power tools is still a huge question.

Considering DevOps is in the earthquake phase, as opposed to the tornado phase, DevOps cannot be successful converting one group at a time. The strategy in the tornado phase is convert industries and markets one at a time, via tornados forming within industries or markets. The tactical execution is to get smaller industries or markets committed and therefore the larger industries or markets will then embrace the movement. That is simply crossing the chasm. DevOps has too many cultural implications to convert believers over a long period of time; DevOps requires reorganization, not a subtle shift.

Since I consider DevOps to be in the earthquake phase, crossing the chasm is not an option. DevOps must jump the chasm in a surge of overwhelming momentum, hence creating a tsunami. DevOps has created a very closely knit culture, DevOps needs to push hard to create a tsunami and flood the pragmatic population. But, be advised, DevOps has to be all but universally accepted, there is no almost in bringing together Dev and Ops. But if DevOps wins, it wins big, DevOps will go from being a niche culture to a truly mainstream practice.

So what’s the strategy to create a tsunami? And more importantly, how do you move beyond the earthquake phase? Here’s how I see it according to Geoff, with my DevOps spin: First, DevOps must clearly establish that the mission-critical software development process is broken and is causing management enough pain that it would gamble on unproven methodology that is populated with a bunch of prima donnas. I am not convinced the DevOps cult has clearly established this as fact. Is the development and operations process broken or have just new power tools been created to improve speed and accuracy? Second, DevOps needs to present the total solution, not just tools. Overwhelm the community; each new DevOps tool should leverage the success and technology of the last hence creating a methodology. Even more importantly, because the DevOps tools are closely related, credibility can cross industries and markets and create the momentum required for a tsunami. Once Devops has reached critical mass by providing the total solution in a few industries or markets simultaneously the tsunami is created and DevOps has jumped the chasm.

The tsunami will emerge suddenly with everyone seeming to adopt DevOps all at once. This will result in explosive growth and will pull, willingly or unwillingly, everyone into its path. It’s driven and sustained by the pragmatists. Here, everybody has to get the latest thing all at once; the masses have been converted to DevOps. DevOps needs mass-market success, DevOps needs to become a tsunami and everyone will want to play DevOps.

Now that you know the formula, go forward and create the perfect storm.

If you would like to say something; email parker@devops.com and don’t forget to comment here.

April 10, 2014
No DevOps Roadmap? No Problem.

I’m a huge fan of The Phoenix Project and I’m honored to count Gene Kim both as one of my professional inspirations and also as a personal friend. There is, however, a dangerous precipice inside of the Phoenix Project that, if misunderstood, could cause many DevOps practitioners and enterprise IT shops a daunting problem in adopting and progressing in the ways of DevOps.

Roadmap or Roadblock?

One of the main concepts discussed in both Kim’s The Phoenix Project and Goldratt’s The Goal is “the bottleneck” and what what to do about it. The basic premise explained is that all actions to increase throughput before a bottleneck, pile work up at the bottleneck and all actions to increase throughput after a bottleneck create a void of work to address after the bottleneck. The logical conclusion from this idea is that: in order to have a net positive effect on throughput, action must be taken at the, holistically viewed, tightest constraint on throughput within the operational system.

This concept model is both accurate and valuable as the IT shop and the manufacturing floor have a lot in common. The power of this concept model, also happens to be where the aforementioned dangerous precipice is revealed – getting alignment on which bottlenecks to focus on can turn into a multi-month effort in many enterprises which can require consultants, workshops, deep-dives, process studies and other activities that are at once highly valuable and not always likely to be approved by a shop that has not fully bought in to the three ways of DevOps.

One of the lesser understood secrets of corporate america is that even when you have formal authority, permission is required to do almost anything of substance. One “no” from the wrong party can kill even the most promising initiatives. This “veto effect” doesn’t require an absolute “no” either. Much like a parent saying “let me think about it”, the veto effect only requires the absence of a “yes”. This absence can take the form of “maybe”, “let me think about it”, “let’s wait and see”, or even a “run this by the other department heads and see what they think”.

Given that any reasonably sophisticated roadmap takes more than a few weeks to develop, vett and roadshow around a medium to large enterprise, a mature DevOps roadmap suffers from a classic chicken/egg paradox for many organizations. I extend my deepest sympathies to those who have found that a clear and easily understood roadmap can somehow both provide and require the catalyst to consensus and permission to proceed in the ways of DevOps. You can take heart, however, in the fact that neither you, nor DevOps, are alone in this conundrum as this is what can be referred to as “the corporate condition” that we must all bear.

While company may not always provide solace to the miserable, there is however, another way.

Just Follow the Sun

What is this one side road still available to you that has the power to relieve almost any non-alignment traffic jam? Start heading west! Just like the American pioneers, you can start following the sun to the wide open spaces of DevOps land where automation is plentiful and provable performance gains via instrumentation are gold. All you need is one or two comrades with some skills worthy of the road ahead and you can be off!

Just because you can’t get the unix team to automate environment creation, doesn’t mean you can’t head west and automate your build processes. Just because you can’t automate your entire production deployment doesn’t mean you can’t head west and automate your performance testing and benchmarking processes. Just because you can’t get everyone to agree on the tightest constraint, doesn’t mean you can’t head west and alleviate the ones you already know about. To head west on any of these things, you don’t need full concensus, you only need a traveling partner or two.

The Trail Has Already Been Marked

For mid to large size enterprises, a true DevOps transformation happens over the course of three or more years. Allowing a lack of consensus to become a total standstill is not the right answer. The tool at your disposal to overcome the veto affect is public consciousness.

Take UX design principles for example. It was less than a decade ago that design activities were considered a luxury item that produced pretty pictures; and then Apple happened and tipped public consciousness over. How many people are willing to casually veto design now?

When public consciousness to the value of DevOps tips over, it will no longer be you going to the powers that be to get the “OK to proceed”. It will be them coming to you and asking “what is taking so long?”. In that one moment, you’ll be able to fire back at them: “Here’s my roadmapping proposal. I’d love to have you be my business owner!”

April 9, 2014
Are the Milliseconds Worth It?

The pressure to prove one’s coding prowess can make a developer do some very clever and creative things with their source code. Sometimes it’s just a unique approach, but every once in a while this cleverness may shave a few milliseconds off of some performance metrics as well. In the end it’s not worth it. While this may seem like the type of innovation we want from our peers I’ve come to the conclusion it often does more harm than good. A quote that ran across my twitter feed a few weeks ago sums it up nicely: “So if you’re as clever as you can be when you write it, how will you ever debug it?” This pearl of wisdom comes from a book called “The Elements of Programming Style”. It’s a programming book from 1978 that is still entirely relevant. There’s a Wikipedia article you can check out if you’re not ready to take my word for it.

I know I’ve coded something in a unique fashion, and was quite sure I was being brilliant at the time, only to go back later and have absolutely no clue why I wrote the code the way I did. If I added comments I may be able to find my place again but sometimes it takes a while. What about when another developer runs into one of these bright ideas? Will any amount of meditation or contemplation allow them to make sense of what I was doing when I wrote it? Should a new or junior developer be expected to have some out of body experience before being able to work with me? Of course not. I hate it when I do this to myself, never mind when I run across someone else’s flash of genius. I certainly try to keep other devs from going through this type of mind freak as often as I can.

While this exercise can make for a fun parlor game and help extend our knowledge of the language we’re coding in it causes more problems than it solves in a team environment. A wiser developer sticks with a tried and true approach. They’ve been around long enough to know that the code needs to make sense to any garden variety developer that may be working with it in the future (which may in fact be the developer that wrote it in the first place).

I first learned this lesson years ago while tutoring mathematics. I had a developed an algorithm to generate the unit circle. Computer Science majors I was tutoring in math had an easier time using this to quickly generate the circle on scratch paper than arbitrarily memorizing it or understanding the standard proof that shows a bunch of triangles somehow morphing into a circle.

Radian Values of the Unit Circle

The algorithm requires a certain amount of seed data. One must know that a full circle is 2π radians in circumference. This puts the other quarter circle measures at π/2, π, 3π/4 respectively. Only a simple understanding of fractions is required to see this. Next we need to fill the first quadrant, which is only three more values, π/6, π/4, and π/3. From here the algorithm helps us extrapolate the remaining radian values for the circle.

First Quadrant: π/6, π/4, and π/3 (initial seed values, you have to remember these)

Second Quadrant: π – First Quadrant Corollary Value ( e.g., 3π/3 – π/3 = 2π/3, our 2nd quadrant value)

Third Quadrant: π + First Quadrant Corollary Value ( e.g., 3π/3 + π/3 = 4π/3, our 3rd quadrant value)

Fourth Quadrant: 2π – First Quadrant Corollary Value (e.g., 6π/3 – π/3 = 5π/3, our 4th quadrant value)

So, the formula for quadrants two through four can take the form of (n – s), (n + s), and (2n -s), where n = π and ‘s’ is one of the three seed values from the first quadrant. Three simple equations that any CS major should be entirely comfortable with using. There are a few other notes on this proof to get the coordinates, signs, and degree measurements correct. This is not a post about mathematics however, so I’ll leave that as a exercise for the more mathematically inclined to complete on their own time.

Other mathematicians where not as impressed as the CS majors and the student athletes were with this. Watching students that actively despised math struggle with memorizing the unit circle in an effort to complete their core was part of the fun of being good with numbers. It was suggested (by some troll of a jerk) that I might be a math noob myself, some poser that wasn’t any smarter than the fools he was tutoring. Impostor syndrome set in and I I felt I had no choice but to prove my mathematical prowess.

First I went big and reproved what I was doing with a Klien Four-Group. The trolls were quick to point out that this, like my first model, wasn’t a circle at all but just a bunch of points on a plane. Next I came back with a version using a rotational matrix. This did in fact generate a full circle with any value you like on it. The trolls had no more jabs for me but they also fell short of any compliment or encouragement for the work I had done.

As for the students I was helping, none of them quite understood the issue I had encountered. They viewed those trolls as nothing but trolls. Peers that only laughed at their struggle and refused to even try to help them. I ultimately found out my students were concerned that I had gotten sucked into this quagmire. The research I was doing was beyond their understanding and they were still using the first model I developed. It did exactly what they needed it to do. I was informed I should take the rising grades as evidence that my work was of merit and move on to a more productive use of my time and energy.

It was only then I noticed the trolls did have a point. The Klien Four-Group model and my original algorithm weren’t circles at all, just dots with values attached to them. A mathematician would have issue with this where a developer would not, since an object oriented developer should have a deeply rooted appreciation for abstraction. A class called Tree rarely resembles what we know to be a tree. It’s the use case that ends up being important. To a paper company a tree only needs to be represented by the amount of pulp they can expect it to produce. To an apple farmer it probably boils down to the number of apples the tree puts out and maybe its location in an orchard. What we take to be a tree, its size, the shape of its leaves, how it smells, the sound it makes when the wind catches it, are all details beyond the scope of most software we’ll ever write.

This happens to be a huge blocker for most new OOP developers, especially the ones that are good at math. The object model I was using to teach CS students how to do trigonometry ended up being the perfect tool to teach mathematicians how object oriented programming works. The rotational matrix version is important because sometimes non-standard values are needed. It’s useful to show how a method call may use standard values or extend onto another model, like the rotational matrix, when standard values don’t apply.

Mathematicians appreciate the Klien Four-Group model and I’m happy to show it off when I have the chance. More importantly, it’s a reminder to not get too fancy with math they’re putting into software. The math is beautiful more than it’s clever, and I’ve written code I feel the same way about. Like most works of art it stems from a hobby and should be left out of any applied scenario if it’s not practical. If we can get the same result with something that’s simple, something that’s easy to understand, easy to use, and even common, then that’s what we should use every single time because it will be easy to maintain. Wisdom is nothing more than knowledge applied over time. Time and time again I’ve come to the conclusion that maintainability trumps performance. We should be quick to sacrifice any minimal performance gain that makes our code difficult to understand. If a major performance gain is within reach then we’d be wise to abstract any strange code that was used to obtain it. I’d also advise you to comment the living day lights out of it as well, since you may need to remember how it worked yourself someday.

April 8, 2014
The Importance of the Bus Driver

The topic of DevOps has been a topic for many a debate over recent ~~months~~ years. Without adding fuel to the fire I’d like to present a slightly different take on the DevOps culture.

The phrase ‘what if I was hit by a bus?’ can often be heard within the vicinity of software development teams. It relates to a measurement known as the ‘bus factor‘. Individual talent should not be discouraged, however when it becomes silo’d that talent translates to risk for a business. Try to think back over your career – do the following sound familiar?

“I don’t know how to do that, X usually does that”

“X is away at the moment so we’ll need to put that on hold”

“X left yesterday and didn’t cover this in their handover”

If you have read ‘Phoenix Project‘ – think of Brent…

Using the fated bus driver I’d like to take us on a journey (excuse the pun) through a DevOps culture.

In-sourcing the bus driver

You work for a small green minded startup. Being green minded and wanting to help your staff, you decide to buy a bus to get everyone to work in the morning. Realising that you don’t actually have a dedicated bus driver, you decide to take it in turns to drive the bus in the morning. Everyone gets involved, everyone knows how to drive the bus, everyone knows which route is best.

Idea 1

Don’t over engineer your processes. At this stage you’re already exhibiting DevOps characteristics!

The talented bus driver

Your small team has performed really well, people like what you do and now your team isn’t quite so small. There are now over 50 people on your bus in the morning. It wasn’t long until you realised that Sarah is the best bus driver you have. Inevitably, Sarah usually just takes up driving the bus each and every morning.

If Sarah is off, Mike takes over but he often makes people late for work and tends to hit a lot of parked cars.

Furthermore the bus recently broke down when Sarah was off, Mike and your bus passengers all took a look at the engine and tried to help but there seemed to be a ‘magic’ piece of string holding two parts together.

When they consulted the bus manufacturer manual, this piece of string was no where to be seen. Mike had to call Sarah whilst on holiday!

Mike calls the bus company to ask for a replacement bus, to be told it will be 2 hours until a second bus arrives. The team sit and wait patiently.

Idea 2

Try to identify your tipping point. Some key indicators are reliance on individuals, information silo’s, phrases such as ‘oh I didn’t know it functioned like that in production’, production environments start to diverge (without communication) from development environments.

How many production environments have the web server, app server, database, load balancer et al all running on the same machine? Is this how your development teams work? Does this differ to the view of the operations team?

Your bus stops

We’ve identified a few problems but lets focus on the positives. Firstly, let’s celebrate actually spotting the problem, it is very easy to continue with the status quo. Our second success was the reaction of our bus passengers. When the bus broke down, instead of complaining, the bus passengers tried to help mend the bus.

Idea 3

When and where your bus stops is less important than who it stops for. It might be stating the obvious, however if you wish to move towards a DevOps culture you need talented team members. You need people that care for the organisation and their profession. A desire to share knowledge for a greater goal.

If your team are not interested or do not care, then no matter how much you say DevOps you need to boost their interest first.

Hopefully you’ll start to hear your teams utter phrases such as “What does production look like?”, “You could do this slightly differently you know”…

What if your talented bus driver has an RTA?

There are no doubting Sarah’s talents – I mean come on, she was able to fix a bus engine with a piece of string, knew the best bus routes and always got your team to work on time…

What would we do if Sarah left or was, somewhat ironically, hit by a bus?

Idea 4

Could your organisation raise the value of sharing and encourage knowledge share?

Assuming you’ve got the right people on the bus, you might also find that other staff members have ideas for improvement when you explain the current process. Your Sarah(s) should not fear sharing their knowledge and should be rewarded/encouraged for doing so.

One of your passengers may even know a faster journey to work or a better way to fix the bus engine.

The bus seating plan

Although you didn’t enforce it, you start to find groups forming on your bus journey, all your teenagers are sat together at the back playing music on their phones…..

Idea 5

Don’t let job titles restrict conversations. Water cooler chats are incredibly productive and help teams share ideas.

Replacement bus service

When the bus broke down Mike was able to order a replacement but it had a 2 hour turnaround time.

Would it be better if we always had 2 or 3 buses running? If each bus had enough capacity for all 50 passengers we’d probably always make sure people were on time for work.

It would be even better if all bus drivers knew the routes that Sarah used, all buses had the same piece of magic string and all bus drivers knew what the magic string did.

Idea 6

Something my team often hear me say is ‘Design for failure, don’t just react to it‘. When designing your applications and services try to think how the system can accommodate failing services better rather than what actions you might take in the event of failure.

The same might be said for your human resources. Have your teams got a shared knowledge? What would your bus factor score be?

Journey times

Try to keep your passengers informed of your progress – you could let them know their ETA for work. Informed passengers tend to be more engaged and understanding. For example when telephoning a business, do you prefer to know what position you are in ‘the queue’ or would you rather sit on the line waiting without knowledge of when you may be spoken to?

Idea 7

Monitoring might be considered a key technical section of DevOps adoption. There are many, many tools that can help with this such as StatsD, Graphite, Librato, New relic, AppDynamics etc but whichever medium you choose, try to make sure your ops stats are in full view of your various teams not simply your ops staff. Get them in front of your dev team, support team, sales team, HR team they impact every member of staff.

Hopefully, you will be surprised at an increased level of employee engagement. When ops are in war rooms fire fighting problems, each team member will understand the pressures they are under, some team members will also express a desire to help.

You have arrived at your destination

At this stage I may have flogged the bus concept a little too far. Are the concepts here a guide for the adoption of a DevOps culture? Does the ‘D’ word have to be attached to the guidelines above? I’ll let the community decide.

April 8, 2014
A Critical First Step in Bringing Dev and Ops Together

We all want to deliver excellent software to our customers. We now have a greater awareness than ever before of the interconnectedness of all the activities required to do so. The agile movement started this but it was not until the advent of DevOps that Ops was considered part of the full picture. In this article I will explain an effective first step that agile organizations can take to complete the picture and bring ops more completely into the product development process.

Delivering excellent software starts with product definition and flows from those initial stories and designs, into dev, verification and testing, into production and then back again in the form of customer feedback and bugs. In reality the feedback paths represent a transitive closure of all activities just mentioned as you can see in the diagram below. Agile recognized the importance of feedback loops in good product development. Agile also realized that for the right information to flow, walls had to be broken down. The motto “Done means tested” arose and indicated that product owners were not done with a story, could not consider it closed, until it had been tested and they had verified it. This represented the integration of PM, development and QA functions. A huge step, but one step too short. Operations was still not viewed as a key part of building excellent software and there was not much insight into the fact that the feedback loop did not stop at testing. DevOps has recognized this shortfall and taken it to its logical conclusion – “Done means in production.”

Product development is a cycle, a cycle that operations is a critical part of. In describing how all parts of the cycle can be integrated lets start at the most natural beginning; product definition. Product definition activities produce stories, designs, functional acceptance criteria (A.C.) and prioritization. In this phase we don’t produce all we need for the finished product but just enough to get us going. With popular agile methodologies like Scrum this is ideally not done solely by a product owner but instead in conjunction with dev and test at story grooming meetings, design reviews and sprint planning. In these meetings dev, QA and product communicate and negotiate stories, comment on designs, and flesh out acceptance criteria. In this way the stories, designs and acceptance criteria are improved and everyone gains an understanding of what is being built; they also become invested in it. A story may look something like this (abbreviated):

Story: Blog administrator can change user roles SO THAT users can be managed effectively

A.C: Given blog administrator is logged in
And on a user profile
When he selects a permission scheme from a permissions drop-down menu
Then the users permissions are updated and stored in the DB

This is a great start but it is not enough. If we are to be excellent we must modernize this just a bit and add in some DevOps spice.

A first step towards bringing ops effectively into the cycle is to include someone responsible for actually operating the software in the story and acceptance criteria grooming sessions, design reviews, and sprint planning meetings. The critical aspect is that when the team negotiates the acceptance criteria associated with a story it will be ops responsibility to ensure we for each story or feature the team adds success and failure metrics, initial alarm values and any operational functionality such as a kill switch that will be needed in order to operate the functionality effectively. So a story will now contain things like the following in the A.C. (acceptance criteria).

Given operation X fails
Then the metric operation-x:failed will be sent through the metrics feed

We will also alter our definition of done to make sure that a story will not be closed until we meet the ops driven acceptance criteria. For example: we can see the metrics we have defined in our A.C. show up on an ops dashboard. There are so many benefits to this such as folks that are unlucky enough to be on on-call will have participated in this story grooming and will know exactly what these metrics mean when they encounter an issue with them for the first time in production. Furthermore, they will know what operational levers they can pull or request to be pulled.

This first step represents a connection from product definition to dev, to QA, to operations. This connection is perfectly natural and intuitive when you think about it. Being a true DevOps organization and developing operationally excellent products requires more but having connected product definition to operations as we have provides a nice toe hold from which to flesh things out further. Best of luck. If you have any questions leave them as comments and I will respond as best I can.

April 2, 2014
THOUGHT LEADERS OR NAVEL GAZERS?

I get tired of hearing what is DevOps? It just tells me that no one really has got it exactly right, but everyone longs to be the authority on the topic. Everyone writes their perceived answers ad nauseam. Reading all the different definitions is kinda like listening to infomercials. An individual puts themselves out there as an expert simply because they have an audience and the topic is in vogue.

But before I go any further let’s put the “What is DevOps?” questions to bed once and for all. Of course the only real source to turn to is Wikipedia, as that is the definitive source and because it’s on the Internet you know it has to be true. Did I tell you I was a French model; bonjour. Now to the definition you have been anxiously anticipating:

DevOps (a portmanteau of development and operations) is a software development method that stresses communication, collaboration and integration between software developers and information technology (IT) operations professionals. DevOps is a response to the interdependence of software development and IT operations.

There you have it; DevOps is a methodology not an industry. Now that that question has finally been answered, I hope that I never hear about it again, let’s turn back to those thought leaders. Let’s go back to time when many of today’s thought leaders were still asking what the internet is. Once again I refer to my expert source Wikipedia:

A thought leader can refer to an individual or firm that is recognized as an authority in a specialized field and whose expertise is sought and often rewarded. The term was coined in 1994 by Joel Kurtzman, editor-in-chief of the Booz & Co magazine Strategy & Business, and used to designate interview subjects for that magazine who had business ideas which merited attention.

Everything on the internet is about branding and the marketing of the brand. Thought leaders typically write a blog, column, book, record a podcast or webinar. As long as the message resonates they don’t care about the delivery mechanism. So called thought leaders are simply marketing their brand whether they truly have the expertise to do so or not. It is a simple process when you have a topic that is loosely defined in an early stage of development, as there is no objective measure of what is correct.

Thought leaders typically are an individual or firm that profit from being recognized as such. I submit many are truly mere navel gazers, as a navel gazer is derived from the phrase “contemplating one’s navel” or “navel-gazing” if you will, typically referred to in jocular fashion, to refer to self-absorbed pursuits. Please someone explain to me the difference between thought leaders and navel gazers?

We have so many self anointed thought leaders I am beginning to wonder is there anyone out there actually practicing DevOps or is everyone just talking about it to make a fast buck. If you are a thought leader why can’t you tell me if and when DevOps is going to cross the chasm?

What have I learned from all this? DevOps needs an infomercial to cross the chasm, not thought leaders. Don’t even get me started on chasms.

If you would like say something; email parker@devops.com and don’t forget to comment here.

April 1, 2014