devopsy.kinsta.cloud

Category: Doin’ DevOps

Moving to DevOps in a traditional enterprise
Introduction

One of the challenges of introducing DevOps in the setting of a traditional (i.e. siloed, waterfall etc) Enterprise is where to begin.

It’s certainly something I wrestled with alongside the CTO for quite some time.

It’s not possible to impart years of experience of moving a large company to a DevOps ways of thinking but I hope to give the main 3 pillars of how I’ve approach it previously.

Note…none of it mentions technology and tools! DevOps is primarily a cultural idea and therefore affects people above anything else.

Step 1 – bring in the Rock Star

If you are moving to DevOps in your organisation, then bringing together different people from different silos and then expecting them (under their own steam) to magically cooperate and agree on everything is a little unrealistic.

If you are dealing with historical teams and employees – which I suspect most of you are – then breaking down silos is likely to be a significant challenge.

Aligning incentives makes things a lot easier. If both your devs and ops guys are paid a bonus depending on how much “change” they do in a year is a simple way of getting them to agree on whether they should automate releases for example. That’s a little too simplistic but hopefully you get what I mean.

This may well come at the cost of something else though. The first thing that springs to mind in the last example is quality.

Let’s put everything straight into live! We get paid for it!

Ok so that isn’t going to work.

Sometimes, at least to begin with, you need to bring people together and ultimately empower someone to make decisions when one can’t be reached through consensus. I’ve seen a real danger in DevOps of death by committee. The more silos you bring together then potentially the greater the possibility of them not making a decision.

You may have heard about this idea of DevOps rockstars. Guys (and gals) that get DevOps inside and out. People that understand all sides of the coin (development, DBA, infrastructure etc).

They try and influence through education and persuasion and point people in the right direction. They help the silos reach the right decisions themselves rather than a top-down “do what I say”.

All said and done sometimes this open inclusiveness doesn’t always work and someone needs to just stand up and be counted.

So what’s my point?

If you are trying to move towards DevOps then hire a rockstar (I doubt Rod Stewart is suitable btw). Someone that has done it before. Someone with a very mature sense of what DevOps is and preferably someone that has done most of the jobs he/she is trying to influence.

Moreover, hire someone who’s first instinct is to try and bring people around to their way of thinking. And lastly, someone who isn’t ultimately afraid to put their privates on the line and make a goddam decision.

Step 2 – look at your people structure

The traditional definition of DevOps is to break down the silos between Development and Operations so that they accomplish shared goals.

However, I’m a keen advocate that DevOps isn’t just Dev and Ops (see http://enterprisedevops.blogspot.co.uk/2014/07/devbiztestreleasethingyops.html).

It’s about all the functions that come together to achieve meaningful change.

The devs, the ops guys, the business and techies should all have shared goals and incentives. Where possible they should be sat around the same desk. The team should rise and fall collectively (there is no “us and them”). The team should manage change from cradle to grave (requirements, build, test, deploy, support).

I think we should rename Devops to “DevBizTestReleaseThingyOps”. Catchy.

To that end, DevOps is as much to do with “people” than it is about new tools and processes.

With that in mind, potentially you are talking about some fairly radical changes across a wide number of people of various skills, experience and personality types. I guess it really depends on what your organisation is structured like, the skills of your team and many other factors. For my company (a large UK insurance company) however, it’s meant some fundamental changes.

We started by aligning change functions (devs, testers, BA’s etc) to individual business units (rather than monolithic shared group services).

That fundamental change has been sponsored on the back of some early and small wins by implementing things like Continuous Integration and labeling it “DevOps”. Ok so it’s not DevOps but if using that term starts to get visibility and movement within your organisation then I don’t think that’s a bad thing. I always believed we needed a banner for the modernization of the department to fly behind. If that banner reads “DevOps” then so be it.

That early small success was really vital to us.

When we started out we had a lot of push back and negativity….
- Why are you telling me how to do my job?
- You are using the wrong tools
- I don’t see the benefit
- I know more about this than you do
- What you are trying to do is easy
- We don’t have the right people
- This is a hipster idea for the likes of Facebook
- There are regulatory reasons this won’t work
Basically everything under the sun.

That’s stressful when you are trying to do the right thing and move the incumbent forward.

I don’t think we helped ourselves too much to begin with. There was too much of my team trying to pick up and change other team’s technologies and processes. Not enough leading the other team to water and helping them implement something themselves. Lesson learnt.

Step 3 – create a team to help others

I mentioned earlier that DevOps is cultural idea and not a job title. Definitely not. Unimaginable.

So how is it I run a “DevOps” team? Surely that’s a collective of “DevOps engineers”?!

My team “Platform Services” supports the idea of the change platform at my company. The change platform consists of tools such as Jenkins, Puppet, uDeploy, Teamcity, TFS and a few others.

Someone ultimately needs to “own” the tools making sure they are maintained, creating best practices and establishing a roadmap for the future.

My team goes beyond this and helps different change functions adapt to the change platform and are evangelists for things like DevOps and Continuous Delivery.

With DevOps being concerned with breaking down silos its contradictory to have a team that bottleneck of knowledge and skills. To this end, Platform Services is the silo-buster – we help others pick up the tools & processes and implement them.

For example, we don’t do deployments ourselves but we do sell the benefits to other teams and help them to adopt the tools and processes.

I think this is a good model and helps spread the success of DevOps around the IT function.

It helps that consultancy from my guys is effectively free to other parts of the IT team (thanks to an enlightened CIO & CTO).

I hope you found some of that useful. Feel free to check out my blog on my move to DevOps: http://enterprisedevops.blogspot.co.uk/

Good luck and get in contact if you have any questions or comments.
August 4, 2014
Think the Developer Tool Market Is Annoying? Just Wait…

I recently attended the Velocity Conference and DevOps Days in the Bay Area. While I usually pride myself in being “in the know” of what is available and possible. It never fails that every time I do a search on some aspect of the development process, or attend events like these, I’m shocked to discover at least three new tools I have not heard of. “What’s this… yet another functional testing tool? Let me guess, you do performance monitoring? System alerts by carrier pigeon? Cool, configuration management that also repairs your TV.” There are so many tools and their competitive advantages are usually so specific you really have to work for the company to understand them. So how do you possibly make sense of this space? And what is next?

(more…)

July 31, 2014
Will you have your IDE in a box? On a Train? With a Firefox?

As predicted the Cloud IDE race is picking up. Now I’m sure. Cloud IDE or no Cloud IDE it is not a choice. But it is an huge opportunity.

(more…)

July 28, 2014
Mixed Martial DevOps vs.The Scrum Master

Royce Grace won the most tournaments in UFC history before he retired. With a record of 14 wins, 2 losses, and 2 draws, it’s understandable why he’s considered by some to be the most influential figure in the history of modern MMA. With 100% of his finishes from submissions, he’s undeniably Brazilian Jujitsu royalty. You’d be hard pressed to find anyone familiar with the fighting world that would argue against building a strong base in BJJ while learning to fight. Royce proved that BJJ was the single most effective form of fighting in the early nineties by submitting opponents much larger than him, like Dan Severn, in UFC 4. Severn had just shy of a 100 pound weight advantage on Gracie but ended up getting caught in a triangle choke. I can remember watching the video and trying to figure out how he made him tap like that. All the sudden fighting wasn’t just about kicking and punching anymore, the game had changed forever.

Fast Forward 12 years to UFC when the Master Royce Gracie is to make his return to the octagon. Matt Hughes crumpled him up like a paper napkin. Royce had continued to practice BJJ since his fight with Dan Severn as it was a successful strategy. The mistake he made was investing in a monoculture. Royce kept a razor point on his BJJ, but fighters like Matt Hughes were smart enough to put an edge on all the dull perspectives of their game. Hughes was a well decorated submission wrestler that knew he needed to work on his striking and maintaining his strength and endurance. Ronda Rousey made it clear after her last victory that she’d been working more on her striking too, which makes sense considering she’s already an Olympic medalist in judo.

At the end of the day it’s a fighter that wins a fight, not the discipline they practice. A well rounded fighter is going to be able to adapt to adversity better than someone that only has nth level experience in a particular area. The same is true for software methodology. A scrum master’s team will consistently be outperformed by a team taking a mixed approach. Not because something is inherently wrong with scrum, but that people get the work done, not the process. A team embracing a diverse approach is more likely to do what works. They will be open to more experimentation to figure out what is successful and to confirm when the current model isn’t working. Sometimes it’s the people not cooperating, but telling them there’s a problem because they’re not fully embracing the methodology is usually wrong. Often enough the methodology is being taken as gospel and is embraced too tightly.

I both like and understand kanban, but feel scrum has too much convention to adhere to specifically. I checked Google and it seems to agree with me:

Just to be fair, I checked to see how Google felt about kanban.

But Google didn’t seem to treat kanban as harshly.

If you can’t take my word for this maybe you’ll listen to someone else?

Just the other day Andrew Clay Shafer posted this on Twitter:

I did ask Andrew to offer an alternative approach if scrum wasn’t going to work, and he directed me to slide 46 of a presentation he did in Pittsburgh for a Ruby User Group (shown below). This slide advocates a mixed approach as well. I agree with everything Andrew encourages there, and can honestly admit I need to dig into XP technical practices a little deeper myself. This will not be an abandonment of current methodology and a complete shift in another direction, but an evaluation to see if I can fill some holes in my game.

July 14, 2014
You can’t automate relationships

As an organization sold on the idea of DevOps what’s your first step in the journey?

Despite a constant deluge of well-reasoned guides for implementing DevOps the most prevalent step most companies have made is simply contributing to a proliferation of DevOps job titles.

Most likely it’s in no small part because asking the HR department to add “DevOps” to the latest sysadmin job posting title and “Chef” or “Puppet” to the requirements is easy. This is a woefully low bar to set for DevOps implementation. Compounding this further, afterwards when the CIO gets a survey asking about DevOps adoption, they gleefully indicate the company is well on their way and join many others as they artificially inflate industry statistics of a very worthwhile but nonetheless major undertaking for large companies.

Surveys can contribute to confused starts. Any survey that does not include questions on specific practices and draws its conclusions from simply asking if a company is doing a practice that has positive buzz in the industry is worthless. A welcome contrast is the Puppet Labs 2014 State of DevOps report which is quite a beneficial collection of supporting information covering various practices, recommendations and nuances of organizational structures.

Even actually making progress on an individual automation project, whether with newly titled DevOps members or from a grass roots effort in an existing engineering or operations team, does not fulfill a complete implementation strategy nor is it the best way to start. Automation is certainly vital to reliably and repeatably managing environments, but groups shouldn’t rush to tackle automation first only because it’s comfortable to start hacking away at a problem they think they can deal with on their own.

These isolated and well-intentioned practitioners may find that without buy in and solid trust and relationships across groups they may not be given access to automate and orchestrate certain key areas.

You can’t automate relationships!

There needs to be a holistic understanding of the value streams driving engineering work and how best to make improvements. Instead of diving headfirst into automation work on identifying common goals, shared pain, and potential wins across organizational groups. Starting either with collaboration between group members directly or through management level alignment can work provided the other quickly follows. Buy in is required from both engineering and management or progress will stutter.

Taking people through a story can bring support more readily than just listing bullet points. People need to feel emotionally engaged and connected to the pains of the culture and process prior to DevOps and how they can be realistically mitigated. For a large company audience, seeing a story from another large company with timeframes and phases that shepherd the transformation in bold, yet manageable steps can be helpful. The story of SAP Global IT is a candid and useful case study.

Once the stage is set then tackle automation together in a cross-functional group with shared measures of success leading your focus. Depending on the complexity of your process it may be beneficial to collaborate on a smaller application or time boxed simple proof of concept to vet new ideas and working relationships.

Despite the potential for rocky or inefficient starts, the transformative power of a meaningful DevOps implementation contributes to incredible innovation speed and satisfaction. What has worked to keep your organization keeping the long view in mind regarding DevOps?

Many companies are misappropriating the term DevOps to get attention, headcount, or perceived leverage in internal turf wars instead of building a cross-team continuous improvement engine. Is this a sign of the early peak of inflated expectations in the hype cycle?

Are we heading for a serious backlash? Is the tireless promotion of proper practices from prolific thought leaders enough to make failures clearly misguided implementations and not fundamental flaws in the movement?

I’m eager to hear your thoughts in the comments.

July 9, 2014
Coaching DevOps Baseball

Coaching DevOps Baseball

Every technology gets surrounded by hype during its maturation, and DevOps is certainly not immune to that phenomenon. If you’ve spent more than five minutes looking into it, you know that, because the current discussion sounds something like this:

“All right, there’s other stuff, but it’s just support, so I want you all to go out there and practice the important part! You only win the game if you score runs, and you only score a run if you run from third to home, so get out there and run from third to home! You’ll be the best team ever on the only part that matters, running it in!”

“What? That hitting the ball and making first, second, third base? That’s supporting stuff, doesn’t really matter, master running from third to home, that’s how you score runs, and runs win games! Oh sure, some hotshot can hit a home run, but then the team relies on the guy who can get home runs, so forget about it, just focus on running third to home!”

Yes, the vast bulk of punditry sound like this right now, and yes, it is as damaging to your DevOps efforts as this coach would be to a baseball team.

And yes, you need to care about all the other parts of the game – because it isn’t baseball if you only do one part, and it isn’t DevOps if you only do one part. No matter how much the coach thinks that part is the important part. After all, in the game, you’ll never get the chance to run third to home if you’re not able to handle the preceding parts of the game.

So when you read pundits at the moment, pay more attention to what they minimize than what they consider the end game. The benefits of putting scripts and configurations into version control, building automated testing, taking the time to write automation scripts you’ve been thinking about off and on for years (and putting those into VC also), all of these things are not only necessary prerequisites to the nirvana some foresee if only you do DevOps “right”, but they also hold benefits all their own. Harness your hotshots to automate what’s hard, and make that automation adaptable enough that it can change with the times. The “right” way to do it is methodically. Indeed, I would add the Etsy slogan “measure everything”. You can’t tell how DevOps is impacting the org if you’re not measuring things.

And doing it methodically is actually pretty easy. Pick a team or (my preference) a project, and tell them to do it. Set standards for keeping each change to configs in version control, demand that they automate as much as possible, but leave points in the process for human checking, require that automated tests are complete and run successfully before the code is considered complete (and set a standard for maintaining tests in the future, since that’s where most orgs have issues). Involve security deeply, because a failed drive, or a server dropped from a bad config can be recovered quickly, a security breech not so much. Then take what you learn, and apply what works well for your org again, perhaps to a larger team or project.

The whole point of DevOps is to improve IT’s responsiveness to changing business needs, increase operational capacity, and reduce human error. It’s not to achieve some goal set by people outside of your org, it’s to do it within the confines of your org.

So go hit some home runs, run the heck out of the first three bases, make IT more efficient and effective. Worry about things like “IT Transformation” when they come up and make sense, not when you’re just learning the rules of the game.

July 9, 2014
The Delivery Pipeline is your DevOps Signature

Where is DevOps? Searching for it is like querying a rainbow for pots of gold. No single element in a development team’s lifecycle defines them as a DevOps-enabled organization. Were anything the rainbow and the gold, it would be the delivery pipeline. The deployment pipeline centers the team and the process, functioning as the signature of any DevOps shop.

(more…)

July 8, 2014
A/B Test with Caution

When the morning news starts talking about A/B Testing of course my ears perk up, and I eat my cereal a little slower. But when it becomes a conversation about how ethical it is to use customer usage data to iteratively improve the product, it spawns this blog post.

(more…)

July 2, 2014
DevOps, Growth Hacking & other ways to rule the world

DevOps will disrupt management styles, speed the end to ancient IT practices, and flatten away processes that don’t focus on doing more with less. But DevOps is not the only dev-led movement that will impact and reshape the High-Tech industry. So called “growth hacking” will also disrupt organizational charts by bringing development tools, methods and practitioners into roles previously filled by marketing, sales, and customer success pros. As DevOps and growth hacking become more mainstream, the inevitable result is developers ruling the world.

(more…)

June 18, 2014
Environment Variables – The Ultimate Gotcha

We all think that we understand variables very well. After all, if you are a coder, a variable is something you create and use every day. But when it comes to DevOps the variable is the thing we seek to eliminate, but often neglect.

(more…)

June 12, 2014