devopsy.kinsta.cloud

Tag: development

Expanding DevOps Beyond Dev and Ops
I’ve been involved in software delivery for longer than I care to admit, and since DevOps has come onto the scene I’ve witnessed the deconstruction of silos that were once actively built to contain separate projects and team functions. Central to this deconstruction is collaboration between Dev and Ops, ushering in new innovation paradigms. What’s exciting is that in just the past five years since DevOps has started permeating organizations, other areas within the enterprise—including technical and marketing teams—have started to recognize the high-value that comes from embracing collaborative tools and processes. DevOps has quickly evolved from sideline projects to a state of mind rapidly adopted by leading companies across various business units.

DevOps and Culture

Accompanying this notion of greater collaboration and cross-silo participation is the need for new tools to help facilitate the pace of change and collaboration. Although there are no silver bullets, you can take steps and implement technologies to help make the odds of success more favorable.

Before DevOps, hostility between Dev and Ops teams was not unusual. Dev and Ops had conflicting goals: Development to introduce change and Operations to maintain stability. The common bond was a love of technology rather than delivering quality products to customers. In my experience, technology employees care more about the technology they can experiment with than the value something they’re working on brings to customers. However, in the five years that DevOps has been on the scene, those of us who follow DevOps closely have noticed staff on both sides becoming much more customer-focused.

Common Goals

Many organizations publish their core values and say that they expect employees to live up to those values. But are those values something that employees are measured against? In my experience, they are not. Organizational values tend to end up being words to make people feel good. But what if every manager in an organization had a common goal of delivering quality products or services to customers, with the goal defined in a measurable way? Then everyone in the organization contributes to delivering the products to customers. It’s a goal is not and should not be department specific.

The departments that often see considerable delays due to red tape are:
- Human Resources
- Budgeting
- Procurement
- Marketing
Any one of these departments can result in projects getting stuck in the mud, usually due to large amounts of lead time required to fulfill a request. Delivering quality products and services frequently obviously doesn’t mean having no process, but when things need to be done quickly, procedures should be flexible enough to accommodate such requirements. The need for agility applies to all parts of an organization. It’s not just a technology problem.

Consider trying to hire someone new to work on a project. Adding an employee could involve HR, Finance and Procurement, each with their own processes and requests for lead time. The multitude of steps in their respective processes frequently take weeks, if not months. This applies to almost any department of the organization and it’s highly likely that there are processes that add significant delays to reacting to business needs.

Setting common goals and measurements is a sure way to cut red tape. Employees don’t want to be the reason something could not be done. Processes will become streamlined and departments will collaborate out of necessity, if not for the good of the organization, then at least as an act of self-preservation.

While this high-level overview makes it sound simple, most organizations outside of Silicon Valley don’t set common goals for employees. It’s up to the Unicorns to spread the word.

Using Technology to Facilitate Collaboration

The DevOps movement resulted in advances in tools and technologies that aided in Dev and Ops collaboration, and took advantage of the opportunities created by this frictionless new environment.

In order for all departments in an organization to be able to take advantage of similar opportunities, other teams will need to look at ways to optimize their workflows and tool chains.

One area where tools have reduced friction and increased collaboration is code review. Code review has a lot in common with collaborating on any other kind of content. Code review used to be done with people sitting in a room with printouts of source code receiving feedback that often resulted in holy wars about coding styles. It wasn’t effective and that was even within the same department.

Tools now allow documents to be reviewed asynchronously with in-line comments, no matter where in the world the reviewers are located. It’s one of the few examples where communicating via text rather than voice lowers tensions, which is surprising given how easy it is to accidentally upset someone via email!

A great example of this is how my working style changed once I started working with Gene Kim, the award-winning author of The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win and The Visible Ops Handbook. Rather than email documents back and forth, we’d collaborate via Google Docs. No matter what our schedules were or where we were in the world, we worked together effectively. The first time I worked like this with Gene changed the way I think about collaborating on non-technical content.

A key element of collaboration on a single set of artifacts is not having multiple copies in different systems in an organization. In fact, the 2014 DevOps Survey by PuppetLabs found that versioning production artifacts in one place was a key indicator of high-performing organizations. Imagine being able to store all artifacts involved in a project–Including requirements docs, marketing campaigns, and spreadsheets–In a single location accessible via tools that fit your needs. Collaboration, passing audits, applying for tax credits, and disaster recovery would be so much easier.

We’re not there yet. As mentioned, tools such as code review, Google Docs, wikis, version control systems, and email all help with collaboration but they are often still silos of information. Just as breaking down silos between teams has been revolutionary, breaking down the walls of data will one day have an equally profound impact.

About the Author

Before joining Perforce as a Technical Marketing Manager focused on Continuous Delivery and DevOps, Jonathan Thorpe held technical marketing roles at CFEngine, Serena, and Electric Cloud, where he specialized in DevOps culture and automation technologies. In his spare time you can find him reading, playing with all of the current video game consoles, dabbling with mobile software development, and playing with the Raspberry Pi.
November 13, 2014
I Like Your Backend

Back in the early 80’s, dumb terminal networks ruled enterprise IT. Mammoth mainframe IT infrastructures worked as centralized corporate computing platforms feeding a vast nexus of thin clients performing mundane IT operations as intended. Perhaps that was all what end-users longed for.

(more…)

November 5, 2014
The Cloud IDE and it’s impact on DevOps

Recently I did an interview with IBMs Ken Walker. Co-lead on the Eclipse Orion Project, a cloud-based integrated development environment (IDE). My interest in Cloud IDEs peaked at the beginning of the year and has been steadily growing since the interview and being an active user of Cloud9 ( My current favorite Cloud IDE). During this time I realized the Cloud IDE has the ability to totally transform teams. We all know where the IDE belongs in the developers life, but what, if any, is the Cloud IDEs impact on the DevOps movement?

(more…)

May 8, 2014
DevOps killed the developer star

The DevOps blogosphere is aflame over Jeff Knupp’s post on “How DevOps is killing the Developer”. While I agree with Knupp’s premise that DevOps, Agile and other technology trends are allowing (is forcing more accurate?) developers to do more outside of pure development, even turning into “full stack” developers, I disagree that this is a bad thing.

I disagree with Knupp’s assertion that developers are the Orcas of the IT world sitting atop the food chain as the top dogs. I think Knupp consciously or not, degraded the very real, important and hard job that ops people perform, DBAs do every day, as well as everyone else at an organization. Reading this I was reminded on one time I was sitting in an executive meeting and the VP of engineering actually had the stones to say that his developers were smarter than the rest of the people in the organization and worked harder to boot. Now that is a real team builder right there.

One thing I have learned in 30 years of business is that when an organization is achieving its goals it is because just about everyone is working as hard as they can. It is usually not one group, whether it be developers or marketing or operations that is pulling the sled alone. It takes everyone to really succeed. There is no top of the totem pole or rather who is on top is a constantly changing position as bottlenecks come and go, priorities shift and the market changes.

In any organization whether it be startup or enterprise, we all have to do what we need to do to accomplish the mission. There you have it. Working in the Federal market all of those years taught me something. In the DoD and armed forces it is all about the mission. We have missions as individuals. Those missions can be writing code, deploying or monitoring (heck even support and sales), but ultimately those missions all fold into the greater mission of the organization. If developers doing full stack work helps the organization better accomplish its mission and goals, than it is good. No man is too big for the army.

At its core that is what DevOps is too don’t forget. Empowering IT to better serve and meet the organizations goals. Not about letting developers do what they love to do or what floats their boat or what they consider the “top of the totem pole”. Enterprises are here for one thing, the bottom line. It is something we should all never lose sight of.

DevOps is changing the mission of developers. It is a change in the environment if you will. Species are faced with changing environments all of the time. Those that adapt to change succeed and even flourish. Those that fail to adapt fall into the extinction bin. Generally species that are generalists adapt and thrive much better than species that are too specialized. This is basic evolution theory.

I think to Mr. Knupp’s point we are going to see a form of DevOps Darwinism play out. Developers who insist they only develop and don’t do full stack or other DevOps type of tasks will find shrinking habitats and less opportunity. They will exist on the margins. Developers who embrace this new environment will find abundant “target rich” environments with rewarding (both monetarily and stimulating) career opportunities.

I am all for doing something you love for your work. Otherwise I would be practicing law somewhere. I stared working in the technology world because playing with computers was the most fun I was having. I am sure many of you had the same experience. Messing around with code or scripting moved from fun and something you did because you liked it to something you also do to earn a living.

Earning your livelihood doing something you love is a great blessing by the way. There are untold numbers of people who trudge off to work every day and have to do something they dislike, are not passionate about and do just for a paycheck. I can’t imagine having to live life like that.

I think that is what Knupp is talking about too. That developers code because they love to, they need to, and it is what they do. Forcing them to do other tasks takes away from their focus and stunts their growth, which in turns comes back to hurt the organization as a whole.

I don’t think developers will ever stop coding. Mr. Knupp is right, developers code because they want to. That is what they love to do and it is why they became developers. I think however they will have even more opportunity to code, not less as software continues to gobble up the world. It may be a different kind of development, with coding as infrastructure as software dominates, but developers are still going to code no matter what.

We may be seeing the rise of the coding generalist versus the pure application developer as the lines between developer, ops, QA and yes even security continue to blur and overlap. But I don’t think it is a bad thing for developers, enterprises or anyone else. It is just part of the constant change in our environment.

I don’t think this change is bad at all. In fact to paraphrase the great Gordon Gekko, “Change is good, change is what made America great.”

April 16, 2014
An outage war room primer

One aspect of the DevOps movement I’ve seen adopted at numerous companies is the idea that everyone supports their products by being on-call for any incidents that occur in the production environment. This responsibility often leads to participation in the outage war room. For those of you who may be new to this experience, I’m offering this primer from a grey-hair ops guy who’s served countless hours in every manner of war room you could imagine, some that stretch the bounds of that imagination, and a few that I just still can’t talk about. Been there. Done that. Still doing it.

In all that time, however, I’ve observed four distinct war room dynamics that can cause conflict. Understanding how that conflict arises is key to mitigating it.

The war room is a stressful place

This statement sounds obvious, but it’s fundamental to understanding the tensions that can arise in a war room. The clock is ticking. In fact, the clock has already been ticking for some time when you join the room. These things don’t happen instantaneously. When you get the call to join a war room, the problem has already manifested and is causing an impact. Even with automation, it takes time to notify people and have them connect to the war room, in whatever form that takes. In environments with a very high SLA target, the set up time for the war room alone can exceed the monthly allowance for downtime. So once you get into the war room, it’s serious business. The clock keeps ticking.

The war room is often a stressful place as a result. Tensions, and even tempers, can run high. But each and every person in the war room has an obligation to keep themselves under control. Tension and stress feed on each other, and if you succumb, it makes it worse for everyone. Even more important, if you get angry, emotional, or display some other decidedly not cool way of coping with the situation, you are likely to lose the confidence of your peers. That really sucks, so it’s important to force yourself to project calm confidence and be deliberate and measured in your actions. If you have the opportunity (and the ability) and can help to disperse some of that tension, do it. Those are the kinds of moments that build credibility.

And there is another very personal connection to the outage that can be stressful. A professional, worst-case scenario. In the worst case, you are actually responsible for the whole mess. When that happens, you are going to be in your own special hell in the war room. But you should know exactly what is expected of you – the one thing you can do that will really make a difference. You need to tell everyone exactly what you did, without skipping anything, and without regard to how stupid it might sound in hindsight. As in, “I continued to run through all the commands in the procedure even though they were failing with errors and even though I had guidance to stop and seek help if they failed.” To help prop you up in this most difficult of tasks, consider this: everyone else in the room has done the same thing. We’ve all been there. If we haven’t, it’s probably just a matter of time. So you have to swallow your pride, tell the whole story and then get actively involved in fixing things, because that’s what the war room is all about.

The goal of the war room is to restore service as quickly as possible

A war room is bad. If you are in one, it probably means that your customers are impacted. You have an obligation, working with the rest of the team, to recover the service in the most effective and least risky way. In a perfect world, that method is also fast, but life doesn’t always work out that way. If there is no obvious and simple solution, though, things can slow down a bit and we see different approaches forming across groups of people in the war room. Some, traditionally the operations staff, want a way to restore service quickly by a reboot, reset or other “big red button” kind of operation. Others feel it’s important to fully understand the nature of the problem before taking action. These are clearly often opposing goals.

One way this manifests in war rooms is when, in order to further analyze the problem, someone asks to run some experiments, or capture data, or otherwise use the production environment to debug the problem. If there is no method available to recover service with a reasonable level of risk, such experiments may be justified. Otherwise, the team has an obligation to execute whatever procedure they have to bring the system back up, even if that destroys information that might be helpful to the root cause investigation.

Another thing I see happen frequently is when the someone jumps the gun in trying to fix things. It plays out like this:

Jane: Do you think we should reboot the server?

Dick: I’ll get set up for it.

Jane: I think we should do it.

Dick: Done!

Jane: Wait…

A good war room manager will easily spot and work through these kinds of conflicts. If there is such a person on your war room, listen to them and obey them. The war rooms I’ve participated in are not democracies. They are run by a strong leader who is empowered to make extremely critical decisions. It’s crucial to respect that authority in the heat of the moment rather than try to be the hero. Not so coincidentally, this is my next point.

We need solutions, not heroes

In a war room, you need to separate your ego from your idea. The war room does not need heroes. The war room needs solutions. Be advised that no matter what part of the organization you come from, your idea is probably going to go up against some alternative ideas. They will not be judged, generally speaking, on how clever or elegant they are. They will be judged on how effectively they can recover the service and how much risk there is. And if yours is not the chosen idea, it does not mean that you suck. The only people who really suck in a war room are those who interfere with the process of service recovery. You know, the people who interrupt the flow of a critical conversation to ask for a status update, or who won’t stop debating a relatively minor point when the team really needs to focus on a larger issue. That kind of thing. But I digress.

Sometimes, the right path forward does not actually address the root cause of the problem.

Sometimes, the right path forward precludes ever figuring out what the root cause is.

Heroes have a hard time coping with that.

You must accept that you may never know why

The incident and problem management processes in most reasonably mature organizations are designed to recover the service, find the root cause and then ensure that it never happens again. But reality is often different than these goals. Incidents sometimes last a lot longer than you like, as when you discover that something has been broken for a long time and you did not know about it. And we should all know by this point that sometimes you never really know why it happened. You just can’t figure it out. It sucks, but again, it doesn’t mean that you suck. It means that the cost of determining the root cause has crossed the point at which it is feasible to continue, given the impact of the problem when it manifests. That’s just common business acumen, but it’s worth stating. Because when you realize that this is a possible outcome, it can give you a better perspective on evaluating your options. Suddenly, restarting the system and walking away from any run-time state information that might be helpful in root cause determination seems more viable. And those are the kinds of trade-offs that everyone in the war room has to come to terms with.

Conclusion

It sucks being called into a war room due to an outage. It is the very definition of failure in the SaaS world. But you can make a bad situation into a great and positive experience if you handle it properly. You can take an event that should mean that you suck and turn it into something that demonstrates your excellence. Being an effective member of the war room is a way for you, as an individual contributor, to demonstrate that awesomeness. You do it through calm, reasoned action and sound leadership. That builds confidence and trust. I guarantee you that your peers in the war room will notice and respect you for it and your customers will benefit from it. Let’s not let any of them down.

April 15, 2014
DevOps & UX: A Star-Crossed Love Story
“Two households, both alike in dignity
In fair Verona, where we lay our scene
From ancient grudge break to new mutiny
Where civil blood makes civil hands unclean.
From forth the fatal loins of these two foes
A pair of star-cross’d lovers take their life
Whose misadventured piteous overthrows
Do with their death bury their parents’ strife.” ~ Romeo and Juliet: Prologue, William Shakespeare

For more than 15 years I have watched as they battle. The chasm between the two houses was so wide that I often wondered if it would ever be bridged. I never wavered from my view that we were all one house striving to find common ground for making working solutions for the world around us. Sometimes there were allies beside me. Sometimes I stood alone, For the first time in my professional history, it seems like the blood-feud could be put aside. Is it time for peace? Even more ambitious, is the day of an alliance at hand?

The Feud

For many years, corporate IT and finance were the only voices in the room. Ruled by an unquestioned and unmeasured desire for quantification and linear solutions, design considerations were usually an afterthought. All parties were subservient to the almighty ROI which ruled over all with an iron fist. Claims of possible productivity gains through better tooling were always met with doubt and even when taken seriously, you could not claim productivity gains unless you were willing to commit to lowering headcount. Tooling for technical employees to manage the environment and its operating data was so scarce, that there were more applications with no maintenance tooling than applications with operational plans. Operational changes were made by going into databases by hand and making changes in a “hit or miss until you hit” fashion.

In the late 90’s UX showed up to the party and with a basis in humanism, the equation changed. This was the time that the “great misunderstanding” happened and 15 years were wasted building up a corporate landscape strewn with enmity and false narratives. The “great misunderstanding” goes something like this:
- “Let’s make something that people can use” – UX practitioner espouses user-centric design and introduces new methods and artifacts into the SDLC processes which take interaction design specifications to a new level in corporate America
- “That’s going to take more time and I don’t agree that it’s any better than what I could do myself” – IT staff view this incursion as a mechanism to take what little time they are given to complete projects while also limiting their creative outlet
- “I don’t care what you think and neither does anyone else” – UX practitioners view the response as a denial of their value proposition and a fundamental lack of care about the people who interact with technology
Several of the basic assumptions in these archetypal interactions between disciplines were short-sighted. For example:
- The “more time” fallacy – adding a design process may take more time if you only think about a single iteration in development and that’s only if you are lucky enough to have a complete interaction model worked out in your head that doesn’t have any design holes in it. Designing interactions on paper saves time in the same way that technical design saves time over “jumping straight to coding”
- The “fighting on the wrong front” mistake – It was never the goal of UX to remove IT budget. The goal was to make things better for the humans that interacted with technology while creating business value. The two different disciplines never saw their goals as aligned. We will never know what the alternate history would have been like had UX and IT teamed up to get larger budgets in an effort to create better and more sustainable experiences for users, developers AND operators.
- The “reductive labeling” error – Both IT and UX created a narrative where the interaction and visual design tasks were the “creative tasks” for “creative people”, while the development was the “technical tasks” for “technical people”. These labels have stuck around for the last 15 years and we are still paying the price for removing the idea that IT staff is fundamentally non-creative and devoid of craft.
- The “self-reinforcing paradox” tragedy – It was never the goal of IT to demean the UX discipline. When one neglected child is joined by another neglected child, the two will almost unavoidably fight over the little attention afforded to either of them. The irony here is that UX’s basis in cognitive science could have possibly seen the hostile response from IT for what it was; a cry for help. Instead, UX took the bait offered by IT and thus we are here 15 years later with a wide chasm between two disciplines that should be united by common causes:
  - a desire to make awesome stuff
  - a desire to create sustainable solutions
  - a desire to express one’s self at work
  - a desire to improve the lives of their customers and employees at the same time
Oh The Humanities

When IT resources and UX practitioners first came together the familiar plot “boy meets girl, boy loses girl” emerged and we may now, thanks to DevOps finally have arrived at the time when “boy gets girl again”. Gene Kim and the folks at IT Revolution Press have hit the target dead center in their manifesto:

“Working in most IT organizations is often thankless and frustrating. People feel as if they’re trapped in an ever-repeating horror movie, helpless to change the outcome The organization abdicates their responsibility to ensure that IT is managed well, plunging the department into relentless intertribal warfare between development, IT operations and information security. And of course, the auditors.

What inevitably results is chronic under-achievement. The life of an IT professional is often demoralizing and frustrating, typically leading to feelings of powerlessness and rife with stress which seeps into every aspect of life. From stress-related health problems, to social issues, to tension at home, it has become clear that the current state of the IT professional is not only unhealthy, but unsustainable.

As people, we’re wired to contribute and to feel like we’re actively making a difference. Yet, all too often when IT professionals ask their organization for support, they’re met with “you don’t understand,” or worse, a barely masked, “you don’t matter.”

As a human being, this is the worst response we could receive. Because the opposite of love isn’t hate — it’s apathy.”

Despite what some IT step-children will tell you, UX is not about making pretty pictures. UX as a discipline was formed to combat apathy and contempt by corporations through a mechanism of introducing empathy. Similarly, DevOps wasn’t formed to “make web pages faster”. DevOps clearly has a tactic of emphasizing performance (as well as other technical tactics), and performance improvements are clearly one of the benefits that come out of a DevOps organization. DevOps was formed to make IT a more sane and human space inside companies for IT employees, it just so happens that DevOps produces other impacts that are financially beneficial for enterprises.

Avoiding The Fates of Romeo And Juliet

Knowing that waters of DevOps and UX flow from the same wellspring allows the practitioners of DevOps several opportunities to repair the tragedies that have already befallen our enterprises before they escalate to the Shakespearian proportions. Three broad categories of opportunities would be:
1. Enroll UX into the DevOps movement – Knowing that we are aligned creates multiple possibilities to bring many industry and enterprise change agents into our cause. How much would the DevOps adoption curve accelerate if people like Don Norman, Jesse James Garret, Alan Cooper or Steve Krug were visible advocates for DevOps? Support from UX at the leadership level in the industry and within your enterprise is an advantage you cannot afford to pass up. The number of consensus driven organizations are just too many to count and the more groups and individuals that believe in advancing our cause, the better.
2. Learn from UX’s travails – Creating artificial barriers to discipline inclusion is ultimately an adoption limiting factor. When we make things pass-fail or turn collaborations into confrontations or competitions for who is DevOps and who is not, we all lose out on the what a wider community could offer. Hold yourself accountable to pivot confrontations into constructive opportunities where learning can happen for people of different disciplines.
3. Take some pages from UX’s playbook – UX’s biggest lever for creating believers is by involving non-UXers in the research methods and activities (think about the last time you went to a usability test behind the 2 way mirror and saw a person struggle to use a system you were part of implementing). One of the best things we the practitioner community can do is to demonstrate what life is like without DevOps in real-life scenarios. When someone really attaches to how ugly a manual process is, or how soul-crushing it is to not be able to self-actualize in their job, they can’t help but want to be a part of making it better. Problem solving is hard-wired into the human condition, let’s leverage it.
April 1, 2014