devopsy.kinsta.cloud

Author: Todd Vernon

State of On-call Survey 2015 – Help Needed!
Last year, we launched the first ever State of On-call Report. We asked about tools, setting up on-call rotations, what team members are involved and what happens after the incident gets resolved…among other things. We got 500 responses and in the process, learned a lot about what people are doing to handle the challenge of on-call management.
A few key takeaways from 2014’s results…

In an effort to see if on-call is getting any better for anyone, we’re conducting the survey again this year. Because what fun is data if you can’t start picking out patterns, checking out year-over-year changes and looking for trends?

Our predictions for this year’s results…
- less folks are using homegrown monitoring and on-call systems because more options are now available off the shelf
- more organizations will report as having moved their infrastructure (or large parts of it) to the cloud
- an increase in the adoption of DevOps practices, especially when it comes to sharing the load of being on-call
- ChatOps will play a bigger role in how organizations do their incident management
- runbooks and collaboration will continue to be the two biggest factors in solving problems faster
- a decrease in the number of people saying on-call is getting worse (hopefully!)
If you’d like to help the cause, we’d love for you to spend a few minutes answering questions about being on-call. Click on the image above or here to begin the survey. We’ll be choosing one lucky winner from all the respondents to win a BB-8 from Sphero.

Check back in December when we release the 2015 State of On-call Report. We’ll be hosting a webinar with devopsy.kinsta.cloud, presenting the results. Stay tuned and thanks in advance for your help!
October 9, 2015
The Sea Change that is DevOps – DevOps is changing tools and infrastructure as we know it: Part 2

In the first part of this series, I explored the factors that have lead DevOps to where it is today. From a three to six month deployment period to regular weekly (or even daily) deployments, DevOps has certainly changed the IT landscape as we know it. Now, with the growing adoption of a DevOps mindset, the IT world is seeing a huge growth in software companies created specifically for DevOps professionals.

The explosion of monitoring tools and evolution of the software engineer role

With the added complexity of SaaS software and the rapid innovation of the platform, issues occur more frequently. With the adoption of unified cloud infrastructures more and more of these issues are application layer in nature. Service providers like Amazon are now consuming the need for foundational monitoring from tools such as Nagios at the IaaS (Infrastructure-as-a-Service) layer. Today, problems are popping up at a higher level in the stack and, monitored by tools such as NewRelic, that output is directly destined for software engineers. The ability to know if something is working correctly is now a comparative operation performed by the engineers who wrote the software.

The changing landscape of the NOC

If getting software built and into production was the nirvana that everyone had been marching toward, the NOC (Network Operations Center) was the manual switchboard of the past standing right in the way.

Traditionally, the NOC has been the catchall command center for people who monitor the systems that monitor the software. NOCs were designed to be the first line of defense against technical problems, customer problems, and general communication of issues to the wider company. Historically, NOCs have had an infrastructure bias as the software higher in the stack changed less often. But as the true infrastructure layer of SaaS companies began to move away from the company and into IaaS platforms, that need was receding. At the same time, as application deployments were increasing in frequency, often first responders that were not highly technical were adding drag in system resolution times.

Today, companies are looking for ways to direct problems straight to the “makers” that created that component of software in the first place. With the speed of SaaS businesses, the company can no longer afford for the first responder to be anyone other than the person or group that created the logic in the first place.

If the NOC of the past had two critical functions, solving technical problems fast and communicating with internal and external stakeholders, suddenly half of their charter was starting to fail dramatically. This led to the explosion of software like VictorOps and monitoring tools like New Relic and AppDynamics to help diagnose these problems quickly.

Operations is dead, long live DevOps

In all this time a lot of things have changed for operations people. While a technical operations manager in the past might hire system admins to do all the manual configurations, now they are looking for team members with coding skills. While understanding hardware was a requirement in the past, now suddenly virtual servers that you can rent from Amazon, Rackspace or any of 30 other IaaS providers can be created (spun up) in an instant with no knowledge of hardware at all. In the past a system admin had little to no knowledge of how the application software worked, but now that is their primary job description. In short, today you want to hire a software developer of a different skill set – one that codes the infrastructure and understands the application. While the NOC was once your first line of defense against system problems, now the team that wrote the software is the only defense necessary for technical matters.

For many, the end game has been realized

Now, for more progressive companies, the big gain has been realized. After literally a decade of innovation, for the first time it is easier to fix a production problem in-stage, rebuild and redeploy (all automatically) than it would be to fix the problem in production. Today the reality of being able to execute an Agile development sprint front to back is possible.

September 8, 2015
The Sea Change that is DevOps – How it all started: Part 1
Over the last 20 years, software has slowly taken over the world. Businesses that historically operated on 8-hour business days are now 24×7, navigating a global economy that never sleeps. The Internet enablement of everything has thrust what was originally an important support role into the lifeline of a business.

A decade ago, Agile software development methodology forced more pressure into the IT system by deploying software at an ever-increasing rate to the point it’s at today – where innovative companies can deploy new software multiple times a day. DevOps is at the forefront of addressing the question of how IT deals with these problems, but like Agile before it, there’s a learning curve.

In this two-part series, I will discuss how the IT world worked before DevOps and how development and operations teams united to adopt more Agile practices.

How things were

Before this sea change, a company would hire IT professionals who would purchase and configure servers, place them in a data center, copy mission critical software onto them and with some help from software engineering, get things running. After some quality checks, the IT team would lock down the datacenter controlling both physical and virtual access and “let things run.” The common way to provide stability was simply to not let anyone touch anything, as most problems have been (and still are) self-inflicted wounds. Keeping the system “static” kept a lot of problems from surfacing. The software was probably not more stable; you just didn’t find the problems as fast.

In this world, IT professionals had a feast and famine existence. If a new service deployment was happening (usually every 3 to 6 months) it was an all hands on deck activity reminiscent of a rocket launch. The same manual deployment would happen, along with software engineering, database engineering, etc. Data migration would occur, new software would be installed, and the system would manually be brought back up. In the event of a software problem, or data migration problem, the team could elect to rollback and bring the old software back up. If things went well, then again some basic end-to-end quality assurance would take place and the new software release would be unleashed on the world. More progressive companies would do this for part of the system, diverting traffic to avoid perceived downtime. It would often take an entire night to do a new service release, and companies generally did them at night because downtime was not unusual.

Then software development changed

The competitive landscape that the Internet created along with the now ubiquitous SaaS business model made companies recognize that their ability to innovate the product was a competitive advantage. A lot of companies had adopted Agile software development practices in response to the increasingly high failure rate of larger efforts. Part of Agile is creating smaller development efforts that get code functional faster, reducing the size of the endeavor. This is commonly referred to as a “software sprint.” This promise of innovating the product faster was very alluring, especially to Web 1.0 companies that were now in feature battles with more nimble Web 2.0 companies who had more quickly adopted Agile. It was becoming more and more evident that the problem was shifting from an issue of creating software fast enough to deploying the necessary code in an expedient manner.
The big problem was the deployment stage of the process. IT teams were accustomed to changing production systems every three to six months. This combined with a trusted model that largely kept software developers out of production systems, effectively nullified any Agile gains in SaaS businesses. In other words, half the product development cycle had been fixed with Agile, but the other half deploying the software had not yet caught up.

Operations as a changing landscape

Major changes were also occurring for operations inside companies, many of which were driven by the development side of the house.
- Virtual server technology started to get major traction within production environments. This allowed teams to treat servers as software. You no longer had to run the operating system and application on a physical server, you could run them on a virtual server (that ran on physical servers). The configuration of servers that, by hand, took an incredible amount of time, could essentially be defined once and then cloned, the same way you would clone a file. Deploying a new server was much easier, and a corrupt server could simply be deleted and reborn in minutes. All these virtual servers could be managed by a single piece of software called a hypervisor, with which creating a new “server” would take a minute, when it used to take weeks.
- Puppet and Chef were born. At the same time development organizations, recognizing this major problem with deployment, started addressing the production configuration problem with tools such as Puppet and Chef that essentially allow software and IT teams to codify the configurations of virtual and physical servers and deployments of new code. This allowed for repeatable processes that could be rapidly done and redone. No longer did your IT team have to install software by hand, instead they could code it.
- Continuous integration and delivery enabled software teams to use newly developed tools like Jenkins to do nightly software builds, perform unit tests of parts of the software and automatically deploy software into production. All of these tasks had been highly manual operations historically and fraught with mistakes and inconsistencies. Codifying these steps would give way to deploying software within minutes or seconds rather than days or hours.
Most importantly, all of these efforts were largely driven by the development of more progressive operations in the company. The operations side of most companies was never conceived as a “maker” culture; the handwriting was on the wall – operations had to change or it will would have become a victim of this revolution. I’ll take a look at some of the resulting changes in the second part of this series.
September 3, 2015
Share Your Thoughts about On-Call

We all know that being on-call sucks. What we don’t know is how other people are dealing with the problem.

It’s time to dig deep, ask the hard questions and collect data around what it’s really like to be doing the hard job of keeping the machines up & running. That’s why VictorOps needs you!

We’re conducting the first State of On-Call survey and would love to hear what you have to say about the topic. Whether you’re currently on-call or manage a team of on-call folks, we’d like to know what tools you’re using to make the process easier and what sort of pain points you’re still experiencing.

We’re thanking you in advance for your time & feedback. Additionally, your survey response gets you automatically entered into a drawing for a Nexus 7 tablet. Help us to better understand the problem so that we can help you make on-call suck less.

October 8, 2014
The Push to Make Ops More Social

Given that social coding is a thing, what about the other side? Operations has the same image of lone wolves solving problems in the dark. The culture of on-call is built around the concept that when you’re holding the pager, you’re the one on the front lines, so that everyone else can sleep.

Social operations, the ops in DevOps, is going to be about the same things that made social coding important: knocking down walls between individuals; making the knowledge of a system or infrastructure explicit and available, in context. But social operations is more difficult in two major ways: operations problems are often multidisciplinary, and often temporal – meaning that you have to be social when the situation arises,not on your time!

When I say that operations problems are multidisciplinary, I mean that solving a problem in an operational context might require a number of people with different domain expertise. The solution is going to be much more complex and may not be portable to other people. Fixing something that’s wrong with a puppet script inside of our own infrastructure is fundamentally different than contributing a bug fix back to an open source project, or otherwise contributing to an open development environment.

And when I say that operations problems are temporal, I mean that many of the forms of social engagement we’re getting used to in a social coding environment – forums, pull requests, etc. – are eminently useless when you’re in a firefight. When something’s wrong enough that you’re being paged at 2am, waiting for someone to respond to a comment thread on a pull request isn’t an option. You have to be able to get the right eyes on the problem, in a timely manner, if you want to solve infrastructure problems faster.

So, how can we make the operations side of DevOps social? What parts of “social” are useful in the context of an operations issue?

Opening up the knowledge of your team is a good place to start. Having the ability to get people involved is another good thing. Basically, what can we do to make it so that you know what you need to know, whether that’s how a problem was solved in the past, who’s available to get help, or what else is going on. That’s what we’re trying to figure out at VictorOps.

May 1, 2014
DevOps makes coding more social, social coding makes better code

One of the main principles of DevOps is to break down silos. What’s more like a silo than the image of a lone developer, shutting himself off from the world in order to code more and interact with people less? “I’m just here to write code and that’s all I do.” There’s nothing collaborative or DevOps about that scenario and unfortunately, it’s one that is all too frequent within organizations.

Enter social coding. This is a concept that didn’t exist twenty, or even ten, years ago and one that continues to grow in popularity. At its essence, social coding is an approach to software development that puts the focus on formal and informal collaboration. There are many benefits to social coding but the most important is probably the opportunity it presents for mutual growth and learning. Developers are allowed to collaborate in an environment that fosters discussion, creation and sharing.

When talking social coding, you can’t avoid a mention of GitHub. Take everything you love about your favorite social network and then apply those same ideas to a code repository. That’s basically what GitHub did, and continues to do, with much success. You can create a profile, follow or message someone, see what others are doing with their code (or yours) and even go so far as to subscribe to RSS feeds of activity streams. GitHub started to take off a mere five years ago and is now held up as one of the more successful examples of how social coding can be commercialized.

Companies like GitHub and SourceForge are leading the way in revolutionizing how developers commit and share their code but the foundation of social coding still rests with code review in a public forum. As Mark Chu-Carroll puts it…

“If you’re programming and you know that your coworkers are going to look at your code, you program differently. You’ll write code that’s neater, better documented and better organized — because you’ll know that people whose opinions you care about will be looking at your code. Without review, you know that people will look at code eventually. But because it’s not immediate, it doesn’t have that same sense of urgency, and it doesn’t have the same feeling of personal judgement.”

What’s next for social coding? After it’s start in the open source community, this is one idea (like many others) that is heading towards the enterprise market. Companies are beginning to recognize that even internal projects can be managed and treated as open source efforts. From wikis to crowdsourcing, there are many ways that the enterprise market can take advantage of social coding. The end user is the one who stands to benefit the most from such a change.

Dave West, a Principal Analyst at Forrester Research, has this to say about social coding:

“We’re social animals, and the development of stuff tends to be social in nature. We think a better way of thinking about it is as social development. It’s not just about coding; it’s about enabling business change and delivering business value in a social way. It will be interesting to see how companies broaden the idea from social coding to social development. Social coding is a fundamental change.”

April 25, 2014
Why Vertical Collaboration Matters for DevOps

Vertical collaboration is defined by Rick Defour as a team of people working interdependently toward a common goal for which they are all mutually accountable.

Now that everyone is clear on the importance of collaboration and the different types, it’s time to delve into the importance of one type of collaboration in particular: vertical. Vertical collaboration is defined by Rick Defour as a team of people working interdependently toward a common goal for which they are all mutually accountable. You can find examples of vertical collaboration in educational settings where teachers of different levels work together to insure student success and in manufacturing settings where the supplier & retailer can better leverage themselves to enhance the bottom line for both.

When you think of vertical collaboration in the tech space, the example of a dev team pushing out code is a good one. Each dev has a different area of specialty but they must work together in order to implement the new feature. Taking this into consideration then, it’s easy to see why a vertical collaboration tool has to be easy to use and not create any new boundaries. The more vertical a collaborative tool is built, the more useful it is to real people.

The problem, of course, is that many companies view all collaboration tools as vertical. They only want to spend money on one tool so they pick the horizontal one (thinking that it’s the one more people will use) and it turns out to not actually be helpful to anyone. So that’s when companies start adding to their toolbox and then end up with a hodgepodge of tools. Instead of making their collaboration process more effective, they have made it more convoluted.

I created VictorOps because there was no existing vertical collaboration platform for DevOps. Going into it, I knew I was solving a problem I’d had at a number of previous companies and therefore already knew the exact requirements needed. The collaborative platform would have to be mobile first, would have to allow for synchronous engagement while enabling asynchronous problem solving and would have to give all participants symmetric access to information from their enterprise monitoring platform.

What I realized after creating VictorOps and using it to solve our own internal issues is that vertical collaboration gives team members the opportunity for scalable “leaning in” while also providing the freedom to lean out to reflect. What’s not to love about that?

April 17, 2014
Not all collaboration types are equal

Talking about collaboration in general terms is easy. I’ve done it and so has everyone else. It’s time now to dig deeper into the specifics of collaboration, meaning the different types – asynchronous vs. synchronous, vertical vs. horizontal – and how different companies fit into the magic collaboration quadrant. And yes, the quadrant is indeed magical.

Let’s start with the two types of collaboration that are time-relevant, asynchronous and synchronous. Asynchronous collaboration takes place outside of real-time. The best example of a ubiquitous asynchronous collaboration tool is email. You send an email and the receiver has time to digest the information. The drawback to asynchronous communication is that it’s inadequate as a means of communicating a sense of urgency. You have to wait for someone to respond to an email and that lack of immediacy can cause damage in troubleshooting or problem solving situations. Email replies tend to include the original message for context because you can’t rely on the timing of the message to keep the context.

On the other side of that collaboration coin, you have synchronous collaboration. Synchronous communication takes place like a conversation and more closely resembles a face-to-face interaction. A chat session is the ideal illustration of this concept. The obvious benefit to synchronous collaboration is the ability to send and receive information right away. This adds to the feeling of immediacy and tends to provide a more effective means of working together to solve issues in real-time.

Now that we have that covered, it’s time to move to the other defining aspect of collaboration: whether it’s vertical or horizontal. Vertical collaboration consists of problem solving tools that are domain-specific. An example of a vertical collaboration tool is one built for certain parts of the business like Salesforce for the Sales & Marketing teams or Rally for the development team. Horizontal collaboration consists of non-domain specific tools, meaning that they have potential use across all teams of an organization. Project management tools like Campfire or remote meeting solutions like Join.me or WebEx are examples of horizontal collaboration tools which tend to be widely adopted but lack the ability to really go very deep in helping in a domain-specific way.

Check out the quadrant I created to better show where different companies currently fit into the ever-growing collaboration landscape, based on the definitions provided above.

When looking at the state of collaboration tools, it’s interesting to note where most companies land within the quadrant. Why so few vertical synchronous ones? I’ll answer that next time.

April 11, 2014
Collaboration – A DevOps Tenet
You can’t turn around in the tech space without hearing about collaboration. Whether it has to do with the new economy being built around collaboration or the platforms designed to improve it or the various strategies one might implement to spur more of it on your team, collaboration is the hot new term, especially when it comes to DevOps.

All that being said, what is collaboration exactly? And the deeper question…can you get to an answer to a problem faster if more viewpoints are involved? Can you cut through space and time to overcome physical barriers and find the specific knowledge someone on the team has that others do not? Can you make the practices of DevOps, which are already more nimble, even faster by getting more eyes on the situation?

Here’s what we talk about when we talk about collaboration internally at VictorOps:
- remote engagement – In order for collaboration to work, you need to be able to get folks involved no matter their physical location. Collaboration doesn’t happen just between 9-5 and doesn’t take place just in the office setting. Team members want to help out (especially during a firefight) but they may not want to give up everything in their life in order to do it.
- real-time troubleshooting – As anyone who has used Salesforce or JIRA can attest, effective collaboration doesn’t happen in a ticketing system. If you want to work together, you need a communication channel that allows for synchronous discussion that keeps everyone on the same page, at the same time. Immediacy is a crucial factor in figuring out the cause of a problem and synchronous messaging, which closely resembles face-to-face conversation, is one of the best means of expressing that.
- the right tools – These days, if you want your team to adopt a collaboration tool, you have to keep two things in mind: the tool must be mobile and the tool must feel natural to use. Without these two things, you’re going to have a hard time getting everyone on board. If a tool is clunky or requires a steep learning curve, no one will actually use it, making for very ineffective collaboration.
- solving problems faster – The end result of effective collaboration is coming to a solution in a quicker manner. Whether it’s because you’re getting the right eyes on the problem, noticing data patterns or reducing new participant spool-up time, collaboration should help the entire team do their jobs better while achieving company objectives.
Collaboration can become just another marketing term or vague business concept without any concrete examples of why it’s important. In our next post, we’ll talk about different types of collaboration (vertical vs. horizontal, asynchronous vs. synchronous) and why collaboration works best when it’s vertical.
April 4, 2014
How to laugh at DevOps
Talking about DevOps is all the rage these days. But with that conversation, it’s easy to become inundated with the amount of serious discussion around the topic. I do agree that DevOps is not something we should joke around about, but since humor is an inherent part of our culture at VictorOps, it makes sense that we would be drawn to the humor of the situation.

Below are the ten of the funniest Twitter accounts that make light of the DevOps movement. Some are obviously one-joke wonders while others continue to make us laugh. Either way, the list below is proof that you can’t take DevOps too seriously:
- @DevOpsBane – The superhero of faster deploys.
- @DevOpsYakov – In Soviet Russia, apps deploy you.
- @UncleDevOps – Everyone’s favorite continuously-deploying uncle. Also a little crazy.
- @DevOps_Jesus – Continuously delivering from evil since 4 BC.
- @DevOpsGrouch – I complain. I swear. I automate. Deal with it.
- @DevOpsTourettes – Development operaRIAK CHEF JUJU FUCK.
- @DevOpsBorat – Cultural Learnings of DevOps for Make Benefit Glorious Teams of Devs and Ops.
- @DevOpsX – Always fighting the good fight for DevOps equality.
- @DothrakiDevOps – Master of War, Horse and Large Scale Deployments.
- @DevTylerDurden – May the build never be complete.
Feel free to add any that I might have missed to the comments below. And remember to laugh…it’s just DevOps after all.
March 26, 2014