Given that social coding is a thing, what about the other side? Operations has the same image of lone wolves solving problems in the dark. The culture of on-call is built around the concept that when you’re holding the pager, you’re the one on the front lines, so that everyone else can sleep.
Social operations, the ops in DevOps, is going to be about the same things that made social coding important: knocking down walls between individuals; making the knowledge of a system or infrastructure explicit and available, in context. But social operations is more difficult in two major ways: operations problems are often multidisciplinary, and often temporal – meaning that you have to be social when the situation arises,not on your time!
When I say that operations problems are multidisciplinary, I mean that solving a problem in an operational context might require a number of people with different domain expertise. The solution is going to be much more complex and may not be portable to other people. Fixing something that’s wrong with a puppet script inside of our own infrastructure is fundamentally different than contributing a bug fix back to an open source project, or otherwise contributing to an open development environment.
And when I say that operations problems are temporal, I mean that many of the forms of social engagement we’re getting used to in a social coding environment – forums, pull requests, etc. – are eminently useless when you’re in a firefight. When something’s wrong enough that you’re being paged at 2am, waiting for someone to respond to a comment thread on a pull request isn’t an option. You have to be able to get the right eyes on the problem, in a timely manner, if you want to solve infrastructure problems faster.
So, how can we make the operations side of DevOps social? What parts of “social” are useful in the context of an operations issue?
Opening up the knowledge of your team is a good place to start. Having the ability to get people involved is another good thing. Basically, what can we do to make it so that you know what you need to know, whether that’s how a problem was solved in the past, who’s available to get help, or what else is going on. That’s what we’re trying to figure out at VictorOps.
There was a lot of brush back last week over Jeff Knupp’s post about how DevOps is killing the developer. Frankly I wasn’t shocked by Knupp’s opinion that developers “are the dentists in the dental office”. In my 30+ years of involvement in the IT industry my experience is that developers often think of themselves as the only smart person in the room. Often times that is the reason they are are the only person in the room, lol. The fact is many developers relate better to code than to people. For me all this talk about DevOps killing the developer is akin to saying power steering killed the driver. Just because you can do more with new technology, does not mean one dies, one just has to pivot and in this case become business social.
Now the sociable part may be the problem. Sometimes I feel that DevOps is trying to create a love fest between Ops, Developers, QA, security and the rest of the IT stack. While they may not have to love each other, I do think they all need to be nice to each other as opposed to ignoring or just going around each other, as was all too often done in the past. A common scenario I have seen for example is for developers to rollover Ops by stating this is mission critical for the business. These tactics will no longer work to let developers have their way.
Developers need to realize they need to become a normal part of the business social structure. They will be required to talk and work with people, not throw tantrums when they don’t get their way. Left unchecked, certain developer personality types can sabotage your project, or worse, make themselves no fun to work with. I am sure we have all met and unfortunately had to work with people like that. In my career, I’ve encountered virtually every personality type. In fact, I’ve also knowingly hired all of them. They know who they are; they are the developer that if challenged or worse accurately taken to task for lack of being part of the team; quit cause they can’t manage the confrontation, thus interpersonal stress. In the old days they would not come out of their house for weeks continually playing Leisure Suit Larry, if your not familiar with it Google it
I remember one of my good friends telling me this story that sums this whole developer persona issue perfectly. This company, we will call DevGoof, was a well funded start up. Every quarter they held a meeting with all the employees to let them know how the company was doing. Most employees had some kind of equity play in the company, so everyone was incented and theoretically interested in company performance. Several quarterly meetings came and went and not one developer showed up. Eventually DevGoof management got concerned and called a mandatory meeting for all developers’.
DevGoof management put out some pastries, juice and coffee for the brain trust. Then proceeded to tell them why these meetings were important to them and their fellow employees. Notably pointing out how it is not just about them but also about the rest of the DevGoof team. This meeting went on for about one hour and not one developer asked a question. So eventually this meeting was over and everyone went back to developing. The next morning, about 50% of the developers quit and some in a not so professional fashion.
The moral of this blog is the DevOps is going to force the development community to become more interpersonal within the business environment. This will lead to personal success within the new world of DevOps. The DevOps Developer/Engineer that can demonstrate leadership and compassion for team will be the one leading the DevOps way within the organization. Don’t be a Devgoof!
If you would like say something; email parker@devops.com and don’t forget to comment here.
The pressure to prove one’s coding prowess can make a developer do some very clever and creative things with their source code. Sometimes it’s just a unique approach, but every once in a while this cleverness may shave a few milliseconds off of some performance metrics as well. In the end it’s not worth it. While this may seem like the type of innovation we want from our peers I’ve come to the conclusion it often does more harm than good. A quote that ran across my twitter feed a few weeks ago sums it up nicely: “So if you’re as clever as you can be when you write it, how will you ever debug it?” This pearl of wisdom comes from a book called “The Elements of Programming Style”. It’s a programming book from 1978 that is still entirely relevant. There’s a Wikipedia article you can check out if you’re not ready to take my word for it.
I know I’ve coded something in a unique fashion, and was quite sure I was being brilliant at the time, only to go back later and have absolutely no clue why I wrote the code the way I did. If I added comments I may be able to find my place again but sometimes it takes a while. What about when another developer runs into one of these bright ideas? Will any amount of meditation or contemplation allow them to make sense of what I was doing when I wrote it? Should a new or junior developer be expected to have some out of body experience before being able to work with me? Of course not. I hate it when I do this to myself, never mind when I run across someone else’s flash of genius. I certainly try to keep other devs from going through this type of mind freak as often as I can.
While this exercise can make for a fun parlor game and help extend our knowledge of the language we’re coding in it causes more problems than it solves in a team environment. A wiser developer sticks with a tried and true approach. They’ve been around long enough to know that the code needs to make sense to any garden variety developer that may be working with it in the future (which may in fact be the developer that wrote it in the first place).
I first learned this lesson years ago while tutoring mathematics. I had a developed an algorithm to generate the unit circle. Computer Science majors I was tutoring in math had an easier time using this to quickly generate the circle on scratch paper than arbitrarily memorizing it or understanding the standard proof that shows a bunch of triangles somehow morphing into a circle.
Radian Values of the Unit Circle
The algorithm requires a certain amount of seed data. One must know that a full circle is 2π radians in circumference. This puts the other quarter circle measures at π/2, π, 3π/4 respectively. Only a simple understanding of fractions is required to see this. Next we need to fill the first quadrant, which is only three more values, π/6, π/4, and π/3. From here the algorithm helps us extrapolate the remaining radian values for the circle.
First Quadrant: π/6, π/4, and π/3 (initial seed values, you have to remember these)
Second Quadrant: π – First Quadrant Corollary Value ( e.g., 3π/3 – π/3 = 2π/3, our 2nd quadrant value)
Third Quadrant: π + First Quadrant Corollary Value ( e.g., 3π/3 + π/3 = 4π/3, our 3rd quadrant value)
Fourth Quadrant: 2π – First Quadrant Corollary Value (e.g., 6π/3 – π/3 = 5π/3, our 4th quadrant value)
So, the formula for quadrants two through four can take the form of (n – s), (n + s), and (2n -s), where n = π and ‘s’ is one of the three seed values from the first quadrant. Three simple equations that any CS major should be entirely comfortable with using. There are a few other notes on this proof to get the coordinates, signs, and degree measurements correct. This is not a post about mathematics however, so I’ll leave that as a exercise for the more mathematically inclined to complete on their own time.
Other mathematicians where not as impressed as the CS majors and the student athletes were with this. Watching students that actively despised math struggle with memorizing the unit circle in an effort to complete their core was part of the fun of being good with numbers. It was suggested (by some troll of a jerk) that I might be a math noob myself, some poser that wasn’t any smarter than the fools he was tutoring. Impostor syndrome set in and I I felt I had no choice but to prove my mathematical prowess.
First I went big and reproved what I was doing with a Klien Four-Group. The trolls were quick to point out that this, like my first model, wasn’t a circle at all but just a bunch of points on a plane. Next I came back with a version using a rotational matrix. This did in fact generate a full circle with any value you like on it. The trolls had no more jabs for me but they also fell short of any compliment or encouragement for the work I had done.
As for the students I was helping, none of them quite understood the issue I had encountered. They viewed those trolls as nothing but trolls. Peers that only laughed at their struggle and refused to even try to help them. I ultimately found out my students were concerned that I had gotten sucked into this quagmire. The research I was doing was beyond their understanding and they were still using the first model I developed. It did exactly what they needed it to do. I was informed I should take the rising grades as evidence that my work was of merit and move on to a more productive use of my time and energy.
It was only then I noticed the trolls did have a point. The Klien Four-Group model and my original algorithm weren’t circles at all, just dots with values attached to them. A mathematician would have issue with this where a developer would not, since an object oriented developer should have a deeply rooted appreciation for abstraction. A class called Tree rarely resembles what we know to be a tree. It’s the use case that ends up being important. To a paper company a tree only needs to be represented by the amount of pulp they can expect it to produce. To an apple farmer it probably boils down to the number of apples the tree puts out and maybe its location in an orchard. What we take to be a tree, its size, the shape of its leaves, how it smells, the sound it makes when the wind catches it, are all details beyond the scope of most software we’ll ever write.
This happens to be a huge blocker for most new OOP developers, especially the ones that are good at math. The object model I was using to teach CS students how to do trigonometry ended up being the perfect tool to teach mathematicians how object oriented programming works. The rotational matrix version is important because sometimes non-standard values are needed. It’s useful to show how a method call may use standard values or extend onto another model, like the rotational matrix, when standard values don’t apply.
Mathematicians appreciate the Klien Four-Group model and I’m happy to show it off when I have the chance. More importantly, it’s a reminder to not get too fancy with math they’re putting into software. The math is beautiful more than it’s clever, and I’ve written code I feel the same way about. Like most works of art it stems from a hobby and should be left out of any applied scenario if it’s not practical. If we can get the same result with something that’s simple, something that’s easy to understand, easy to use, and even common, then that’s what we should use every single time because it will be easy to maintain. Wisdom is nothing more than knowledge applied over time. Time and time again I’ve come to the conclusion that maintainability trumps performance. We should be quick to sacrifice any minimal performance gain that makes our code difficult to understand. If a major performance gain is within reach then we’d be wise to abstract any strange code that was used to obtain it. I’d also advise you to comment the living day lights out of it as well, since you may need to remember how it worked yourself someday.
During your teenage years, you may have figured out a way to get a ‘yes’ from your parents by playing off of the responsibility and knowledge gap between them…walking up to your mom and saying something like, “Hey Mom, Dad said he would drive me, can I go out tonight?” and then proceed to your dad and say something like, “Hey Dad, Mom said I can go out tonight and you need to drive me.” There is nothing cleverer than a determined teenager wanting to get his or her way. Such is the case of targeted attackers and the way they play one IT department’s responsibilities off another. This is the grey area where the organizational chart and separation of duty can introduce weaknesses exploited by the more advanced threats we face today.
You don’t have to look far into your organization to see this at work. The security team does its thing, networking engineers do their thing, and IT operations teams in general all have administrative realms that define functional boundaries as to what they can see and do. To borrow from Colonel John Boyd’s OODA loop, each team has its observation and orientation (OO) process for gaining operational visibility and making decisions, as well as specific decisions and actions (DA) they can execute within their administrative realm or set of capabilities to bring change to their organization. The weakness I’m highlighting in this post is when the decision and action appropriate for a set of observations, and the observations themselves, exist in two separate departments and can be exploited by the attacker.
To exemplify the problem, let’s begin with the question: Is a distributed denial-of-service (DDoS) attack a security or a network engineering issue? The problem I see most often is that one department will have a precise view into network telemetry for early threat detection, but will not yet have the ability to mitigate or remediate the threat.
Or how about the situation where credentials for one of your executives are harvested, and now the attacker need not trigger any security events at all in order to carry out an attack. In this situation, the attacker merely needs to log in and plan the next stage of the attack, roaming freely and without suspicion across your network because they are only generating networking traffic. The network team does not act because everything is still available, and the security team does not act because there is nothing being denied in access logs, no firewall violations, and nothing triggered on the IDS/IPS.
Let me be clear in that I am not saying that everyone has to become an expert in everything; specialization is healthy, but only when it does not compromise communication! In the example of Mom and Dad being played by the teenager, if Mom and Dad had a better communication protocol, the teenager’s tactics would be thwarted. If you look at some of your operational procedures, and start to think like an attacker (or teenager again), I’m sure you will find at least half a dozen communication gaps that a clever attacker could exploit.
This organizational dysfunction is going to require change that may appear to be radical because the systems that support it are so ingrained. Big analyst firms have defined their technology categories as law and they police them as if you were breaking the law if you dare to create something different – something more cross functional. Some best practices out there are, in my opinion, worst practices because they are so focused on the operational effectiveness of the department, and they leave out other departments or the active threat entirely. Lastly, even some prescriptive compliance requirements force organizations into strict separation of duties, but when implemented, create a broken OODA loop where the ‘OO’ does not connect with the ‘DA.’
There is an opportunity for DevOps to completely recalibrate how we deliver a cross functional operational unit which is why I took the time to write this article for devopsy.kinsta.cloud. Through the lens of Boyd’s OODA loop, you can already see the security ramifications because the tempo of DevOps allows the defense to loop at a faster rate than the adversary – the dominant strategy of the OODA Loop.
I hope this post is a call to action for all you big thinkers. Get creative, get innovative, and do not be afraid to erase some lines and boundaries within your IT organization if it fosters a higher level of communication amongst your teams. Think of your organization like a great band playing live music. All of the performers play different instruments, yet they are listening to one another just as hard if not harder than they are producing sound. They improvise and adapt to one another; monitoring systems are put in place so they can have just enough of each sound in their own ear. This is what effective, cross-functional teams look like. I’m just going to say it: DevOps is Rock and Roll!
Who hasn’t heard the classic statements “It works on my machine!” or, “Man that would have been easier with real data!”? In either case, had the development happened in production, problems would have been avoided! Fortunately, in a DevOps world, the walls between production and non-production get broken down making the reality of developing and testing in production more tangible. In no particular order, here are some of the things I’ve done to breakdown those walls.
Develop on and deploy from a single code line.
Many times we find ourselves in complex processes around deployments and production bug fixes. For example, the code is branched for release candidates and deployments are done from the branches. Branching before release by itself isn’t complicated, however, when there is a production bug, engineers find themselves repeating or merging the fix into at least two, if not several, different places. Repeat that a few times over and, uh oh, now there is a problem because there are many moving parts and questions about what exactly is in production. Unfortunately, the differences are most likely unintentional.
Instead, simplify the process and have everyone work off the same code line and have that same code line deployed directly to production. Now there is only one code base, therefore no question about what code is running in production. Production isn’t static, so change will still happen after deployment. Notation is required indicating what version was deployed, so you can still reproduce production.
Let testing happen on the production database.
Ok, even I’m not that crazy, but let developers use a copy of the production database. There are two very important aspects to this. First, refresh it often. I suggest daily. I have had very good luck using EBS snapshots to get a copy of the database on a different server, but if your database is small, good old SQL dumps also will work. Second, make it safe. Put some safe guards in place to make sure a change doesn’t accidently get made to the real production database. Keep your ACLs tight, change the passwords and practice proper database user management. Lastly, a bonus outcome is that you are constantly testing your database backups and recovery procedures.
Minimize the steps between development and production.
Each step, especially the manual ones, between the developer’s machine and production is another point where unexpected changes can be introduced. It sounds obvious, but the gates code must pass through before entering production must be purposeful and prove their usefulness. For example, if code must go through a regression, but no regressions are being found, then that step is not useful and just increases the probability of unintended change. I have been a part of a process where code had to be deployed to “staging” where it was then hooked up to the live production database, but was only to be exercised in a read-only manner. After doing that for many, many releases and very rarely finding defects before production, we decided to just get better at deploying to production and eliminated the deployment to staging. In the end, we got faster and with no impacts to quality. Less can be more.
Depending on your situation, database size, regulatory compliance, etc. you might to be able to do all of them, but breaking down those walls will yield better code, fewer bugs, happier engineers and fewer sleepless nights to name a few. So, grab that wrecking ball and break down a few walls.
VMware IT has deployed a private cloud based on the vCloud Suite. Dev and Ops teams are one tenant using the cloud to automate dev test provisioning and streamline application development. Automated provisioning has improved how these teams work together. And here is how.
