devopsy.kinsta.cloud

Author: Chris Corriere

The DevOps Juice is Worth the Squeeze

“To such an extent does nature delight and abound in variety that among her trees there is not one plant to be found which is exactly like another; and not only among the plants, but among the boughs, the leaves and the fruits, you will not find one which is exactly similar to another. “

~ Leonardo DaVinci

DaVinci never stepped inside the produce section of a modern grocery store. The food we get today is horribly consistent. The store may have five kinds of apples, but any two of the same kind seem cloned from a mother apple locked in a factory vault somewhere.

A tradition from my childhood my family has recently adopted is the summer peach run. Growing up in south west Georgia we’d regularly stop at random road side stands and buy whatever produce they were selling. Regular fruits and vegetables like corn and peaches but other more colloquial varieties like butter beans and zipper peas as well.

Since I moved to Atlanta it’s become less frequent. We go to the farmers’ markets and occasionally we will find a road side stand in the north west suburbs where I reside, but the quality just isn’t the same. I’m starting to think that many of the road side stands up here are just reselling stuff they bought at the grocery store down the street.

A couple of summers ago my wife and I decided our kids were old enough to start visiting historical landmarks, as we both endured long summers of being educated against our will under the ruse of a road trip ourselves. Our first trip was to Andersonville, near Montezuma, GA. While there’s not much left of Andersonville itself, the story and the national POW museum (which is an international account throughout history) are worth stopping by should you be through middle Georgia.

When we passed a produce stand while driving home on some two lane back road I insisted we turn around. I new we were in the height of peach season and close to the orchards. Being close matters because the best peaches do not ship well.

The peaches they send off elsewhere are uniform in size and at a ripeness where they can survive the truck ride and some time on the shelf before they’re bought. We don’t have to worry so much about them going bad before we eat them because of this particular ripeness. In pursuit of this portability we have sacrificed taste. They won’t get ripe enough to rot but they won’t get ripe enough to really taste like a peach either.

This year we bought 75 pounds of peaches (those are my kids with the actual peaches in the picture). At this volume they end up being pretty cheap, which is good given the high rate of attrition. They’re late off the tree and inconsistent in age and size more than ripeness. Some have rotten pits and must be thrown away immediately. Others are lost in transit. I carefully brought 30 peaches to work with me and ended up tossing the 10 that were on the bottom of the box. They were bruised just from the weight of the fruit on top of them.

Most the folks that come to my daily stand-up told me the peaches they picked had some bruising, but they trimmed it off and the rest was juicy and sweet. I gave one to a VP that stopped by my desk. I hope it was good but haven’t had a chance to ask. A few other co-workers, however, had the best peach of their entire life. I have confirmations that the peaches they selected from the box ruined store bought peaches for them, rendering them oddly disappointing since they tried the fruit I brought from middle Georgia.

Food is meant to be eaten and enjoyed, not shipped. Some time ago taste played a huge part in what seeds were planted the following season. Instead of being shipped across the country produce was bought local and could afford to be tender. It didn’t have the greatest shelf life, but that’s why things like canning, pickling, and preserving are popular in agrarian societies. Preserves made with these peaches (along with muffins, cobblers, and ice cream) taste that much better too.

Software is best served fresh as well

Our users are more concerned with their experience than how the software was developed or how the infrastructure was provisioned. It stands to reason that our innovation should strive to continuously improve their experience as users more than ours as technology professionals. For us that innovation will be an arduous task. DevOps needs the enthusiastic rejection of any yield that is not up to standards. it’s how we get to the juicy sweet quality on the right side of the curve. It’s how you find the changes that redefine how you operate.

The ability to make changes quickly and painlessly tightens our feedback loops, but it is only an intermediate step towards improving our user experience. It allows us to take the energy we spend on deploys and apply it elsewhere so we can continue to raise our standards every time we push code. This is not light work and it is not for the lazy or the faint of heart, but the juice is definitely worth the squeeze.

September 4, 2014
It’s web performance, let’s not call it WebDevPerfOps

Velocity was great this year. It was only my 2^nd time out to Santa Clara, but from what I understand every Velocity conference is great. Few conferences can provide such rapid payback. Velocity does a good job setting the right vibe. The talks are often as entertaining as they are informative. This video of Scott Hanselman is a great example of that. He did an artful job of telling an open source audience they can run their code in the Azure cloud too. I happen to know a few companies that are doing just that. It’s good to see Microsoft continue to embrace open source, as I am a fan of running C# on Linux myself.

Getting a room of passionate experts together and making them laugh over plenty of food and drinks is a setup for awesome collaboration. When this crowd is having a good time the conversation drifts to tech talk naturally. If you’re not working the hallway track at the conferences your attending you’re missing out. I always enjoy finding others dealing with problems similar to the ones I’m currently facing and the two way peer review that comes out of it. On occasion someone has a silver bullet too, where they killed a problem dead in their organization and the solution ends up being easy to implement and highly portable.

As Velocity wrapped up I found myself with a 3:30pm flight back to Atlanta but a hotel with a strict 12pm checkout. I knew devOpsDays was coming up, google.io was going on, and a friend at the conference reminded me webperfdays was about to happen as well. Webperfdays was at Google’s Mountain View campus just up the street. When I looked into it the ticket was only $15.00, so why not spend half a day on Google’s campus talking about web performance if I didn’t have anywhere else to be?

The session was awesome. Webperfdays was a much smaller group than at Velocity. The day included open spaces discussions in addition to main talks and an ignite session. I was learning so much I actually called at lunch to move to a red-eye back to Atlanta so I could stay for the full day.

There were a couple of things that factored into staying after lunch. The first was I didn’t really know anyone there. I recognized a couple of people but it wasn’t anyone I had spoken with much. Looking at the line up for DevOpsDays, I didn’t see too many new names and had talked to most of the main presenters within the past couple of months.

This was all enough to lure me in the door but I stayed because they blew my mind. Andrea Giammarchi’s talk on mobile web performance did real time demos showing how slowing your app can provide a more consistent experience across devices, but assuming only state of the art equipment can lead to performance problems for the majority of your audience. This blog post walks through the talk and provides links to benchmark tools used during the demonstrations.

I also enjoyed Tony Parisi’s talk on webGL. In contrast, he made a lot of good points on how awesome our hardware can be and the crazy things it can do for us. This video covers a lot of the same points Tony made at webPerfDays. I was not aware of the 3D capacity our browsers have or how accessible it is.

My mind sufficiently blown, I came to the realization I hadn’t really been thinking of web performance at all. I’ve been very focused on the CAMS model of devOps, which for the most part meant getting changes to production as safely and quickly as possible with lots of monitoring and reporting. The focus was really on not breaking anything. A degradation in performance does qualify a change as breaking if it’s severe, but making sure our numbers don’t slip isn’t the same as actively trying to make them better. This is a cultural perspective within a shop, and to promote that from within I have to become a web performance advocate myself.

I don’t think I’m the only one guilty of adopting this devOps anti-pattern. It is human nature to pick teams and eventually get competitive. DevOps wants to de-silo our organizations, but I have witnessed industry experts self-segregate at the conferences where we’re working on correcting such problems. This is why I hadn’t really spoken with anyone at webPerfDays while at Velocity: We had all stuck to our respective tracks. The folks I knew were either at devOpsDays or had gone home.

I didn’t have a lot of session overlap with the folks I work with because they were mostly on the web performance track. Since I’m a devOps Engineer it only seemed natural that we stuck to our areas of interest. I’ll even admit I’m guilty of actively turning off my attention on some deep dive performance discussions, because, “it’s not my area and I need to stay focused”.

That thought process stopped as soon as I got back to Atlanta, and I hope you avoid viewing our industry so myopically as well. We have the capacity to study devOps, web performance, and more all at the same time. While we may not always be the experts in the room we can be fluent enough in other areas to be mindful of our impact and appreciate the wins and losses our teams face daily in their respective arenas. We can create a culture of sharing in our organizations, but if we can’t empathize with the information we’re receiving we cannot respond proportionally.

August 14, 2014
Mixed Martial DevOps vs.The Scrum Master

Royce Grace won the most tournaments in UFC history before he retired. With a record of 14 wins, 2 losses, and 2 draws, it’s understandable why he’s considered by some to be the most influential figure in the history of modern MMA. With 100% of his finishes from submissions, he’s undeniably Brazilian Jujitsu royalty. You’d be hard pressed to find anyone familiar with the fighting world that would argue against building a strong base in BJJ while learning to fight. Royce proved that BJJ was the single most effective form of fighting in the early nineties by submitting opponents much larger than him, like Dan Severn, in UFC 4. Severn had just shy of a 100 pound weight advantage on Gracie but ended up getting caught in a triangle choke. I can remember watching the video and trying to figure out how he made him tap like that. All the sudden fighting wasn’t just about kicking and punching anymore, the game had changed forever.

Fast Forward 12 years to UFC when the Master Royce Gracie is to make his return to the octagon. Matt Hughes crumpled him up like a paper napkin. Royce had continued to practice BJJ since his fight with Dan Severn as it was a successful strategy. The mistake he made was investing in a monoculture. Royce kept a razor point on his BJJ, but fighters like Matt Hughes were smart enough to put an edge on all the dull perspectives of their game. Hughes was a well decorated submission wrestler that knew he needed to work on his striking and maintaining his strength and endurance. Ronda Rousey made it clear after her last victory that she’d been working more on her striking too, which makes sense considering she’s already an Olympic medalist in judo.

At the end of the day it’s a fighter that wins a fight, not the discipline they practice. A well rounded fighter is going to be able to adapt to adversity better than someone that only has nth level experience in a particular area. The same is true for software methodology. A scrum master’s team will consistently be outperformed by a team taking a mixed approach. Not because something is inherently wrong with scrum, but that people get the work done, not the process. A team embracing a diverse approach is more likely to do what works. They will be open to more experimentation to figure out what is successful and to confirm when the current model isn’t working. Sometimes it’s the people not cooperating, but telling them there’s a problem because they’re not fully embracing the methodology is usually wrong. Often enough the methodology is being taken as gospel and is embraced too tightly.

I both like and understand kanban, but feel scrum has too much convention to adhere to specifically. I checked Google and it seems to agree with me:

Just to be fair, I checked to see how Google felt about kanban.

But Google didn’t seem to treat kanban as harshly.

If you can’t take my word for this maybe you’ll listen to someone else?

Just the other day Andrew Clay Shafer posted this on Twitter:

I did ask Andrew to offer an alternative approach if scrum wasn’t going to work, and he directed me to slide 46 of a presentation he did in Pittsburgh for a Ruby User Group (shown below). This slide advocates a mixed approach as well. I agree with everything Andrew encourages there, and can honestly admit I need to dig into XP technical practices a little deeper myself. This will not be an abandonment of current methodology and a complete shift in another direction, but an evaluation to see if I can fill some holes in my game.

July 14, 2014
The Double-Edged DevOps Job Title

Last month at GWO in Atlanta I met Nathen Harvey with Chef. I liked his DevOps presentation, “DevOp With Me!”, especially how he used “Walk This Way” by Run DMC and Aerosmith as an example for how collaboration in a DevOps environment often ends up playing out. They literally break down walls in the music video.

While not all of Nathen’s presentations are safe for work ( many include some explicit language. It’s safe to assume so unless clearly labeled otherwise) they are all entertaining and informative. I’d encourage you to check his stuff out, especially if you’d like to know more about how and why Chef and DevOps mix well.

I asked Nathen if he minded me linking his deck and giving him a shout out in a blog post. I even promised to say nice things about him. The answer he gave me only reconfirmed he does really get it:

Of course! (even if it’s not all nice things, as long as they’re honest)

If I’m being completely honest I have to admit this changed the topic of this blog post. I liked the idea of setting devOps to music so much I was going to post a DevOps playlist. I may still write or talk about that another time. It ended up being pretty easy to think of other songs that would fit for one reason or another. It actually even makes for a decent drinking game in the right crowd.

Instead I’m going to talk about the one thing I know Nathen and I don’t see eye to eye on. Nathen’s an advocate for doing away with using the word “DevOps” to describe a role or a team. I, being a DevOps Engineer, strongly disagree with this perspective. When I asked Nathen about this he responded with a single question:

Can you describe what you do without using the word DevOps?

It’s true there are a lot of folks with the word DevOps in their title that are adequately described as a systems administrator or a build automation engineer. Coming up with job titles that don’t contain the word DevOps is also a decent drinking game in the right company. A good serious answer for me would be ombudsman or public advocate. I think KTLO (keep the lights on) custodian would also accurately describe how I’m earning a large part of my keep these days. Still, the job I interviewed for was DevOps Engineer, and for now I’m going to keep using those words to describe what I do. I like Nathen’s Socratic response to why we shouldn’t be describing people and teams with the word DevOps, but what I am doing isn’t accurately described by other job titles where I work. If another title is a better fit for the work you do where you’re at, then I would encourage you to take Nathen’s advice.

The more common reason I’m given as to why I should have changed my job title on day one is really pretty disappointing. I’ve read more than once that I should be concerned that my head will be the first one on a pike should the devOps movement fail. This really isn’t a concern I have.

First of all, DevOps does work. I’m not sure who would dispute that the CAMS model for DevOps is made of things any organization should be trying to improve Kaizen style. I’d even say that improving Culture, Automation, Measurement, and Sharing together results in a sum that is greater than any of its parts.

There are however places that aren’t ready to embrace the change needed for this type of improvement. I caught a quote from @filler when @RealGeneKim retweeted it last November:

If you’re concerned about a lack of progress in echo chamber, go offer yourself up in a different forum. And bring cookies.

Advice I took quite literally. I found a job with a DevOps title at a place that was actively trying to change. I regularly bring in food. Just this past week I baked six dozen toll house cookies from scratch for a DevOps round table meeting I hosted. Why? Because I genuinely give a damn. I want people to come to that meeting to listen and to share. People who are passionate about technology and have faith that there’s always a better path. This group should find such a round table conversation quite enjoyable. The cookies help us remember this is supposed to be fun. They help keep the mood collaborative instead of conflicting. Such a job shouldn’t be left to baked goods that have been sitting on a shelf for weeks. If I want the best from the people I’m working with then I should give them my best whenever I can. It’s often the smaller details that could be neglected or ignored without issue that are noticed and have a more lasting impact.

Some of you right now are in an environment where the bigger details that are making life difficult for everyone are actively being ignored. You may have been told not to worry about them or that the real solution is for everyone to work weekends. This is garbage and you know it. Please do not change your title to something safe and hide under your desk while in the office. Be the change you want to see in the world. There are lots of opportunities to help organizations make this transition. Find one that’s a good fit for you and make the jump.

My advice for businesses is to be very careful when you describe a position as a DevOps role. We know this to be a very specific thing, but if you’re simply shoving the word DevOps into a job title in an attempt to attract talent there’s a good chance you’re in for a rude awakening. The person applying for that position knows there’s always room for improvement. They know that any transition is going to cause some friction but are willing to work through it. They’re also not scared to rock the boat. So, if your organization is not ready and willing to collaborate as one big team it won’t take too long for your devOps staff to go somewhere else on their own. Threatening their job security in an effort to maintain the status quo will only accelerate the process. You may very well end up with a lower head count at the company picnic as warned, but if it’s because all of your talent moved on to greener pastures you’ve effectively turned a bad situation into a terrible one.

If you’re a DevOp-er that has a work environment in that gray area between heaven and a coal mine I’d encourage you to push the envelope. Share your honest opinion with someone you know doesn’t agree with you. Set up a meeting to discuss conflicting perspectives diplomatically. Drop an F bomb in a conference room if it will wake the room up and get some blood pumping (use sparingly and with caution). Don’t expect people not to get upset. Instead, try really heard to shut up, listen to everything they’re trying to say and do your best to give them what they need. If DevOps is anywhere near your title then you have some responsibility in getting these conversations started. If your current employer didn’t know this when they used DevOps as an adjective then shame on them.

May 23, 2014
Are the Milliseconds Worth It?

The pressure to prove one’s coding prowess can make a developer do some very clever and creative things with their source code. Sometimes it’s just a unique approach, but every once in a while this cleverness may shave a few milliseconds off of some performance metrics as well. In the end it’s not worth it. While this may seem like the type of innovation we want from our peers I’ve come to the conclusion it often does more harm than good. A quote that ran across my twitter feed a few weeks ago sums it up nicely: “So if you’re as clever as you can be when you write it, how will you ever debug it?” This pearl of wisdom comes from a book called “The Elements of Programming Style”. It’s a programming book from 1978 that is still entirely relevant. There’s a Wikipedia article you can check out if you’re not ready to take my word for it.

I know I’ve coded something in a unique fashion, and was quite sure I was being brilliant at the time, only to go back later and have absolutely no clue why I wrote the code the way I did. If I added comments I may be able to find my place again but sometimes it takes a while. What about when another developer runs into one of these bright ideas? Will any amount of meditation or contemplation allow them to make sense of what I was doing when I wrote it? Should a new or junior developer be expected to have some out of body experience before being able to work with me? Of course not. I hate it when I do this to myself, never mind when I run across someone else’s flash of genius. I certainly try to keep other devs from going through this type of mind freak as often as I can.

While this exercise can make for a fun parlor game and help extend our knowledge of the language we’re coding in it causes more problems than it solves in a team environment. A wiser developer sticks with a tried and true approach. They’ve been around long enough to know that the code needs to make sense to any garden variety developer that may be working with it in the future (which may in fact be the developer that wrote it in the first place).

I first learned this lesson years ago while tutoring mathematics. I had a developed an algorithm to generate the unit circle. Computer Science majors I was tutoring in math had an easier time using this to quickly generate the circle on scratch paper than arbitrarily memorizing it or understanding the standard proof that shows a bunch of triangles somehow morphing into a circle.

Radian Values of the Unit Circle

The algorithm requires a certain amount of seed data. One must know that a full circle is 2π radians in circumference. This puts the other quarter circle measures at π/2, π, 3π/4 respectively. Only a simple understanding of fractions is required to see this. Next we need to fill the first quadrant, which is only three more values, π/6, π/4, and π/3. From here the algorithm helps us extrapolate the remaining radian values for the circle.

First Quadrant: π/6, π/4, and π/3 (initial seed values, you have to remember these)

Second Quadrant: π – First Quadrant Corollary Value ( e.g., 3π/3 – π/3 = 2π/3, our 2nd quadrant value)

Third Quadrant: π + First Quadrant Corollary Value ( e.g., 3π/3 + π/3 = 4π/3, our 3rd quadrant value)

Fourth Quadrant: 2π – First Quadrant Corollary Value (e.g., 6π/3 – π/3 = 5π/3, our 4th quadrant value)

So, the formula for quadrants two through four can take the form of (n – s), (n + s), and (2n -s), where n = π and ‘s’ is one of the three seed values from the first quadrant. Three simple equations that any CS major should be entirely comfortable with using. There are a few other notes on this proof to get the coordinates, signs, and degree measurements correct. This is not a post about mathematics however, so I’ll leave that as a exercise for the more mathematically inclined to complete on their own time.

Other mathematicians where not as impressed as the CS majors and the student athletes were with this. Watching students that actively despised math struggle with memorizing the unit circle in an effort to complete their core was part of the fun of being good with numbers. It was suggested (by some troll of a jerk) that I might be a math noob myself, some poser that wasn’t any smarter than the fools he was tutoring. Impostor syndrome set in and I I felt I had no choice but to prove my mathematical prowess.

First I went big and reproved what I was doing with a Klien Four-Group. The trolls were quick to point out that this, like my first model, wasn’t a circle at all but just a bunch of points on a plane. Next I came back with a version using a rotational matrix. This did in fact generate a full circle with any value you like on it. The trolls had no more jabs for me but they also fell short of any compliment or encouragement for the work I had done.

As for the students I was helping, none of them quite understood the issue I had encountered. They viewed those trolls as nothing but trolls. Peers that only laughed at their struggle and refused to even try to help them. I ultimately found out my students were concerned that I had gotten sucked into this quagmire. The research I was doing was beyond their understanding and they were still using the first model I developed. It did exactly what they needed it to do. I was informed I should take the rising grades as evidence that my work was of merit and move on to a more productive use of my time and energy.

It was only then I noticed the trolls did have a point. The Klien Four-Group model and my original algorithm weren’t circles at all, just dots with values attached to them. A mathematician would have issue with this where a developer would not, since an object oriented developer should have a deeply rooted appreciation for abstraction. A class called Tree rarely resembles what we know to be a tree. It’s the use case that ends up being important. To a paper company a tree only needs to be represented by the amount of pulp they can expect it to produce. To an apple farmer it probably boils down to the number of apples the tree puts out and maybe its location in an orchard. What we take to be a tree, its size, the shape of its leaves, how it smells, the sound it makes when the wind catches it, are all details beyond the scope of most software we’ll ever write.

This happens to be a huge blocker for most new OOP developers, especially the ones that are good at math. The object model I was using to teach CS students how to do trigonometry ended up being the perfect tool to teach mathematicians how object oriented programming works. The rotational matrix version is important because sometimes non-standard values are needed. It’s useful to show how a method call may use standard values or extend onto another model, like the rotational matrix, when standard values don’t apply.

Mathematicians appreciate the Klien Four-Group model and I’m happy to show it off when I have the chance. More importantly, it’s a reminder to not get too fancy with math they’re putting into software. The math is beautiful more than it’s clever, and I’ve written code I feel the same way about. Like most works of art it stems from a hobby and should be left out of any applied scenario if it’s not practical. If we can get the same result with something that’s simple, something that’s easy to understand, easy to use, and even common, then that’s what we should use every single time because it will be easy to maintain. Wisdom is nothing more than knowledge applied over time. Time and time again I’ve come to the conclusion that maintainability trumps performance. We should be quick to sacrifice any minimal performance gain that makes our code difficult to understand. If a major performance gain is within reach then we’d be wise to abstract any strange code that was used to obtain it. I’d also advise you to comment the living day lights out of it as well, since you may need to remember how it worked yourself someday.

April 8, 2014