devopsy.kinsta.cloud

Category: Features

Q&A: Speaking DevOps and Threat Modeling with the author of Threat Modeling: Designing for Security

If you want to understand how to threat model systems and applications in most any environment you turn to someone who has done so. That’s why we took 30 minutes to speak with Adam Shostack. Adam is responsible for security development lifecycle threat modeling at Microsoft and he is one of the very few threat modeling experts in the world. We thought, with more and more organizations moving toward DevOps and Continuous Deployment, would be a great time to discuss how this impacts the design of secure software. Earlier this year Adam published his book, Threat Modeling: Designing for Security, available here. The book provides pragmatic and actionable advice on how to make security an integral part of systems, software, or services from the outset.

This advice applies to systems and development teams both big and small and is especially crucial as application deployment times accelerate.

devopsy.kinsta.cloud:Why threat modeling? What benefits does it provide the business as well as the application development and operations teams?

Adam Shostack: For the business, executives, they don’t want to deal with individual problems as they arise, they want to address problems as a category. They want to manage a class of things that can be completed in a consistent, repeatable, or predictable way. When it comes to managing application risks, threat modeling gives you an opportunity to answer questions as to whether you are dealing with certain vulnerabilities across all of the applications – rather than coming at them as one-offs. It’s a way to model and abstract away many of the details and look at things in a more strategic fashion.

The other value threat modeling provides is the degree of predictability as you’re developing things – instead of forging ahead and building things and then running a pen test at the end only to learn then that you have serious security problems. What ends up happening in that scenario is that once you fix those problems, you then run another pen test and learn of even more or new security problems and you end up in that very unproductive cycle for a very long time.

So threat modeling helps you think about your security before you start building. This way you can bake security into what you’re doing. My book helps to make both of those value propositions concrete and approachable. You can take these lessons in the book, give them to people and say go do these things and you really get an increased degree of control and predictability through what you’re designing through what you’re delivering.

Also, when you threat model, you get a picture of where things are likely to go wrong. A lot of the threat modeling practice can be very process heavy. That’s why there’s a lot of very process heavy documentation out there. But actually threat modeling integrates well inside a test driven design approach to software development. Why? Because threat modeling tells you where you need to concentrate your security testing. If you’re looking for bug classes, don’t look at random sets of things or relatively random sets of things and you won’t get nearly as much test coverage.

devopsy.kinsta.cloud: So, they can operationalize secure development into the processes, which is what security advocates have often spoke of as the optimal state of security’s relationship with application development and operations – rather than be a barrier at the end of the development process.

Adam Shostack: The whole barrier meme comes from the idea that you do security activity late. A lot of people come into security through hacking. You hear the hacking metaphors all of the time. They are memes like “you can’t secure something if you don’t know how to break it.” Now, knowing how to break things is great. It’s a fun skill. It’s an important skill set. But you notice when you ask about making it operational is that one of the things that security people love to say is you’re never done threat modeling. Now, if you go to a VP of development or a VP of Ops or a VP of DevOps and say, “I want you to do this thing and it’s never done, but it’ll fit into our four week sprints….” You will get a big “huh?” The conversation won’t go anywhere from there.

In order to work it has to be complete-able. It has to be focused and it has to be prescriptive with checklists. It has to have exit criteria, and it has to have metrics that answer: “Are we doing this well?”

devopsy.kinsta.cloud: That type of thinking strikes me as fitting well with the continuous improvement objectives of DevOps.

Adam Shostack: Absolutely. I’ve been threat modeling in various ways for a good 20 years. When I joined Microsoft I encountered people in development and in operations who were passionate about security, who knew that they had to do security and they wanted to do it right. But they also wanted to focus in on reliability, on accessibility, on internationalization, on performance. They wanted to focus on all of these other things each of which is an important property of the system you’re developing. I had to learn to make threat modeling easy. I had to learn to focus in on the highest value deliverables and make sure that those deliverables were accessible and capable. This has to be so even when you didn’t have a security expert groomed because people don’t understand that their decision has a security implication. They’ll make a decision without a security person handy. If you make the security goals accessible and understandable, they’re more likely to understand that what they’re doing has a security implication and they’re more likely to do it well.

Let me give you an example of this. One of the areas that most often goes wrong in the design of a system is people don’t draw their trust boundaries. They don’t know where different principles are going to interact with one another. And application accounts with different authorization levels interacting are a natural place for security problems to arise.

So what you can do is draw these trust boundaries on a white board. It doesn’t have to be a fancy diagram. Scribble on a white board and say, this is where things happen in admin. This is where things happen in the database. This is where things happen as the web user, and this is where web sessions are authenticated and you put boundaries between each of them. Developers understand that these things that they’re writing are run at different privilege levels.

They then say, okay, this code runs on the web user context, this code runs on the database context. That’s great because once you’re doing that you can see the places that your threats are going to arise and look for places to exploit. There are lot of elements in threat modeling that you can work toward incorporating, but the fundamentals are actually easy to teach, easy to learn, and they’re easy to apply.

devopsy.kinsta.cloud:We started out by talking, from a high level, what threat modeling means to different constituencies in an organization. More concretely, what is the payback of threat modeling for developers? Is it more resilient applications, fewer problems that have to be dealt with after the fact, and generally improved outcomes?

Adam Shostack: Those are great values. The biggest “what’s in it for me” for a developer is that threat modeling takes all of these details about security that they want to deal with and provides a model, an abstracted way to frame them. So, instead of constantly worrying “Gosh, I might have another security issue” you can get a sense for how much you’ve done to solve the risk, and how much of it might remain. You get a sense of where you want to prioritize that work.

I think a big misconception is that people believe that only an expert can do threat modeling. One of my aspirations for this book is to show that threat modeling should be like version control. Every developer, every ops person should know a little bit of threat modeling. And it doesn’t have to be a security expert. Perhaps it’s the person who manages your source tree. Or, if you work in a small organization threat modeling will be a side job for someone and that’s okay too. The important thing is to make it a part of your work.

April 17, 2014
DevOps killed the developer star

The DevOps blogosphere is aflame over Jeff Knupp’s post on “How DevOps is killing the Developer”. While I agree with Knupp’s premise that DevOps, Agile and other technology trends are allowing (is forcing more accurate?) developers to do more outside of pure development, even turning into “full stack” developers, I disagree that this is a bad thing.

I disagree with Knupp’s assertion that developers are the Orcas of the IT world sitting atop the food chain as the top dogs. I think Knupp consciously or not, degraded the very real, important and hard job that ops people perform, DBAs do every day, as well as everyone else at an organization. Reading this I was reminded on one time I was sitting in an executive meeting and the VP of engineering actually had the stones to say that his developers were smarter than the rest of the people in the organization and worked harder to boot. Now that is a real team builder right there.

One thing I have learned in 30 years of business is that when an organization is achieving its goals it is because just about everyone is working as hard as they can. It is usually not one group, whether it be developers or marketing or operations that is pulling the sled alone. It takes everyone to really succeed. There is no top of the totem pole or rather who is on top is a constantly changing position as bottlenecks come and go, priorities shift and the market changes.

In any organization whether it be startup or enterprise, we all have to do what we need to do to accomplish the mission. There you have it. Working in the Federal market all of those years taught me something. In the DoD and armed forces it is all about the mission. We have missions as individuals. Those missions can be writing code, deploying or monitoring (heck even support and sales), but ultimately those missions all fold into the greater mission of the organization. If developers doing full stack work helps the organization better accomplish its mission and goals, than it is good. No man is too big for the army.

At its core that is what DevOps is too don’t forget. Empowering IT to better serve and meet the organizations goals. Not about letting developers do what they love to do or what floats their boat or what they consider the “top of the totem pole”. Enterprises are here for one thing, the bottom line. It is something we should all never lose sight of.

DevOps is changing the mission of developers. It is a change in the environment if you will. Species are faced with changing environments all of the time. Those that adapt to change succeed and even flourish. Those that fail to adapt fall into the extinction bin. Generally species that are generalists adapt and thrive much better than species that are too specialized. This is basic evolution theory.

I think to Mr. Knupp’s point we are going to see a form of DevOps Darwinism play out. Developers who insist they only develop and don’t do full stack or other DevOps type of tasks will find shrinking habitats and less opportunity. They will exist on the margins. Developers who embrace this new environment will find abundant “target rich” environments with rewarding (both monetarily and stimulating) career opportunities.

I am all for doing something you love for your work. Otherwise I would be practicing law somewhere. I stared working in the technology world because playing with computers was the most fun I was having. I am sure many of you had the same experience. Messing around with code or scripting moved from fun and something you did because you liked it to something you also do to earn a living.

Earning your livelihood doing something you love is a great blessing by the way. There are untold numbers of people who trudge off to work every day and have to do something they dislike, are not passionate about and do just for a paycheck. I can’t imagine having to live life like that.

I think that is what Knupp is talking about too. That developers code because they love to, they need to, and it is what they do. Forcing them to do other tasks takes away from their focus and stunts their growth, which in turns comes back to hurt the organization as a whole.

I don’t think developers will ever stop coding. Mr. Knupp is right, developers code because they want to. That is what they love to do and it is why they became developers. I think however they will have even more opportunity to code, not less as software continues to gobble up the world. It may be a different kind of development, with coding as infrastructure as software dominates, but developers are still going to code no matter what.

We may be seeing the rise of the coding generalist versus the pure application developer as the lines between developer, ops, QA and yes even security continue to blur and overlap. But I don’t think it is a bad thing for developers, enterprises or anyone else. It is just part of the constant change in our environment.

I don’t think this change is bad at all. In fact to paraphrase the great Gordon Gekko, “Change is good, change is what made America great.”

April 16, 2014
Microsoft bridges the gap between Azure and DevOps

DevOps can be easy for a startup. Many of the concepts and principles of DevOps come quite naturally to a fresh company just getting started. It’s a different story, however, for large, established enterprises trying to wrap their arms around this DevOps thing. For IT admins working in Microsoft’s Azure cloud platform, though, things just got a lot easier thanks to the integration of Chef and Puppet Labs.

Many organizations are heavily invested in the Microsoft ecosystem. IT teams and administrators are familiar with Windows domains, Active Directory, Hyper-V, and Azure. They know how to write scripts and leverage PowerShell to get things done, but they’re much less familiar with the core principles of DevOps, or common DevOps automation tools like Chef and Puppet Labs.

One of the driving forces of DevOps is the ability—or perhaps necessity—to automate those tasks that can be automated so IT resources are freed up to focus on more important tasks. Deploying and configuring virtual servers and applications is necessary, but it can also be tedious, and it is a task that can be easily automated using DevOps tools.

The arrangement also works the other way. Smaller companies that are already familiar with, and invested in DevOps may need to scale to a platform like Azure. They should be able to build on what they’ve started, and continue to use the tools they’re familiar with as the company migrates to Microsoft’s Azure.

One of the main attractions of Microsoft Azure is the degree of flexibility, choice and control it gives customers. Microsoft Azure’s partnership with Puppet and Chef allows customers to choose from an array of existing technologies and communities to easily deploy, configure, automate and manage their VMs.

According to a Microsoft spokesperson Puppet Labs and Chef complement and extend Microsoft technologies such as Resource Groups and PowerShell Desired State Configuration (DSC) to offer more options for customers to migrate existing or deploy new applications to the cloud. The partnership offers Microsoft the ability to support Linux and UNIX environments, and also alleviates the concerns that organizations already using Puppet or Chef will have about abandoning their investment. Now, they can continue to use their Puppet modules or Chef recipes in Azure.

Luke Kanies, CEO of Puppet Labs believes the integration of Puppet Labs and Azure will help Microsoft IT admins by making it easier to deploy applications across any platform or cloud environment. Kanies explained in a press release, “Our goal is to help you automate the configuration and ongoing management of your machines and the software running on them, so you spend less of your time firefighting and running simple scripts across a thousand machines, and more time getting great software in front of your users.”

I spoke with Kanies about the difference in dynamics between a small company and a large enterprise. He told me that in a small company it’s much easier to communicate and coordinate resources, and the cost of failure of any particular project is much lower. For a large, established enterprise, however, there is assumedly a large base of existing customers, and there is more fear and worry about possibly disrupting normal operations.

I also had a chat with Ken Cheney, VP of business development for Chef. His philosophy mirrors a lot of what Kanies described. Cheney told me that emerging companies have the luxury of starting from ground zero with a clean slate. There aren’t yet any established processes to navigate. The challenge is trying to transform an organization with established processes, and a team of existing developers over to a DevOps mentality. Integrating DevOps into Azure helps organizations to meet that challenge.

DevOps adds an element of speed and simplicity that makes it easier for larger enterprises to be more agile. Using tools like Puppet Labs and Chef enables IT admins to add speed as a core requirement for infrastructure projects without disrupting or conflicting with other requirements—like security and stability.

Ultimately, the new alliance illustrates Microsoft’s commitment to empower Azure customers with the best-in-class configuration and automation tools by embracing the technologies, communities and ecosystems that developers know and love. Combining Azure with powerful and popular platforms like Chef and Puppet Labs bridges the gap between traditional IT and DevOps—enabling old school IT admins to embrace DevOps, and DevOps startups to mature and evolve into larger enterprises.

April 15, 2014
Expanding DevOps requires a supportive IT community

How do we bring newbies into the DevOps community? Until they know the melody, they can’t appreciate the song.

While leading a DevOps event, the speaker polled the audience about their knowledge of DevOps. All of the attendees were well acquainted and actively engaged. It was obvious there were no newbies in the room. It made me wonder if we were yelling into an echo chamber filled with people who were already on board. Where were the newbies?

My concerns that day reminded me of a 1990 Stanford University study led by graduate student Elizabeth Newton. She assigned groups of two people to play a communications game. The first person picked a simple song that everyone would know (like Happy Birthday), and tapped the rhythm of the song on the table. The second person was the listener and tried to guess the song. On average, the tappers anticipated a 50 percent success rate. In reality, the listeners only guessed the song correctly 3 out of 120 attempts (2.5%). [Source: Health & Health, 2007 pp. 19-20]

That happened because the tappers already had the multidimensional melody playing in their heads. Conversely the listener was unaware of the melody in the tappers’ head and was limited to the one-dimensional tapping patterns. When tapping was the only means of communicating the song, the listeners almost always failed to guess the song title.

What’s the Point?

As DevOps enthusiasts we have experience and expertise — we know the melody. We have pre-existing knowledge and confidence that this new model works. Many of us have years of experience, experimentation and a network of insightful supporters to advance our learning. We know the DevOps founding stories and have read a myriad of blogs, articles and books. We have listened to the webinars and podcast cafes.

As the DevOps message tappers, we hear the melody of innovation and progress. And, in our minds, we assume we are communicating our DevOps stories with clarity. We point to effective projects, tools and teams, and create new models and terminology, expecting everyone in the IT community, or those still working in more traditional silo-ed organizations, to comprehend our activities. The listeners, however, may not have the same experience, confidence or base of knowledge. This is especially true of IT teams in enterprise organizations. Many of them have never read the books or heard the compelling stories. They have achieved success and grown their careers with Waterfall, in silos, with hand-off challenges and with technologies from past generations. Maybe they have never even played with Open Source technologies, worked with an open community, or explored the benefits of Agile methodologies. Thus, when many DevOps enthusiasts speak, their message is not understood by traditional IT professionals or those still working in silo-ed organizations.

Break through the IT barriers and silo-ed structures!

Breaking through to traditional IT professionals and silo-ed enterprise contributors begins with making the assumption that some may not have sufficient knowledge or experience in this area. This does not mean enthusiasm in DevOps presentations and conversations have to be toned down. Rather, it means they need to begin at the shallow end with stories and then move listeners to the deep end with more technical and detailed information (e.g., new ways to systematically reintroduce lean, agile, continuous integration, continuous delivery, continuous improvement models, etc.). The best way to accomplish this goal is to think specifically about traditional IT professionals and their specialty niches. When making presentations or engaging with them, consider their view, context, roles, work silos, and lack of opportunity to see their contributions make a difference. Then endeavor to inspire them to increase their knowledge base and expand their network by exploring collaboration opportunities internally with co-workers and externally with other groups.

Some presenters and team leaders find it effective to begin their group sessions and new conversations with stories from their initial exploration or foray into DevOps. Others employ familiar IT language, ideals, expectations and activities before transitioning into a DevOps melody. This will serve to expand the community to those who are well entrenched in older models (e.g., traditional Waterfall-based organizations, etc.).

Through greater understanding and deliberate behavior, much can be accomplished when legacy and DevOps groups collaborate: common goals are often more rapidly reached; polarization between old and new models avoided; and DevOps practices expanded across a wider IT community. Externally, this would ensure a wider community of support for DevOps and an ongoing flow of cross discipline collaboration and insights: and internally this can lead to faster innovations, improved corporate unity and accelerated success in the marketplace. Leaving new invitees outside of the DevOps conversation (intentionally or unintentionally) could hurt our DevOps efforts more than we might ever know. Without building in correlation or transition concepts between old and new, in everything we do, speakers and leaders could spend endless amounts of time and effort tapping “Happy Birthday,” but very few “newbies” would ever have a chance to recognize, support or see the value in our DevOps melody.

April 14, 2014
Camp DevOps @ GlueCon 2014

devopsy.kinsta.cloud is very pleased to announce that in conjunction with Eric Norlin and Glue Conference 2014, we will be presenting our inaugural Camp DevOps @ Gluecon on May 20th, one day before Gluecon 2014. Camp DevOps will take place from 8:30 am until 4:30 pm. The event will be held at the Atlas Institute Building at the scenic University of Colorado/Boulder campus. Here is a link to a flyer with more information here. You can go also get more information by going to Campdevopsy.kinsta.cloud (site is still going up, so be patient if there is a problem) or our Eventbrite page.

Moving devopsy.kinsta.cloud beyond the 4 walls of the website has always been part of our business plan. Camp DevOps is our own conference brand. Martin Logan, devopsy.kinsta.cloud co-founder has held Camp DevOps before, but this will be the first event since our commercial launch. Over time what Camp DevOps is and what to expect we hope will be better understood. But since this is the first, let me quickly give some background.

We recognize there are no shortages of conferences around DevOps. Camp DevOps is not meant to compete with DevOps Days and similar conferences. We think DevOps Days are great events and we will be working with local Meetup groups to help them with DevOps Days in the future. Camp DevOps are also not meant to compete with Velocity or other enterprise events. Instead Camp DevOps like devopsy.kinsta.cloud itself is meant to pitch a big tent. We will have content and activities for the different constituents of the growing DevOps community.

CampDevOps @ Gluecon 2014 will feature keynote presentations (Sanjeev Sharma of IBM and author of “DevOps for Dummies”, Rajat Bhargava, CEO JumpCloud and Howard Diamond, CEO of MobileDay already confirmed), multi-track presentations with one geared for technical audiences and another aimed at business and cultural topics, panel discussions and what we are calling “Hello World”. Hello World will feature hands on learning with DevOps tools. Attendees will need their own computing device to follow along.

There will also be table top displays, Continental breakfast and lunch served. We are hoping to have a sponsor for dinner as well. The cost to attend Camp DevOps is just $49.99. If you are already registered for Gluecon, you can attend for free. If you are not registered for Gluecon and register for Camp DevOps you will receive a $100 dollar credit towards Gluecon registration. There will also be complimentary shuttle service from the Gluecon hotel, The Omni to the Atlas building at CU/Boulder.

Attendance is limited so please don’t wait to register!

Besides the keynote speakers, we also will have presentations from AppDynamics, MongoDB, VictorOps and others. We will be updating the agenda and presenters daily. We still have room for more track speakers, keynotes, panel and Hello World presenters. If you are interested contact us at CampDevOps@DevOps.com.

Sponsorship packages are also still available. There are sponsorship opportunities in conjunction with Gluecon and CampDevOps or just Camp DevOps. Contact us at CampDevOps@DevOps.com for details.

April 9, 2014
The Importance of the Bus Driver

The topic of DevOps has been a topic for many a debate over recent ~~months~~ years. Without adding fuel to the fire I’d like to present a slightly different take on the DevOps culture.

The phrase ‘what if I was hit by a bus?’ can often be heard within the vicinity of software development teams. It relates to a measurement known as the ‘bus factor‘. Individual talent should not be discouraged, however when it becomes silo’d that talent translates to risk for a business. Try to think back over your career – do the following sound familiar?

“I don’t know how to do that, X usually does that”

“X is away at the moment so we’ll need to put that on hold”

“X left yesterday and didn’t cover this in their handover”

If you have read ‘Phoenix Project‘ – think of Brent…

Using the fated bus driver I’d like to take us on a journey (excuse the pun) through a DevOps culture.

In-sourcing the bus driver

You work for a small green minded startup. Being green minded and wanting to help your staff, you decide to buy a bus to get everyone to work in the morning. Realising that you don’t actually have a dedicated bus driver, you decide to take it in turns to drive the bus in the morning. Everyone gets involved, everyone knows how to drive the bus, everyone knows which route is best.

Idea 1

Don’t over engineer your processes. At this stage you’re already exhibiting DevOps characteristics!

The talented bus driver

Your small team has performed really well, people like what you do and now your team isn’t quite so small. There are now over 50 people on your bus in the morning. It wasn’t long until you realised that Sarah is the best bus driver you have. Inevitably, Sarah usually just takes up driving the bus each and every morning.

If Sarah is off, Mike takes over but he often makes people late for work and tends to hit a lot of parked cars.

Furthermore the bus recently broke down when Sarah was off, Mike and your bus passengers all took a look at the engine and tried to help but there seemed to be a ‘magic’ piece of string holding two parts together.

When they consulted the bus manufacturer manual, this piece of string was no where to be seen. Mike had to call Sarah whilst on holiday!

Mike calls the bus company to ask for a replacement bus, to be told it will be 2 hours until a second bus arrives. The team sit and wait patiently.

Idea 2

Try to identify your tipping point. Some key indicators are reliance on individuals, information silo’s, phrases such as ‘oh I didn’t know it functioned like that in production’, production environments start to diverge (without communication) from development environments.

How many production environments have the web server, app server, database, load balancer et al all running on the same machine? Is this how your development teams work? Does this differ to the view of the operations team?

Your bus stops

We’ve identified a few problems but lets focus on the positives. Firstly, let’s celebrate actually spotting the problem, it is very easy to continue with the status quo. Our second success was the reaction of our bus passengers. When the bus broke down, instead of complaining, the bus passengers tried to help mend the bus.

Idea 3

When and where your bus stops is less important than who it stops for. It might be stating the obvious, however if you wish to move towards a DevOps culture you need talented team members. You need people that care for the organisation and their profession. A desire to share knowledge for a greater goal.

If your team are not interested or do not care, then no matter how much you say DevOps you need to boost their interest first.

Hopefully you’ll start to hear your teams utter phrases such as “What does production look like?”, “You could do this slightly differently you know”…

What if your talented bus driver has an RTA?

There are no doubting Sarah’s talents – I mean come on, she was able to fix a bus engine with a piece of string, knew the best bus routes and always got your team to work on time…

What would we do if Sarah left or was, somewhat ironically, hit by a bus?

Idea 4

Could your organisation raise the value of sharing and encourage knowledge share?

Assuming you’ve got the right people on the bus, you might also find that other staff members have ideas for improvement when you explain the current process. Your Sarah(s) should not fear sharing their knowledge and should be rewarded/encouraged for doing so.

One of your passengers may even know a faster journey to work or a better way to fix the bus engine.

The bus seating plan

Although you didn’t enforce it, you start to find groups forming on your bus journey, all your teenagers are sat together at the back playing music on their phones…..

Idea 5

Don’t let job titles restrict conversations. Water cooler chats are incredibly productive and help teams share ideas.

Replacement bus service

When the bus broke down Mike was able to order a replacement but it had a 2 hour turnaround time.

Would it be better if we always had 2 or 3 buses running? If each bus had enough capacity for all 50 passengers we’d probably always make sure people were on time for work.

It would be even better if all bus drivers knew the routes that Sarah used, all buses had the same piece of magic string and all bus drivers knew what the magic string did.

Idea 6

Something my team often hear me say is ‘Design for failure, don’t just react to it‘. When designing your applications and services try to think how the system can accommodate failing services better rather than what actions you might take in the event of failure.

The same might be said for your human resources. Have your teams got a shared knowledge? What would your bus factor score be?

Journey times

Try to keep your passengers informed of your progress – you could let them know their ETA for work. Informed passengers tend to be more engaged and understanding. For example when telephoning a business, do you prefer to know what position you are in ‘the queue’ or would you rather sit on the line waiting without knowledge of when you may be spoken to?

Idea 7

Monitoring might be considered a key technical section of DevOps adoption. There are many, many tools that can help with this such as StatsD, Graphite, Librato, New relic, AppDynamics etc but whichever medium you choose, try to make sure your ops stats are in full view of your various teams not simply your ops staff. Get them in front of your dev team, support team, sales team, HR team they impact every member of staff.

Hopefully, you will be surprised at an increased level of employee engagement. When ops are in war rooms fire fighting problems, each team member will understand the pressures they are under, some team members will also express a desire to help.

You have arrived at your destination

At this stage I may have flogged the bus concept a little too far. Are the concepts here a guide for the adoption of a DevOps culture? Does the ‘D’ word have to be attached to the guidelines above? I’ll let the community decide.

April 8, 2014
Secrets to Collaboration in DevOps

In IT and engineering departments, the notion of teamwork hasn’t been a priority until recently. People were hired based on their in-depth knowledge of a particular technology category, such as networks or operating systems. There wasn’t much need for someone in networking to talk to his server colleagues, unless the whole team was going out for a beer. Those were the happy days of ITIL, in which processes and procedures for maintaining systems and solving problems were linear and preprogrammed. With a solid configuration management database on hand, the world of IT management was a well-oiled (although sometimes slow) machine.

This approach was okay when companies were only relying upon a few large application suites. But now, of course, this arrangement has been turned upside down. With hundreds of applications running the business, and those applications dependent upon hundreds or thousands of services and components, ITIL falls short. People working in silos can no longer effectively manage performance and resolve issues with this level of infrastructure complexity. Companies nor their customers don’t have the patience to wait hours for a website to resume service or a critical application to function properly again. These disruptions cost too much money to fix and can darken revenues and customer relationships.

Enter DevOps: the new way to deal with rapid change and complexity through ongoing releases and frequent adaptations to the environment. In a 2013 survey of IT decision-makers by research firm Vanson Bourne, the top reasons for adopting DevOps were the need for simultaneous deployment across different platforms, and business pressures to release apps more quickly to meet customer demand and to improve the customer experience. A majority of the 1300 survey participants, 66%, said they either have a DevOps strategy or are planning to implement one.

At the core, DevOps success relies heavily on collaboration between developers and IT operations staff. Only through the daily, open sharing of knowledge and best practices can a company solve problems and optimize applications efficiently in the DevOps fashion. To complicate matters, employees are living and working all over the place today. Bringing their insights together when it matters requires not merely the right tools, but the right culture. Here are some ideas for fostering collaboration in DevOps environments.

1. Get rid of departmental silos and rigid processes. With a culture focused on ITIL, problems eventually get resolved—but often after a few hours or even days of running a question through the ranks; by then the damage is done. This is no longer acceptable when employees and customers expect always on, always speedy service. After all, that’s the standard they readily achieve from their personal applications and devices. If your organization relies upon modern, highly scalable and visible applications, you need to break down internal barriers without delay. Forget about DevOps for a moment, and determine how roles might need to shift or rigid processes (such as approval chains for changes) may need to dissolve for the sake of fixing a problem right now. Empowering individuals to make changes immediately when conditions justify it is how technology organizations must operate today.

2. Encourage discussion: Do employees only talk in the break room? If so, start scheduling daily standup meetings for the purpose of sharing status, timelines and exploring ideas. Finding the right time to accommodate people in different time zones might be tricky; in large, dispersed organizations, holding two daily meetings may be required.

3. Eliminate hierarchies: Even in the flattest of organizations, informal hierarchies persist. With DevOps, however, it matters little if you are “the smartest person in the room” or have been with the organization the longest. Maintaining organizational hierarchies too often works against the goal of sharing information across the organization to better IT operations and application performance. Managers should work hard to encourage open sharing and the notion of equality among team members. Reward employees for their teamwork efforts and for showing mutual respect. Instill the notion that everyone is responsible for delivering excellent service and for resolving problems, not guarding turf. While this is an idealistic philosophy that’s hard to carry through on a daily basis, working toward this goal will in time grow a collaborative culture.

4. Develop a business orientation: In a DevOps culture, understanding the needs of the business is the highest priority. In fact, respondents to the Vanson Bourne survey said that the top three skills needed for DevOps are knowledge of business, priorities, strategies, or metrics (47%); knowledge of current business processes (42%); and communication skills (32%). Keep these qualities in mind when hiring new employees and when promoting those from within to senior roles. Create opportunities in which technical staff can interface with business leads to gain that critical business perspective.

5. Set the tone at the top: As in most companies, the people in charge maintain a heavy influence on corporate culture. So, what if your CEO places a higher value on individual contributions than teamwork? It may feel like fighting an uphill battle, yet the CTO of a CIO or other person in charge may need to spearhead an education initiative to help higher-ups strike a balance. That can result in small yet powerful changes, such as instituting new employee incentives and more frequent performance review programs that align with DevOps goals.

6. Sometimes, outsiders are required: DevOps is still a fairly new practice, and companies may not have in-house expertise to develop a new culture around it. Fortunately, there are an increasing number of outside coaches that can help organizations jumpstart new ways to organize and work. These experts can also objectively assess progress and help team members navigate difficult transitions or high-stakes projects. If such a coach can help your team get up to speed 50% faster on new collaborative practices, it will be time and money well spent as the business will see results faster too.

7. Attitude is everything: On the surface, it would seem that younger people, those highly coveted “Millenials”, will be the front runners in DevOps organizations. Yet don’t make assumptions. If an older employee with decades of experience in operations and across different industries is willing and open to change, he or she could be the most valuable person on your team.

8. Get rid of the dinosaur tools. Traditional APM and IT monitoring tools from enterprise software vendors may not have the right interface nor feature set to help DevOps teams keep up with cloud and hybrid infrastructures. Whatever tools you select, make sure they incorporate the real-time, collaborative elements embedded in today’s top consumer apps and social media websites. Think open forums, comment streams, “more like this” intelligence, messaging and the ability to upload videos and images that enhance a conversation. If the tools are fun, simple and allow people to get the information they need in one place, they’ll use them. If manual correlation of data and events is required using a clunky interface, they won’t.

What’s powerful about DevOps is that while it requires new attitudes, methods and skills, the end result is that people are working together toward a common goal. Better collaboration doesn’t just serve DevOps, but an organization at large. It could result in a new development approach that kills the competition, or an IT management best practice that saves the company hundreds of thousands of dollars a year. Collaboration is the quintessential soft business skill, yet with the right intent, it can bring hard results that move the business forward.

April 7, 2014
DevOps Is About Serving The Customer: Q&A with Shridhar Mittal

As a longtime proponent of continuous delivery and fast deployment schedules, Shridhar Mittal has advocated for DevOps operational patterns since before the DevOps moniker was even coined. It was a philosophy that stemmed from what he and his colleagues at what was then Interactive TKO saw happening in response to the service-oriented architecture movement: many organizations were slowing down their releases as opposed speeding them up, which was the whole “promise” of SOA, he says.

“So we started a whole company around how do you change your development processes to get speed with quality in this new world of distributed architectures,” he explains.

Since iTKO’s acquisition by CA Technologies in 2011, Mittal has used his current role as general manager of application delivery at the company to focus his efforts on evangelizing DevOps cultural shifts within the enterprise. He believes that DevOps is on the brink of revolutionizing the way IT departments function and in addition to his anecdotal evidence from frequent talks with IT leaders, he points to a recent survey CA conducted together with IDG for evidence. According to technology decision-makers, while only 7 percent of organizations have already adopted DevOps practices with enterprise-wide scope, it is gaining traction. Another 39 percent report that they’ve either implemented DevOps principles with limited scope or are in the process of adopting DevOps. And another 28 percent are either planning on implementing or are evaluating DevOps.

devopsy.kinsta.cloud recently spoke with Mittal to discuss the mainstream uptake of DevOps, the biggest transition challenges and the attitudes about IT job performance in a new era of cooperation.

devopsy.kinsta.cloud: Why do you believe that DevOps is gaining so much traction today?

Mittal: I think the whole promise of DevOps is around better customer experience. Better quality for the customer, but in a much, much faster way than you could do it before. To me, that is ultimate goal.

Ever since the whole consumerization of IT, everyone uses the web and everyone uses mobile devices, and there is no buffer in between the application and the end consumer now. Previously to that you used to have, in the call center, you had people who are using applications and the consumer never actually interacted with the application.

So if it sucked, its internal people that had to deal with that–it wasn’t the consumer. Now, you cannot afford to have your customers have a bad experience because then you are going to lose business. One of our case studies is the .com part of a very large company–one of the Fortune 100. They had done a study that showed that for every one second delay in the response in the website dropped the conversion rate of sending someone to the next step of purchasing. It dropped significantly for every second.

Think about it. If you’re doing a big marketing event, you’re sending hundreds of thousands of people to your website, and just because your website doesn’t respond properly, you lose that many more potential customers.

That’s the kind of value that we’re talking about right now. Take a simple example, say there’s the launch of a phone and the telecomm company wants to release it at the same time the manufacturer is releasing it and the systems are not connected in time. They would end up losing hundreds of millions if not billions of dollars of business because they didn’t release it at the same time, and one of the competitors actually released on time.

These are the kinds of the hundreds of millions worth of value we’re talking about. In the past, people thought about it as, maybe if I can get three percent or four percent more efficiency I can save x numbers of millions. But what I think they’ll really be looking for now is the ability to grow their business much faster.

devopsy.kinsta.cloud: How can organizations start meeting the challenge of transitioning to DevOps?

Mittal: Start small and then build up to the big There are at least three different on-ramps we talk about to get to DevOps. One is, just start with the development side of the word and decouple the development teams from each other so that they can do their things faster and still be able to deliver things on a higher quality. That’s using things like server virtualization and API testing. The second on-ramp is just improving your environment management, the release management capability. People today are spending tens or hundreds of man-hours creating and maintaining these environments then tearing them down and then bringing them up again.

So figure out how automate that process where you can take so many man-days of work and bring it down to something that can be handled by one person and can be done as a self-service in a matter of minutes as opposed to taking days weeks or months to happen.

And then the third on-ramp is to put more automated testing processes in place, for instances doing performance testing. Trying to do it in-house today is becoming more and more difficult because there’s never enough infrastructure internally to be able to handle simulating these tens of thousands and hundreds of thousands of people coming to your web site. It’s just not possible. So figure out ways to leverage cloud-based automated testing capabilities to support that.

There are many different on-ramps and people can start from one or another and then start building to much bigger things. Start small and eventually expand to put it across the enterprise.

It is also important to get the trust built between developers and operations through better visibility. The classic scenario is, the developers just want to write code, release it, then they want to go back to writing code again. Operations want to slow things down because all they care about is a stable environment. If I could get devs to have visibility and ownership and be measured on how the products are actually behaving in production and, same thing with ops, if I could give them visibility to what’s in the pipeline, what’s coming out, why is this coming out, what is the business case and how this impacts your business, they’ll all start feeling an ownership for the whole process.

devopsy.kinsta.cloud: That brings up an important question: how critical is it to change job performance metrics in both development and operations in order to make DevOps work?

Mittal: I think it’s very important. I think it’s not something that has to be done from the get-go but I think somewhere after you have done the first couple of projects and some of the people have started buying into DevOps, that’s when you have to fundamentally change the culture of the company. In my mind, that’s contracted behavior—and it’s not because people are coin-operated.

I’m not saying this in a bad way. To me, people look at the way they’re comped as a way of understanding what their direction is. What are they being asked to do?

So if they are not getting measured on quality, to them, that means “Just make sure that you just get things out as fast as possible and then if you have to roll back, then you have to roll back.” And if you’re not getting measured on constant speed, then that’s like being told by the executives that, “All I want you to do is have a stable environment; I don’t care as much about getting new stuff out there.”

So there has to be changes in the comp structure, how people are rewarded, to really change the culture of a company.

devopsy.kinsta.cloud: What other things need to be done on the line-of-business executive leadership side of the house to make DevOps work?

Mittal: I think that the business side needs to be the driver to a large extent, because the business needs to say why something is important. If supply of IT capabilities is much less than demand, that means there has to be some kind of prioritization of work. And who’s making that determination? Who is doing the prioritization? To me, without a business context, you can’t be doing direct prioritization.

I think the business has to be very involved in helping IT understand a prioritized list of what needs to get done. They don’t say how it needs to get done, they just need to say, “This is more important than this, and this is the impact to our business of that.”

April 7, 2014
DevOps Can Be Your Fountain of Youth

There is a different, almost magical dynamic, at a startup that seems to fade away over time as companies establish themselves. One of the benefits of DevOps, though, is that it creates an environment where that startup magic can continue to thrive. DevOps is like a fountain of youth that lets corporations continue to act like startups where it counts.

Once upon a time Microsoft consisted of nothing more than Bill Gates and Paul Allen hacking away at a keyboard. Like any successful company, it grew….and grew, and grew, and grew. And, as it grew the culture changed, and departments were created, and bureaucratic policies were implemented. A task or feature that Bill or Paul would have just cranked out on the fly in an afternoon during the early days now must be vetted and approved, and go through a chain of command, and maybe a focus group before it can finally see the light of day.

Startups are often formed by a few friends over pizza and beer while watching TV. In the beginning, things just get done. There is generally no real hierarchy of control, and everyone just owns what they’re working on and they get it done.

Startups also remind me of watching Star Trek: The Next Generation. It always seemed like everyone new everyone else’s job—and anybody could fill in for anybody else. If Geordi La Forge—the chief engineer of the USS Enterprise—was captured or incapacitated, Worf—the Klingon chief of security could just step in and manage the warp core in his absence. Although everyone had a designated role, they all understood enough about the other roles to take over in a pinch. That’s how startups are before things grow so big that each person barely has enough time to keep up with their own department and duties, and barely knows what anyone else is doing.

That brings me back to DevOps. The essence of DevOps requires that individuals take both initiative and responsibility for what they’re working on. The culture of DevOps is more fast-paced, and it doesn’t allow for protracted bureaucracy to drag things down. An organization that embraces the DevOps culture is forced to retain the innovation and agility that defined the startup stage.

Granted, a large company like Netflix, or IBM still has a management hierarchy. There are policies in place, and different departments for IT, and human resources, and accounting. The IT guy can’t simply cut his own checks, and the accounting person can’t just go rogue and hire people without the knowledge and consent of HR. There are defined roles, and most of them serve a distinct and valuable purpose.

That corporate structure still exists, but with DevOps you maintain that startup mentality where it counts—where new ideas are being developed and implemented. Of course, it’s also not without its challenges. Balancing the autonomy of DevOps with the policies and bureaucracy of a larger, established corporation can be confusing, and it’s important to have boundaries in place for just how long the DevOps leash is, and where the threshold is at which things need to be escalated up the chain.

DevOps isn’t about a bunch of rogue employees running around autonomously wreaking havoc on the company, though. Much of DevOps seems to focus on tools and technology—and those are a crucial element of automating routine tasks and facilitating DevOps—but, the power of DevOps from a cultural point of view lies in collaboration. Done right, DevOps fosters an environment that frees employees to work together to find the most effective and efficient solutions.

You probably won’t be able to recreate a situation like Star Trek: The Next Generation in a larger company. There will be departments, and there will be designated roles that need to be filled. With the right DevOps tools and technology, though, you can automate and speed up routine tasks, and free up time previously spent on those tedious jobs for employees to devote to solving bigger problems. With the right DevOps culture, you will unchain those employees and enable them to think and interact more creatively to come up with innovative ideas that may never have seen the light of day in a traditional corporate bureaucracy.

DevOps can enable your company to function with the drive and agility of a startup, while still maintaining the wisdom and maturity of an established corporation.

April 3, 2014
Deployment and Monitoring Automation with glu
glu is a free/open source deployment and monitoring automation platform. glu is a project that was started at LinkedIn mid-2009 to address the exponentially growing needs of deploying the set of applications and services that make up the LinkedIn experience. Although some projects like Chef and Puppet existed at the time, they were mostly good at configuring the infrastructure (creating users, installing java, etc…). glu lives in a higher space: provisioning dynamic applications on an ensemble of machines (change often, real-time failure detection, etc…).

glu is in the application deployment space

First, let’s start with some definition: when I say application, I mean a piece of code that usually needs to run, and usually offers some form of api to talk to (which, at a lower level, is a socket listening on a port). Equivalent terminology is service or server. For example, a webapp server is an application. Although glu can also deal with one-off programs (run, do some computation and then stop) or infrastructure (perl, java, etc…), it is not optimized to handle these use cases and other projects are better at this task (Chef/Puppet).

What is particular about applications in general, is that they tend to change a lot more often than infrastructure. In the age of continuous deployment, it is not uncommon sometimes to have the same application updated and deployed several times a day!

glu has been designed to handle this use case: deploy applications to an arbitrary large set of nodes:
- efficiently
- with minimum/no human interaction
- securely
- in a reproducible manner
simply click “OK”!

glu is in the application monitoring space

Once an application is up and running, many failures can happen from the low level hardware crashing to the application itself misbehaving. This is the nature of it, nothing is perfect, things will crash one way or another. So it is very important to monitor the state of the system in general in order to be able to react appropriately. glu has been designed to give a full view of the set of nodes on which you have deployed applications. When something goes wrong, you can know in a near real-time basis. You can then decide on the proper course of action like restarting the failed application on the node it crashed, or downgrading it to the previous version and redeploying the previous version to all the appropriate nodes. As a side note, you can also build a full monitoring solution on top of glu as explained in this blog post.

glu detects failure

glu is a platform

It is very hard to build a one-size-fit-all solution. This is why glu has been designed to be a platform in order for you to build the solution that fits your environment and workflow on top of glu:
- glu offers a REST api so that you can control glu yourself via a simple and documented api. For example, glu is not auto-reactive by design (which means that if something fails, glu will tell you about the failure, but it will not act on its own about it), but using the REST api, it is fairly easy to build a reactive solution on top of it.
- the console (which is the main UI / REST api), is highly customizable (plugins to change the behavior of some features, css tweaks to let you change the look of it, etc…)
- glu comes with its own package/distribution builder which lets you customize every component to suit your own needs (as simple as changing the port numbers or as complex as providing your own specializations).
- glu does not dictate what a deployment mean to you, nor how to model your own system (clusters, groups, etc…), nor how to model your workflow (aka state machine in glu’s terminology). You are in control and you decide.
glu in practice

glu is not an academic exercise. glu has been built and successfully deployed at LinkedIn early 2010 prior to being released as open source. glu helps LinkedIn manage the complexity of releasing hundreds of applications/services on a 1000+ node environment (numbers from 2010). Other big names like Orbitz and Outbrain are using glu for their deployment needs.

Over the years, glu has been refined and enhanced in order to support more use cases but the core and fundamental concepts on which glu was based originally and that I briefly introduced in this post are still what makes glu what it is!

How to start with glu?

If you want to take a look at glu, the best is to head to the main documentation page. I would also recommend to download glu and try the tutorial which gives you a feel of what glu is capable.
April 2, 2014