devopsy.kinsta.cloud

Category: Features

  • DevOps: Caution Ahead

    DevOps: Caution Ahead

    Despite the continued adoption of enterprise DevOps practices, some organizations, especially those in highly-regulated industries remain cautious about moving forward too quickly.

    “There’s no doubt that DevOps brings benefits for some organizations,” says Martin Fisher, director of information security at Atlanta-based WellStar Health System. “However, many pushing for DevOps underestimate the amount of technological and cultural change that is necessary to overcome before moving ahead to boldly, especially among those in security and regulatory compliance roles that are concerned with securing and auditing processes they see as they’re potentially losing control over,” Fisher says. “You can’t audit and secure what you can’t control,” he says.

    Eric Cowperthwaite, former chief information security officer at Providence Health and Services agrees. “There are some who want to move too fast. In many organizations the culture just isn’t yet there, especially where they are used to very rigid quality assurance and audit controls,” says Cowperthwaite, currently a VP at Core Security.

    It’s certainly not uncommon to find chief security officers in heavily regulated industries who are concerned that the move to DevOps is an excuse to cut corners and move more quickly by cutting out necessary oversight. “You’ll find if you have a developer that’s producing a dozen mistakes a day, if you poke at them about it then you’ll find out that they’re tired of the structure, of the bureaucracy,” he says. “DevOps need to be done in a mindful and thoughtful way,” Fisher says.

    Cowperthwaite agrees, and contends that in certain environments DevOps will be – should be – taken slowly. “You have legacy systems and very highly audited and controlled systems where DevOps is just not going to flow for some time,” he says.

    It’s not just the cultural shift, but also the shift in necessary for good governance capabilities to keep up, they contend. “Not all systems are geared to handle 200 commits a day, nor do they have the ability for reasonable tracking and accountability for that kind of speed. In fact, there are many legacy systems were it’s not even possible at all,” Fisher says.

    However, not that all companies that move toward DevOps are attempting to automate everything – at least not at first. But it is higher levels of automation most certainly seek. A survey recently conducted by automated server management software provider JumpCloud found that many companies are turning to DevOps to automate time consuming activity, such as deployment, patching, user management, log file analysis, and forensics activities.

    Some of the trepidation is warranted, to be sure, and some of it may be due to the media and industry focus on the outlier DevOp implementers. “I think, justifiably, a lot of the fear comes from the fact that a lot of the companies that have gotten the most press regarding DevOps are doing things that are radically different than what is going on within the bulk of IT work at traditional enterprises such as banks, healthcare, and organizations like that,” says David Mortman, chief security architect and distinguished engineer at Dell Software Enstratius.

    When it comes to enterprises with trepidations when moving toward DevOps, all of the interviewees we spoke with agreed the key is to move slow. But still move and develop the culture and the toolsets slowly. “Pick one system that is separate and use it as a concept,” Fisher advises. “The DevOps ‘true believers’ are given a hall pass to fully run their systems and group in a DevOps model. As they learn and improve you slowly assimilate into the broader environment,” he says.

    Mortman sees it in a similar way, but notes that it’s not just about technology and absolute speed, but also collaboration, culture, and relative speed. “How fast is fast for your organization? If you’re only making changes to your big medical records database every 18 months, and you switch to every three months – that’s faster. That’s more continuous deployment than you were doing before. No one has to jump right in,” Mortman says.

    “I started working with those sort of DevOps-y concepts back before I was aware of the term “DevOps,” Mortman adds. “I was calling it “Agile InfoSec” and “Agile Ops.” And with that mindset, even the most regulated, paranoid enterprise can embrace DevOps. Start with small, modestly achievable goals and build from there. Collaborate and more tightly integrate the teams.

  • What Cisco’s OpFlex means for DevOps

    What Cisco’s OpFlex means for DevOps

    Some big news in networking land at Interop this year revolved around Cisco’s latest contribution to open source and the Internet, its proposed “OpFlex” control plane protocol.

    The protocol itself is focused on communicating with network elements and specifies encoding in a variety of formats, including JSON, that are developer-friendly. That’s nice, of course, but the big question as someone interested in operations and devOps is why should you care about what appears to be a networking protocol? After all, beauty is only interface deep. What’s in it for you?

    To see where DevOps fits into the picture, consider the description of OpFlex as specified in the IETF draft of the OpFlex protocol:

    The OpFlex architecture provides a distributed control system based on a declarative policy information model. The policies are defined at a logically centralized policy repository (PR) and enforced within a set of distributed policy elements (PE). The PR communicates with the subordinate PEs using the OpFlex Control protocol. This protocol allows for bidirectional communication of policy, events, statistics, and faults. This document defines the OpFlex Control Protocol.  [emphasis mine]

    What’s being offered is not just a protocol for bidirectional communication, but an entire architecture for sharing policies and actionable data. What’s more important to those who for whom “networking” is not their native technical tongue is that policies are designed to be declarative and abstract. That means instead of issuing a set of commands, often specific to the device or service, a policy declares (describes) what’s needed and each component participating in the architecture interprets those needs according to their specific function and capabilities.

    You might be thinking at this point (I know I certainly was) “What is this policy and who is defining it?” because that seems pretty central to the value proposition of this whole endeavor.

    For that, we need to look to the OpenDaylight (ODL) project which has created a new project called the ODL Group Policy plug-in. As explained by Cisco in an OpFlex white paper, “the goal of this project is to provide a policy-based API that can serve, in practice, as a standard information model in OpFlex implementations.”

    Jumping over to the ODL Group Policy Plug In wiki, then, the intention of the Group Policy Plugin is to introduce “the notion of groups of endpoints and policy abstractions that govern communication between these groups. The goal is to support a northbound API that accepts abstract policy based on application requirements while offering numerous concrete southbound implementations for programming network element with configuration/programming stanzas rendered from application policies.”

    Okay, so that’s a whole lot of .. words. What it boils down to is that instead of a policy describing a set of rules governing routing with the appropriate ACLs, one might simply declare that “Web Server A can talk to Database Server A to perform database transactions” while “Administrative Console B can talk to Database Server A about administrative tasks.” If you were to translate that into specifics you would likely have multiple imperative command lines, each describing VLAN participation and routing  as well as entries allowing traffic between specific VLANs or IP addresses on the ports appropriate to the application layer protocols. It would be long and prone to misconfiguration simply due to the number of commands required. Using a declarative (abstract) policy language simplifies not only the policy but also addresses the disparity inherent in competing products across a given market by eliminating it from being problematic.

    The problem with traditional proposals to solve the “different definitions and object models” across a given market space (like ADCs or IPSs or switches) is that they’ve all relied on finding the lowest common model, which inevitably ended up with all offerings needing to dumb down (commoditize) their value in order to ensure consistent support for all “XYZs”. By abstracting to the policy – and not the object model – level, the OpFlex approach effectively eliminates this issue and emphasizes defining what is desired by the application, but not how.

    Now, the actual policy declaration is unlikely to take the form of natural language (that’d be nice, but we’re not quite ready for that… yet, though Watson might disagree with me on that) but the general premise is to provide a far more, shall we say, friendly policy language.

    What this means is the operations folks most likely to be familiar with the application architecture and who understand what it needs and how it needs to work, could be the ones to actually codify that into policies. You’ll note that the intention for OpenDaylight, at least, is to “support a northbound API that accepts abstract policy based on application requirements.” That means someone – hint, hint, that’s you – might be able to integrate into the application deployment lifecycle provisioning of the network services and topological constraints required based on an understanding of the complex web of connections between each piece of an application. Someone who knows from experience that a “three tier architecture” means more than just three components, and can more effectively describe how they should be interacting.

    But that’s just the beginning. You’ll note that in the RFC summary above it specifies “bidirectional communication of policy, events, statistics, and faults.” That means actionable data is being exchanged between participating components. Actionable data enables, well, action. Actions that might address a fault that occurs between end-points. Actions that might address performance issues. Actions that might mitigate attacks in real-time.

    Ultimately this level of capability will drive new solutions – new workflows and processes – that take advantage of event-based systems and policy-driven networking. OpFlex should enable statements like “When X happens, do Y” to be executed upon, even if it requires coordination of three or four or more network components.

    What OpFlex means for the devops practitioner is a way to influence (or, one hopes, specify directly) what services the network provide for the applications you’re tasked with deploying and delivering, and how the network responds to the things that happen on any network that impact the performance, security or stability of those applications.

    As networking continues to recognize and address the need to provide application-driven services, including provisioning and real-time adaptation, it’s going to continue to become more programmable and integrated with the systems driving a more automated, orchestrated data center. And that means devops practitioners are going to have more and more options as to how they manage the entire application (and network) service lifecycle.

  • Defeating the irrational DevOps jobs displacement fear

    Defeating the irrational DevOps jobs displacement fear

    People are generally resistant to change, but when that change comes accompanied with the whiff of potential for job loss down the road you can count on gentle inertia to snowball in to out-and-out opposition.

    Unfortunately, as early misperceptions swirl around the DevOps philosophy, many in the ops world have gotten their backs up in fear that the automation that’s tied to DevOps will endanger their jobs.

    “I think it’s almost impossible to overstate the amount of angst that  DevOps creates in operations,” says Gene Kim, author of The Phoenix Project. “The big fear  in operations is that in order to get to lead times not in months, not to weeks, but to days, you can’t be  manually configuring servers, you can’t be manually reviewing things all of the time.  And so there’s a deeply held fear that these configuration management tools are going to automate everyone’s job away.”

    Now, to the degree that some within IT operations ranks rely solely on large vendors to set the course of their career track, maybe there’s some jeopardy there, Kim says, but competent and capable ops staff needn’t fear the automation reaper.

    “It’s a reasonable fear, but it is ridiculous at  the same time because we all have so much more work that we could get done if we had the time,” agrees David Mortman chief security architect and distinguished engineer for Dell Enstratius. “Doing things manually is negative value for most organizations.”

    And that negative value is having a direct impact on how IT services the business. In the mind of Shridhar Mittal, general manager of application delivery for CA Technologies, one of the big DevOps value propositions is that it helps IT from CIO on down to finally start to meet the pent up demand that the business has been clamoring for from the IT department.

    “I’ve yet to see a CIO tell me that they have more supply than demand. Why do you think the business is heading out to directly work with cloud vendors and doing their own thing? It’s not because the IT said no so that 10 people could sit on the sidelines,” he says. “It’s because these projects are added onto the list of 1,500 other things they got asked to do at once.”

    Mittal says that CIOs who are gung-ho for DevOps should be proactive about potential resistance from the trenches. This starts with open communication.

    “The first thing is start to educate staff that this isn’t about eliminating jobs it is about reducing that gap between demand and supply,” he says.

    And as IT leaders do garner more buy in, both they and operations staff will find that these workers actually become more indispensible rather than expendable through automation. DevOps allows workers that might have seemed lackluster to shine as rock stars because they’re more frequently doing higher value work.

    “Once you get rid of the garbage work or manual work, which is very time consuming and error-prone, you’re going to find that your engineering team is allowed to perform better,” says Nick Galbreath, vice president of engineering at IPONWEB.

  • Merging creation with operations: facilitated collaboration enables continuous delivery

    Merging creation with operations: facilitated collaboration enables continuous delivery

    So much has been written lately about the challenge of improving IT agility in the enterprise. The best sources of insight on why this challenge is so difficult are the CIOs, application owners, ecommerce and release engineering executives, and VPs of I&O, grappling to change their organizations right now.

    At a conference I attended recently, I met two Fortune 100 IT executives from the same company: one the head of development and the other operations. Their story is emblematic of just how hard this is in the real world. As interesting background, both the development and operations leaders were childhood best friends, participated in each others’ weddings, and spend time together socially on an almost weekly basis – but by their own admission, even they couldn’t get effective collaboration and communication to work between their two organizations.

    The lesson learned from this example is that the DevOps collaboration and communication challenge cannot be solved by sheer will, desire or executive fiat. Instead, you must breakdown the barriers that inhibit collaborative behavior and facilitate new ways of communicating and working together. The old standbys of email, instant messaging, SharePoint sites, and conference calls don’t cut it.

    The challenge of two opposing forces: Dev and Ops

    Imagine yourself helping your children put together a new jigsaw puzzle. Each time you turn your attention to a specific piece, the kids reorganize what you have already completed and they add new pieces, but in the wrong places. For sure, three pairs of hands can be better than one, but they can also create chaos, confusion and significantly elongate the completion of the puzzle.

    The collaboration challenge in the DevOps movement is grounded in this analogy. How do you get multiple people working together across teams, locations, and time zones to build and get things deployed faster without chaos, confusion, and delay? How do you get these teams to speak the same language and collaborate together with a singular purpose when their priorities and motivations are so different?

    Faced with this challenge, it’s easy to see why many organizations have stayed in their comfort zone of ‘waterfall’ releases and keep the number of releases per year small. The issue is this method isn’t meeting the demands of the business, the market and competition. As a result, more and more business leaders are going around their IT organizations. Options like public cloud, SaaS, open source tools, skunk works IT and outsourcing are making it easier for them to control IT decisions and implementations within the business unit or department itself. 

    So let’s dive deeper to understand the two forces at the heart of the issue: development (focused on the creation or modification of applications to support a business need) and operations (delivering defined services with stability and quality). It appears these forces are working in opposition, but both groups are focused on doing what leadership asks of them.

    Developers tend to think their job is done once the application is rapidly created, but it’s not because no actual value has been delivered to the business until the application is operational and in production. Operations is severely disciplined when services experience performance and availability issues and they have come to learn that uncontrolled change is the biggest cause of these issues.  As a result, operations teams often believe their number one job is to minimize change to better control impact on performance and availability. This causes operations to be a barrier to the rapid change required to give the business the speed and agility it needs.

    Critical to enabling DevOps is an explicit recognition of this situation and the ability to link discrete phases of the application development and operations lifecycle to enabling ‘fast, continuous flow’ – from defining requirements, to architecting the design, to building the functionality, to testing the application, to deploying the application to both pre-production and production environments and to managing all the underlying infrastructure change required for the application to operate efficiently and effectively in all environments.

    Why current approaches don’t work

    There are several challenges in achieving this ideal.

    1. Developers hate to document (can you blame them?), and, when they do, their communication is in a context they understand, not necessarily empathetic with the language that operations speaks. The view from operations is that the documentation they receive is incomplete, confusing, and/or misleading. With the rapid pace of development, this challenge is getting worse with documentation becoming more and more transient as developers “reconfigure the puzzle” on the fly.

    2. Today’s operations teams typically take responsibility for production environments and their stability. That means there is usually a group wedged in between the two – the quality assurance (QA) team. QA’s job is to validate the application works as expected and often they require multiple environments for each release. This group is typically juggling multiple releases and is, in essence, working on and reconfiguring multiple puzzles at the same time. The challenge of keeping QA environments in sync with both in-process releases and production can be maddening (just talk to any QA leader and they’ll tell you first-hand). The documentation coming from development is inadequate, and the documentation coming from production is often no better, since most operations teams store much of their most current information about configurations in personal files or simply in their brains.

    3. The ad hoc handoffs from development to operations and QA take time away from development’s primary mission: creating applications that advance the business. Some suggest developers should operate and support what they code in order to reduce handoffs and the risk of information distortion or loss. A fundamental risk with this approach is opportunity cost. Does a developer really understand the latest and greatest technology available for infrastructure and how to flex and scale those technologies to fit the organization need? Do you even want them to or would you rather they be coding instead?

    4. Others have suggested that Operations move upstream and own all environments from dev to QA to production, and treat configuration and deployment scripts as code just like a developer would. This may sound like a good option, but it can create a constraint on your operations team and cause valuable intelligence to become hidden in scripts. A particular application deployment could have one or more software packages required and potentially hundreds of different configuration settings. If all that information is embedded in a script, how will other team members know this if they go to make a change to the underlying infrastructure to apply a security patch, upgrade an OS version, or any of the other changes made in IT every day?

    Real DevOps transformation doesn’t mean that you give everyone new jobs, instead, it’s about creating an environment where teams can collaborate together with a common language and where information is immediately available at the point of execution and in a context unique to each team.

    A better way forward

    In The Phoenix Project, written by DevOps thought leaders Kevin Behr, Gene Kim and George Spafford, the authors promote the need for optimizing the entire end-to-end flow of the discrete processes required for application delivery – applying principles as they achieved agility in the discrete manufacturing process.  

    Manufacturing in the 1980s resembled IT operations today, employing rigid silos of people and automation for efficiency and low cost, but this became a huge barrier to the agility, stability and quality the market demanded. They learned if you optimize each workstation in a plant, you don’t optimize for the end-to-end process. They also learned that if quality and compliance processes were ancillary to the manufacturing process, it slowed things down, drove up costs and actually decreased quality and compliance. 

    Successful manufacturers brought a broader view and optimized end-to-end flow rather than operate in a particular silo. They also brought quality and compliance processes inline with the manufacturing process. By addressing quality and compliance early in the cycle and at the moment that an issue occurred, cycle times decreased significantly, costs plummeted and quality and compliance increased dramatically.

    These same principles can be applied to IT resulting in:

    • faster time to market;
    • greater ability to react to competitive pressures;
    • deployments with fewer errors
    • continuous compliance with policies; and
    • improved productivity.

    DevOps can best be realized when IT operates in a social, collaborative environment that ensures all groups are working with a visual model in their context with the necessary information from downstream and upstream teams, as well as in collaborating with relevant experts at the moment when clarifications are needed or issues arise.

    To merge creation with operation, the core idea behind DevOps, requires a cultural change and new methods in which cross-functional teams are in a state of continuous collaboration as they deliver their piece of the puzzle at the right time, in the right way, in-context with the other teams in other silos. Operating something that never existed before requires documentation so that operations teams have the information they need to manage change with stability and quality.

    With more modern collaboration methods, self-documenting capabilities are now possible as development, release and operations do their respective jobs, including visualization of documentation with analytics and with the perspective and context each team needs to effectively do their job downstream. These types of capabilities will transform organizational culture and break down barriers to collaboration that impede agility, stability and quality.

    Is this simply nirvana, unachievable in the real world?  No. Manufacturing achieved the same results by applying these principles; the fundamental point being made in The Phoenix Project.

    The goal is not to write code or to keep infrastructure up and running, or to release new applications or to maintain quality and compliance. Instead, the goal is for IT to take the discrete silos of people, tools, information and automation, and create a continuous delivery environment through facilitated collaboration and communication. This will drive the cultural and operational transformation necessary to enable IT to respond to business needs with agility while ensuring operational stability and quality.

  • Q&A: Speaking DevOps and Threat Modeling with the author of Threat Modeling: Designing for Security

    Q&A: Speaking DevOps and Threat Modeling with the author of Threat Modeling: Designing for Security

    If you want to understand how to threat model systems and applications in most any environment you turn to someone who has done so. That’s why we took 30 minutes to speak with Adam Shostack. Adam is responsible for security development lifecycle threat modeling at Microsoft and he is one of the very few threat modeling experts in the world. We thought, with more and more organizations moving toward DevOps and Continuous Deployment, would be a great time to discuss how this impacts the design of secure software. Earlier this year Adam published his book, Threat Modeling: Designing for Security, available here. The book provides pragmatic and actionable advice on how to make security an integral part of systems, software, or services from the outset.

    This advice applies to systems and development teams both big and small and is especially crucial as application deployment times accelerate.

    devopsy.kinsta.cloud:Why threat modeling? What benefits does it provide the business as well as the application development and operations teams?

    Adam Shostack: For the business, executives, they don’t want to deal with individual problems as they arise, they want to address problems as a category. They want to manage a class of things that can be completed in a consistent, repeatable, or predictable way. When it comes to managing application risks, threat modeling gives you an opportunity to answer questions as to whether you are dealing with certain vulnerabilities across all of the applications – rather than coming at them as one-offs. It’s a way to model and abstract away many of the details and look at things in a more strategic fashion.

    The other value threat modeling provides is the degree of predictability as you’re developing things – instead of forging ahead and building things and then running a pen test at the end only to learn then that you have serious security problems. What ends up happening in that scenario is that once you fix those problems, you then run another pen test and learn of even more or new security problems and you end up in that very unproductive cycle for a very long time.

    So threat modeling helps you think about your security before you start building. This way you can bake security into what you’re doing. My book helps to make both of those value propositions concrete and approachable. You can take these lessons in the book, give them to people and say go do these things and you really get an increased degree of control and predictability through what you’re designing through what you’re delivering.

    Also, when you threat model, you get a picture of where things are likely to go wrong. A lot of the threat modeling practice can be very process heavy. That’s why there’s a lot of very process heavy documentation out there. But actually threat modeling integrates well inside a test driven design approach to software development. Why? Because threat modeling tells you where you need to concentrate your security testing. If you’re looking for bug classes, don’t look at random sets of things or relatively random sets of things and you won’t get nearly as much test coverage.

    devopsy.kinsta.cloud: So, they can operationalize secure development into the processes, which is what security advocates have often spoke of as the optimal state of security’s relationship with application development and operations – rather than be a barrier at the end of the development process.

    Adam Shostack: The whole barrier meme comes from the idea that you do security activity late. A lot of people come into security through hacking. You hear the hacking metaphors all of the time. They are memes like “you can’t secure something if you don’t know how to break it.” Now, knowing how to break things is great. It’s a fun skill. It’s an important skill set. But you notice when you ask about making it operational is that one of the things that security people love to say is you’re never done threat modeling. Now, if you go to a VP of development or a VP of Ops or a VP of DevOps and say, “I want you to do this thing and it’s never done, but it’ll fit into our four week sprints….” You will get a big “huh?” The conversation won’t go anywhere from there.

    In order to work it has to be complete-able. It has to be focused and it has to be prescriptive with checklists. It has to have exit criteria, and it has to have metrics that answer: “Are we doing this well?”

    devopsy.kinsta.cloud: That type of thinking strikes me as fitting well with the continuous improvement objectives of DevOps.

    Adam Shostack: Absolutely. I’ve been threat modeling in various ways for a good 20 years. When I joined Microsoft I encountered people in development and in operations who were passionate about security, who knew that they had to do security and they wanted to do it right. But they also wanted to focus in on reliability, on accessibility, on internationalization, on performance. They wanted to focus on all of these other things each of which is an important property of the system you’re developing. I had to learn to make threat modeling easy. I had to learn to focus in on the highest value deliverables and make sure that those deliverables were accessible and capable. This has to be so even when you didn’t have a security expert groomed because people don’t understand that their decision has a security implication. They’ll make a decision without a security person handy. If you make the security goals accessible and understandable, they’re more likely to understand that what they’re doing has a security implication and they’re more likely to do it well.

    Let me give you an example of this. One of the areas that most often goes wrong in the design of a system is people don’t draw their trust boundaries. They don’t know where different principles are going to interact with one another. And application accounts with different authorization levels interacting are a natural place for security problems to arise.

    So what you can do is draw these trust boundaries on a white board. It doesn’t have to be a fancy diagram. Scribble on a white board and say, this is where things happen in admin. This is where things happen in the database. This is where things happen as the web user, and this is where web sessions are authenticated and you put boundaries between each of them. Developers understand that these things that they’re writing are run at different privilege levels.

    They then say, okay, this code runs on the web user context, this code runs on the database context. That’s great because once you’re doing that you can see the places that your threats are going to arise and look for places to exploit. There are lot of elements in threat modeling that you can work toward incorporating, but the fundamentals are actually easy to teach, easy to learn, and they’re easy to apply.

    devopsy.kinsta.cloud:We started out by talking, from a high level, what threat modeling means to different constituencies in an organization. More concretely, what is the payback of threat modeling for developers? Is it more resilient applications, fewer problems that have to be dealt with after the fact, and generally improved outcomes?

    Adam Shostack: Those are great values. The biggest “what’s in it for me” for a developer is that threat modeling takes all of these details about security that they want to deal with and provides a model, an abstracted way to frame them. So, instead of constantly worrying “Gosh, I might have another security issue” you can get a sense for how much you’ve done to solve the risk, and how much of it might remain. You get a sense of where you want to prioritize that work.

    I think a big misconception is that people believe that only an expert can do threat modeling. One of my aspirations for this book is to show that threat modeling should be like version control. Every developer, every ops person should know a little bit of threat modeling. And it doesn’t have to be a security expert. Perhaps it’s the person who manages your source tree. Or, if you work in a small organization threat modeling will be a side job for someone and that’s okay too. The important thing is to make it a part of your work.

  • DevOps killed the developer star

    DevOps killed the developer star

    The DevOps blogosphere is aflame over Jeff Knupp’s post on “How DevOps is killing the Developer”. While I agree with Knupp’s premise that DevOps, Agile and other technology trends are allowing (is forcing more accurate?) developers to do more outside of pure development, even turning into “full stack” developers, I disagree that this is a bad thing.

    I disagree with Knupp’s assertion that developers are the Orcas of the IT world sitting atop the food chain as the top dogs.  I think Knupp consciously or not, degraded the very real, important and hard job that ops people perform, DBAs do every day, as well as everyone else at an organization. Reading this I was reminded on one time I was sitting in an executive meeting and the VP of engineering actually had the stones to say that his developers were smarter than the rest of the people in the organization and worked harder to boot.  Now that is a real team builder right there.

    One thing I have learned in 30 years of business is that when an organization is achieving its goals it is because just about everyone is working as hard as they can. It is usually not one group, whether it be developers or marketing or operations that is pulling the sled alone.  It takes everyone to really succeed. There is no top of the totem pole or rather who is on top is a constantly changing position as bottlenecks come and go, priorities shift and the market changes.

    In any organization whether it be startup or enterprise, we all have to do what we need to do to accomplish the mission.  There you have it.  Working in the Federal market all of those years taught me something. In the DoD and armed forces it is all about the mission. We have missions as individuals. Those missions can be writing code, deploying or monitoring (heck even support and sales), but ultimately those missions all fold into the greater mission of the organization.  If developers doing full stack work helps the organization better accomplish its mission and goals, than it is good. No man is too big for the army.

    At its core that is what DevOps is too don’t forget. Empowering IT to better serve and meet the organizations goals. Not about letting developers do what they love to do or what floats their boat or what they consider the “top of the totem pole”.  Enterprises are here for one thing, the bottom line. It is something we should all never lose sight of.

    DevOps is changing the mission of developers. It is a change in the environment if you will. Species are faced with changing environments all of the time.  Those that adapt to change succeed and even flourish. Those that fail to adapt fall into the extinction bin.  Generally species that are generalists adapt and thrive much better than species that are too specialized. This is basic evolution theory.

    I think to Mr. Knupp’s point we are going to see a form of DevOps Darwinism play out. Developers who insist they only develop and don’t do full stack or other DevOps type of tasks will find shrinking habitats and less opportunity. They will exist on the margins. Developers who embrace this new environment will find abundant “target rich” environments with rewarding (both monetarily and stimulating) career opportunities.

    I am all for doing something you love for your work. Otherwise I would be practicing law somewhere. I stared working in the technology world because playing with computers was the most fun I was having. I am sure many of you had the same experience. Messing around with code or scripting moved from fun and something you did because you liked it to something you also do to earn a living.

    Earning your livelihood doing something you love is a great blessing by the way. There are untold numbers of people who trudge off to work every day and have to do something they dislike, are not passionate about and do just for a paycheck. I can’t imagine having to live life like that.

    I think that is what Knupp is talking about too. That developers code because they love to, they need to, and it is what they do. Forcing them to do other tasks takes away from their focus and stunts their growth, which in turns comes back to hurt the organization as a whole.

    I don’t think developers will ever stop coding. Mr. Knupp is right, developers code because they want to. That is what they love to do and it is why they became developers.  I think however they will have even more opportunity to code, not less as software continues to gobble up the world. It may be a different kind of development, with coding as infrastructure as software dominates, but developers are still going to code no matter what.

    We may be seeing the rise of the coding generalist versus the pure application developer as the lines between developer, ops, QA and yes even security continue to blur and overlap. But I don’t think it is a bad thing for developers, enterprises or anyone else. It is just part of the constant change in our environment.

    I don’t think this change is bad at all. In fact to paraphrase the great Gordon Gekko, “Change is good, change is what made America great.”

  • Microsoft bridges the gap between Azure and DevOps

    Microsoft bridges the gap between Azure and DevOps

    DevOps can be easy for a startup. Many of the concepts and principles of DevOps come quite naturally to a fresh company just getting started. It’s a different story, however, for large, established enterprises trying to wrap their arms around this DevOps thing. For IT admins working in Microsoft’s Azure cloud platform, though, things just got a lot easier thanks to the integration of Chef and Puppet Labs.

    Many organizations are heavily invested in the Microsoft ecosystem. IT teams and administrators are familiar with Windows domains, Active Directory, Hyper-V, and Azure. They know how to write scripts and leverage PowerShell to get things done, but they’re much less familiar with the core principles of DevOps, or common DevOps automation tools like Chef and Puppet Labs.

    One of the driving forces of DevOps is the ability—or perhaps necessity—to automate those tasks that can be automated so IT resources are freed up to focus on more important tasks. Deploying and configuring virtual servers and applications is necessary, but it can also be tedious, and it is a task that can be easily automated using DevOps tools.

    The arrangement also works the other way. Smaller companies that are already familiar with, and invested in DevOps may need to scale to a platform like Azure. They should be able to build on what they’ve started, and continue to use the tools they’re familiar with as the company migrates to Microsoft’s Azure.

    One of the main attractions of Microsoft Azure is the degree of flexibility, choice and control it gives customers. Microsoft Azure’s partnership with Puppet and Chef allows customers to choose from an array of existing technologies and communities to easily deploy, configure, automate and manage their VMs.

    According to a Microsoft spokesperson Puppet Labs and Chef complement and extend Microsoft technologies such as Resource Groups and PowerShell Desired State Configuration (DSC) to offer more options for customers to migrate existing or deploy new applications to the cloud. The partnership offers Microsoft the ability to support Linux and UNIX environments, and also alleviates the concerns that organizations already using Puppet or Chef will have about abandoning their investment. Now, they can continue to use their Puppet modules or Chef recipes in Azure.

    Luke Kanies, CEO of Puppet Labs believes the integration of Puppet Labs and Azure will help Microsoft IT admins by making it easier to deploy applications across any platform or cloud environment. Kanies explained in a press release, “Our goal is to help you automate the configuration and ongoing management of your machines and the software running on them, so you spend less of your time firefighting and running simple scripts across a thousand machines, and more time getting great software in front of your users.”

    I spoke with Kanies about the difference in dynamics between a small company and a large enterprise. He told me that in a small company it’s much easier to communicate and coordinate resources, and the cost of failure of any particular project is much lower. For a large, established enterprise, however, there is assumedly a large base of existing customers, and there is more fear and worry about possibly disrupting normal operations.

    I also had a chat with Ken Cheney, VP of business development for Chef. His philosophy mirrors a lot of what Kanies described. Cheney told me that emerging companies have the luxury of starting from ground zero with a clean slate. There aren’t yet any established processes to navigate. The challenge is trying to transform an organization with established processes, and a team of existing developers over to a DevOps mentality. Integrating DevOps into Azure helps organizations to meet that challenge.

    DevOps adds an element of speed and simplicity that makes it easier for larger enterprises to be more agile. Using tools like Puppet Labs and Chef enables IT admins to add speed as a core requirement for infrastructure projects without disrupting or conflicting with other requirements—like security and stability.

    Ultimately, the new alliance illustrates Microsoft’s commitment to empower Azure customers with the best-in-class configuration and automation tools by embracing the technologies, communities and ecosystems that developers know and love. Combining Azure with powerful and popular platforms like Chef and Puppet Labs bridges the gap between traditional IT and DevOps—enabling old school IT admins to embrace DevOps, and DevOps startups to mature and evolve into larger enterprises.

  • Expanding DevOps requires a supportive IT community

    Expanding DevOps requires a supportive IT community

    How do we bring newbies into the DevOps community? Until they know the melody, they can’t appreciate the song.

    echo-chamberWhile leading a DevOps event, the speaker polled the audience about their knowledge of DevOps.  All of the attendees were well acquainted and actively engaged.  It was obvious there were no newbies in the room.  It made me wonder if we were yelling into an echo chamber filled with people who were already on board.  Where were the newbies?

    My concerns that day reminded me of a 1990 Stanford University study led by graduate student Elizabeth Newton.  She assigned groups of two people to play a communications game. The first person picked a simple song that everyone would know (like Happy Birthday), and tapped the rhythm of the song on the table. The second person was the listener and tried to guess the song.  On average, the tappers anticipated a 50 percent success rate. In reality, the listeners only guessed the song correctly 3 out of 120 attempts (2.5%). [Source: Health & Health, 2007 pp. 19-20]

    That happened because the tappers already had the multidimensional melody playing in their heads.  Conversely the listener was unaware of the melody in the tappers’ head and was limited to the one-dimensional tapping patterns.  When tapping was the only means of communicating the song, the listeners almost always failed to guess the song title.

    What’s the Point?

    As DevOps enthusiasts we have experience and expertise — we know the melody.  We have pre-existing knowledge and confidence that this new model works.  Many of us have years of experience, experimentation and a network of insightful supporters to advance our learning. We know the DevOps founding stories and have read a myriad of blogs, articles and books.  We have listened to the webinars and podcast cafes.

    ideasAs the DevOps message tappers, we hear the melody of innovation and progress. And, in our minds, we assume we are communicating our DevOps stories with clarity. We point to effective projects, tools and teams, and create new models and terminology, expecting everyone in the IT community, or those still working in more traditional silo-ed organizations, to comprehend our activities. The listeners, however, may not have the same experience, confidence or base of knowledge. This is especially true of IT teams in enterprise organizations. Many of them have never read the books or heard the compelling stories. They have achieved success and grown their careers with Waterfall, in silos, with hand-off challenges and with technologies from past generations. Maybe they have never even played with Open Source technologies, worked with an open community, or explored the benefits of Agile methodologies. Thus, when many DevOps enthusiasts speak, their message is not understood by traditional IT professionals or those still working in silo-ed organizations.

    Break through the IT barriers and silo-ed structures!

    Breaking through to traditional IT professionals and silo-ed enterprise contributors begins with making the assumption that some may not have sufficient knowledge or experience in this area. This does not mean enthusiasm in DevOps presentations and conversations have to be toned down. Rather, it means they need to begin at the shallow end with stories and then move listeners to the deep end with more technical and detailed information (e.g.,  new ways to systematically reintroduce lean, agile, continuous integration, continuous delivery, continuous improvement models, etc.). The best way to accomplish this goal is to think specifically about traditional IT professionals and their specialty niches. When making presentations or engaging with them, consider their view, context, roles, work silos, and lack of opportunity to see their contributions make a difference. Then endeavor to inspire them to increase their knowledge base and expand their network by exploring collaboration opportunities internally with co-workers and externally with other groups.

    PolarizedSome presenters and team leaders find it effective to begin their group sessions and new conversations with stories from their initial exploration or foray into DevOps. Others employ familiar IT language, ideals, expectations and activities before transitioning into a DevOps melody. This will serve to expand the community to those who are well entrenched in older models (e.g., traditional Waterfall-based organizations, etc.).

    Through greater understanding and deliberate behavior, much can be accomplished when legacy and DevOps groups collaborate: common goals are often more rapidly reached; polarization between old and new models avoided; and DevOps practices expanded across a wider IT community. Externally, this would ensure a wider community of support for DevOps and an ongoing flow of cross discipline collaboration and insights: and internally this can lead to faster innovations, improved corporate unity and accelerated success in the marketplace.  Leaving new invitees outside of the DevOps conversation (intentionally or unintentionally) could hurt our DevOps efforts more than we might ever know. Without building in correlation or transition concepts between old and new, in everything we do,  speakers and leaders could spend endless amounts of time and effort tapping “Happy Birthday,” but very few “newbies” would ever have a chance to recognize, support or see the value in our DevOps melody.

  • Camp DevOps @ GlueCon 2014

    Camp DevOps @ GlueCon 2014

    campdetailsdevopsy.kinsta.cloud is very pleased to announce that in conjunction with Eric Norlin and Glue Conference 2014,  we will be presenting our inaugural Camp DevOps @ Gluecon on May 20th, one day before Gluecon 2014. Camp DevOps will take place from 8:30 am until 4:30 pm. The event will be held at the Atlas Institute Building at the scenic University of Colorado/Boulder campus. Here is a link to a flyer with more information here.  You can go also get more information by going to Campdevopsy.kinsta.cloud (site is still going up, so be patient if there is a problem) or our Eventbrite page.

    Moving devopsy.kinsta.cloud beyond the 4 walls of the website has always been part of our business plan. Camp DevOps is our own conference brand. Martin Logan, devopsy.kinsta.cloud co-founder has held Camp DevOps before, but this will be the first event since our commercial launch.  Over time what Camp DevOps is and what to expect we hope will be better understood. But since this is the first, let me quickly give some background.

    We recognize there are no shortages of conferences around DevOps. Camp DevOps is not meant to compete with DevOps Days and similar conferences. We think DevOps Days are great events and we will be working with local Meetup groups to help them with DevOps Days in the future.  Camp DevOps are also not meant to compete with Velocity or other enterprise events.  Instead Camp DevOps like devopsy.kinsta.cloud itself is meant to pitch a big tent. We will have content and activities for the different constituents of the growing DevOps community.

    CampDevOps @ Gluecon 2014 will feature keynote presentations (Sanjeev Sharma of IBM and author of “DevOps for Dummies”, Rajat Bhargava, CEO JumpCloud and Howard Diamond, CEO of MobileDay already confirmed), multi-track presentations with one geared for technical audiences and another aimed at business and cultural topics, panel discussions and what we are calling “Hello World”. Hello World will feature hands on learning with DevOps tools. Attendees will need their own computing device to follow along.

    There will also be table top displays, Continental breakfast and lunch served. We are hoping to have a sponsor for dinner as well. The cost to attend Camp DevOps is just $49.99. If you are already registered for Gluecon, you can attend for free. If you are not registered for Gluecon and register for Camp DevOps you will receive a $100 dollar credit towards Gluecon registration. There will also be complimentary shuttle service from the Gluecon hotel, The Omni to the Atlas building at CU/Boulder.

    Attendance is limited so please don’t wait to register!

    Besides the keynote speakers, we also will have presentations from AppDynamics, MongoDB, VictorOps and others. We will be updating the agenda and presenters daily.  We still have room for more track speakers, keynotes, panel and Hello World presenters. If you are interested contact us at CampDevOps@DevOps.com.

    Sponsorship packages are also still available. There are sponsorship opportunities in conjunction with Gluecon and CampDevOps or just Camp DevOps. Contact us at CampDevOps@DevOps.com for details.

  • The Importance of the Bus Driver

    The Importance of the Bus Driver

    The topic of DevOps has been a topic for many a debate over recent months years. Without adding fuel to the fire I’d like to present a slightly different take on the DevOps culture.

    The phrase ‘what if I was hit by a bus?’ can often be heard within the vicinity of software development teams. It relates to a measurement known as the ‘bus factor‘. Individual talent should not be discouraged, however when it becomes silo’d that talent translates to risk for a business. Try to think back over your career – do the following sound familiar?

    “I don’t know how to do that, X usually does that”

    “X is away at the moment so we’ll need to put that on hold”

    “X left yesterday and didn’t cover this in their handover”

    If you have read ‘Phoenix Project‘ – think of Brent…

    Using the fated bus driver I’d like to take us on a journey (excuse the pun) through a DevOps culture.

    In-sourcing the bus driver

    You work for a small green minded startup. Being green minded and wanting to help your staff, you decide to buy a bus to get everyone to work in the morning. Realising that you don’t actually have a dedicated bus driver, you decide to take it in turns to drive the bus in the morning. Everyone gets involved, everyone knows how to drive the bus, everyone knows which route is best.

    Idea 1

    Don’t over engineer your processes. At this stage you’re already exhibiting DevOps characteristics!

    The talented bus driver

    Your small team has performed really well, people like what you do and now your team isn’t quite so small. There are now over 50 people on your bus in the morning. It wasn’t long until you realised that Sarah is the best bus driver you have. Inevitably, Sarah usually just takes up driving the bus each and every morning.

    If Sarah is off, Mike takes over but he often makes people late for work and tends to hit a lot of parked cars.

    Furthermore the bus recently broke down when Sarah was off, Mike and your bus passengers all took a look at the engine and tried to help but there seemed to be a ‘magic’ piece of string holding two parts together.

    When they consulted the bus manufacturer manual, this piece of string was no where to be seen. Mike had to call Sarah whilst on holiday!

    Mike calls the bus company to ask for a replacement bus, to be told it will be 2 hours until a second bus arrives. The team sit and wait patiently.

    Idea 2

    Try to identify your tipping point. Some key indicators are reliance on individuals, information silo’s, phrases such as ‘oh I didn’t know it functioned like that in production’, production environments start to diverge (without communication) from development environments.

    How many production environments have the web server, app server, database, load balancer et al all running on the same machine? Is this how your development teams work? Does this differ to the view of the operations team?

    Your bus stops

    We’ve identified a few problems but lets focus on the positives. Firstly, let’s celebrate actually spotting the problem, it is very easy to continue with the status quo. Our second success was the reaction of our bus passengers. When the bus broke down, instead of complaining, the bus passengers tried to help mend the bus.

    Idea 3

    When and where your bus stops is less important than who it stops for. It might be stating the obvious, however if you wish to move towards a DevOps culture you need talented team members. You need people that care for the organisation and their profession. A desire to share knowledge for a greater goal.

    If your team are not interested or do not care, then no matter how much you say DevOps you need to boost their interest first.

    Hopefully you’ll start to hear your teams utter phrases such as “What does production look like?”, “You could do this slightly differently you know”…

    What if your talented bus driver has an RTA?

    There are no doubting Sarah’s talents – I mean come on, she was able to fix a bus engine with a piece of string, knew the best bus routes and always got your team to work on time…

    What would we do if Sarah left or was, somewhat ironically, hit by a bus?

    Idea 4

    Could your organisation raise the value of sharing and encourage knowledge share?

    Assuming you’ve got the right people on the bus, you might also find that other staff members have ideas for improvement when you explain the current process. Your Sarah(s) should not fear sharing their knowledge and should be rewarded/encouraged for doing so.

    One of your passengers may even know a faster journey to work or a better way to fix the bus engine.

    The bus seating plan

    Although you didn’t enforce it, you start to find groups forming on your bus journey, all your teenagers are sat together at the back playing music on their phones…..

    Idea 5

    Don’t let job titles restrict conversations. Water cooler chats are incredibly productive and help teams share ideas.

    Replacement bus service

    When the bus broke down Mike was able to order a replacement but it had a 2 hour turnaround time.

    Would it be better if we always had 2 or 3 buses running? If each bus had enough capacity for all 50 passengers we’d probably always make sure people were on time for work.

    It would be even better if all bus drivers knew the routes that Sarah used, all buses had the same piece of magic string and all bus drivers knew what the magic string did.

    Idea 6

    Something my team often hear me say is ‘Design for failure, don’t just react to it‘. When designing your applications and services try to think how the system can accommodate failing services better rather than what actions you might take in the event of failure.

    The same might be said for your human resources. Have your teams got a shared knowledge? What would your bus factor score be?

    Journey times

    Try to keep your passengers informed of your progress – you could let them know their ETA for work. Informed passengers tend to be more engaged and understanding. For example when telephoning a business, do you prefer to know what position you are in ‘the queue’ or would you rather sit on the line waiting without knowledge of when you may be spoken to?

    Idea 7

    Monitoring might be considered a key technical section of DevOps adoption. There are many, many tools that can help with this such as StatsD, Graphite, Librato, New relic, AppDynamics etc but whichever medium you choose, try to make sure your ops stats are in full view of your various teams not simply your ops staff. Get them in front of your dev team, support team, sales team, HR team they impact every member of staff.

    Hopefully, you will be surprised at an increased level of employee engagement. When ops are in war rooms fire fighting problems, each team member will understand the pressures they are under, some team members will also express a desire to help.

    You have arrived at your destination

    At this stage I may have flogged the bus concept a little too far. Are the concepts here a guide for the adoption of a DevOps culture? Does the ‘D’ word have to be attached to the guidelines above? I’ll let the community decide.