devopsy.kinsta.cloud

Tag: devops

  • Zen and The Art Of Infrastructure Maintenance

    Zen and The Art Of Infrastructure Maintenance

    There’s a famous passage I’ve quoted from time to time when I see someone “stuck” with a problem, flailing potential fixes and solutions wildly in hope that one of them does the trick.

    “Assembly of Japanese bicycle require great peace of mind.”

    Just like that. Improper grammar and all, quoted verbatim from Robert Pirsig’s philosophical classic Zen and the Art of Motorcycle Maintenance. The quote is in reference to the first printed words on a set of technical instructions, which the narrator discusses with friends and colleagues on a long-distance motorcycle trip.

    And the response I get is the same every time, likely the same response you’re having right now…

    What in the world are you talking about?

    Well good, now I have your attention. And I’d be willing to bet that attention is on 1) the ludicrous loss of translation from Japanese to English and 2) debating whether or not something as seemingly superficial as “peace of mind” is really required to assemble a bicycle.

    Language barrier aside, let’s abstract away from modes of bipedal transport into a more generic “machine”. Do we really require “peace of mind” to assemble or create a machine?

    cory2

    Reliable? Maintainable? Usable?

    We certainly all want to receive peace of mind from our machines. We want them to do the job they were created to do without fail so we can move our attention onto other things.

    What words do we use to describe such machines that give us this peace of mind? Reliable? Maintainable? Usable? When we think of the reliability, maintainability or usability of a machine, we’re really just describing a machine that can produce peace of mind. I’d argue if we want that machine to produce peace of mind, then it must be created under peace of mind.

    Why is that? If we, the creators of machines, are in a state of peace of mind when we start our endeavor, we’re highly motivated to do that which will continue our state of peace of mind. We don’t want to lose this state. Since if the machine does break, we the creators will be forced to lose our peace of mind and fix it, we’re motivated to create a machine that’s resilient against failure and resilient against our loss of peace of mind.

    Without this key precondition to creation, we’re likely to build our existing personal problems right into the machine itself! Under time pressure? We’ll cut corners to make deadlines. Dealing with a different machine breaking? Likely to gloss over a key failure condition in the machine we’re actively building so we can fight a fire elsewhere. Worried about the problems of today? Then I can guarantee there’s no thought going into a machine that will be resilient to the problems of tomorrow.

    Quality creations produce serene creators as much as serene creators produce quality creations.

    Those in a state of serenity will continue to create that which maintains the state of serenity or peace of mind; those that are not in a state of serenity are unable to escape a vicious cycle of build-fix-build-fix, never achieving peace of mind. Ultimately, the serenity of the creator is the key test for the quality of creation.

    And ultimately, this passion for peace of mind from our machines is what you’ll find when you meet the folks in the DevOps community. We’re guided by a desire to maintain our peace of mind — for ourselves, for our families and for the folks that count on us the most: our customers.

    If the machine produces tranquility, it’s right. If it disturbs you, it’s wrong until either the machine or your mind is changed.

    The material object of observation, the bicycle or [in this case, the global infrastructure], can’t be right or wrong. Molecules are molecules. They don’t have any ethical codes to follow except those people give them. The test of the machine is the satisfaction it gives you. There isn’t any other test. If the machine produces tranquility it’s right. If it disturbs you it’s wrong until either the machine or your mind is changed. The test of the machine is always your own mind. There isn’t any other test.
    – Robert Pirsig

    So the next time you find yourself struggling with a problem, take a lesson from Robert Pirsig: step back, achieve peace of mind, combat that which threatens peace of mind and strive to maintain peace of mind.

    You’ll be a better (and happier) creator for it.

  • DEVOPS AND CONTINUIOUS DELIVERY; THE NEED FOR SPEED

    DEVOPS AND CONTINUIOUS DELIVERY; THE NEED FOR SPEED

    Continuous-Delivery-Infographic-ImageSome good stuff around the DevOps scene this week on DevOps and Continuous Delivery, which the pundits are saying will be a major spend area in 2014.

    First up is the Devopsguys who with a little commentary reposted this infographic from Zend Technologies. It demonstrates the real value in DevOps and continuous delivery. One thing that jumped out at me was that according to the data, ninety- nine percent of the Companies surveyed are now recognizing a greater need for DevOps strategies than before. Zend built the graphic utilizing data from various credible industry sources.

    Of course you can’t talk about Continuous Delivery without at least mentioning Jez Humble, after all he literally wrote the book on it. He recently posted on his blog four visualizations by Nhan Ngo, a QA Engineer at Spotify. The four visualizations were created while reading Jez’ book, Continuous Delivery. Some pretty intense stuff. Nhan Ngo made it look almost simple, almost like like a comic strip, but there is nothing funny about it.

    If you really want to get into the trenches, here is an interesting video that I found called Continuous Delivery and Zero Downtime from the site Night Hacking. The scenario is your “Continuous Integration system is well oiled and kicks in after every commit. Your code is compiled. Your tests are green. You feel ready to take your project to the next level and move to Continuous Delivery.”

    We aren’t done with our look at DevOps and Continuous Delivery just yet, DevOps Reactions probably illustrates best how Continuous Delivery works.  It works in the model, but in the real world it requires real skill.

    If you would like something mentioned here or have something you think I should Shout Out; email parker@devops.com and don’t forget to comment here.

     

  • DevOps: An Opportunity to Build a Better Resiliency

    DevOps: An Opportunity to Build a Better Resiliency

    Security professionals like to check code for security-related defects before the code is delivered. When they start hearing about multiple builds a day, or even a few builds a week, and a radical decrease in development time along with operational changes, they are bound to give pause. After all, most are having a hard time keeping up even now.

    The industry is moving in this direction. An Evans Research survey of more than 600 software development professionals in the UK and US found that 51% of respondents have already started the practice of continuous delivery across a number of teams. The same survey found that both culture and technology are the primary stumbling blocks, with 53% believing that they have the technology in place for continuous delivery and 59% believing that their staff is ready.

    However, those that do successfully make the move to continuous delivery and embrace DevOps find that the real impact on security will be much brighter than the naysayers predict, according to those  interviewed. In fact, it is automation that underpins continuous delivery and much of the DevOps movement, when done properly, goes a long way to enhance security and system resiliency.

    Security often gets blamed for slowing down the development process, explains Andrew Storms, senior director of DevOps at CloudPassage. But the reality is that it slows it down no more than QA and other development checks. “Those who are at the end of the chain are typically the ones who get the fingers pointed at them. They are the ones who are blamed for slowing down time to market, or delays in getting product in front of the customers,” says Storms “DevOps is an opportunity to integrate all of that together.”

    “One of the interesting things about moving to DevOps is that it requires a very high level of automation,” says David Mortman, chief security architect at Dell Enstratius and a contributing analyst at security research firm Securosis. “Part of the whole continuous deployment model is the move to continuous building and continuous integration. So you need to start writing security-oriented tests that can be automated against that code; when you are running your regression, integration, and other tests, you are testing for security states as well,” Mortman says.

    Jeff Sussna, founder of the IT service innovation consultancy Ingineering.IT, agrees “In continuous delivery and DevOps you start thinking, from a nonfunctional requirements perspective, about continuous integration—and that’s about finding software problems sooner, when they’re easier to fix. That’s what security professionals have been fighting for, for years,” Sussna says.

    That certainly makes DevOps an easier sell to information security teams. “When you suggest to security teams that security scans be completed with every build so that problems are found sooner, my experience is that security folks, who don’t seem to smile a lot, suddenly start to smile,” Sussna says..

    It’s not just talk, “it works,” says Sussna. “When we run those kinds of scans, we find problems earlier, and it’s easier to fix them. When it comes to running through the traditional gates that security puts up, it’s a lot less painful because they tend to run clean within the processes.”

    Another, and perhaps much more subtle, yet deep, benefit, is how continuous security testing keeps security at the forefront of developers’ minds. “Even when a build comes back clean, and the notification says, ‘Security violations, zero,’ it puts in their minds—daily—that security is something they need to think about it. And it’s not confrontational; DevOps puts it within the development process because the developers are seeing the feedback from their reporting tools, and so they are getting instant feedback that they need to fix security issues. Therefore, the feedback isn’t a nag from QA or information security; it’s coming from their systems,” .

    The benefits of DevOps can run still deeper. “Forget the deployment. Forget the automation. Just having this culture of sharing and working together that DevOps fosters, as opposed to being combative, helps to drive excellence,” says Mortman.

  • DevOps and Security Are Compatible

    DevOps and Security Are Compatible

    When I speak with information security organizations faced with the prospect of moving to DevOps, one of the most common fears I hear is that this transition will degrade security of infrastructure and applications.  If you’re one of these folks, I understand this fear but you can rest assured:  when you do things correctly security will actually improve.

    One big reason security benefits in this model is due to improved alignment and tighter feedback loops.  You see, DevOps is about creating a unified, engaged team and doesn’t make it easy to fall into the “silo thinking” that traditionally leads to security as an afterthought.

    DevOps embraces automation and consistency, which benefits security by allowing you to add automated checks during coding to look for obvious security issues and flag things for human review (such as the linking of new libraries or the introduction of new third-party components that could add risk).  This means you will be able to identify “areas of concern” earlier in the development process where they will be much cheaper and more timely to address.  Twitter is a well-known advocate of this kind of automation, in which security checks are executed each time code is checked in and obvious security issues are presented to the developer immediately, giving them a chance to correct the problem right away and reducing the likelihood they will repeat the errors.  That is a much more effective process than the delayed feedback typically associated with manual code reviews.

    Security isn’t just better in the “Dev” portion of DevOps – you can also add automated configuration validation and scope checking into the environment build process to minimize the chance that a fat-fingered error or an out-of-policy change will introduce problems into production.

    This sort of automated approach also improves as you learn from retrospectives and post-incident reviews.  As you encounter preventable issues, you can often improve your automated checks to increase the odds of catching these issues before they impact production.

    Remember that security is an important stakeholder in the DevOps process, and the onus is on you to become a part of the conversation and find ways to add value to the goal of the organization.  Also, don’t forget to tap into the community of DevOps practitioners – others have walked this path before you and can help you in your journey.

    Finally, if there are specific security challenges related to DevOps that are vexing you (myths you would like to see busted) I’d love to hear from you.  Leave a comment below and become part of the conversation.

  • Security Policies For DevOps 101

    Security Policies For DevOps 101

    A while back I was talking with some friends about how to ensure security inside a devops environment.  The simple first thought was that no where does DevOps say to do away with security.  Instead, consider how your existing information security practices and policies need to adjust given the changes that DevOps brings to the working environment.  Veteran security professionals understand that many security practices begin with policies.  Furthermore, no policy is ever set in stone. Failing to update policies and procedures at least every year is already a recipe for disaster.

    I tell a lot of people to think back when the iPhone was first introduced.  The natural tendency of security professionals inside IT organizations was to either 1) simply ignore the device or 2) establish a flat out denial kind of policy.  Neither choice was even near the right answer.  I can say first hand that I was victim to this kind of tunnel decision making.  We updated our mobile device policies with a flat out “no iPhone for you…come back in 2 years” doctrine.  My feeble attempts to fend off iOS devices lasted maybe a year.  Meanwhile, lots of people at the office were seen with iPhones in hand who continuously said things like “but I’m not using it for work”.  Flat out denial on both mine and their parts.

    Bottom line, policies are meant to be changed for the changing environments.  DevOps is no different.

    The Minimum Requirements

    Just like in development how we talk about the minimum viable product, we can apply the same concepts to implementing security practices.  For starters, make sure your organization has a few policies drawn up, agreed upon and stamped with the approval of your executive staff.  The minimum set of policies any organization needs should include: network or Internet security, computer security, information and intellectual property security policies; secure software development policies and procedures.

     Be the Champion

    But the job doesn’t stop at policy creation.  Even having your executives agree to the policies doesn’t mean they will be followed or even understood by your DevOps teams.

    I speak to security heads often and tell them they need to get out of their seats and speak to all the employees in a company.  One of the first conversations to have is to begin championing the security policies.  Make sure people understand the policies and more importantly understand the reasons for the policies which have been selected.

    Do you remember going to high school and being told you have to follow a rule that you thought was totally asinine?  Non security folks will probably think the same about silly policies and might even act out purposely against them. A champion needs to teach and live those policies.  And when someone challenges the policies, then take the time to have a two-way conversation with them.  Make sure you understand their point of view first before you try and push your silly security ideals on them.

    Taylor Your Policies

    While the policy template available online, like those available from SANS, are a great starting place, don’t just slap those on the intranet as-is.  Read them and make adjustments for your company, for your culture and your development practices.  Even better, include policies that are specific to cloud and DevOps.

    One area of customization to start with is to create policies for specific cloud vendors.  For example, consider setting policies around how to the company should be using the Amazon’s IAM feature.  Make specific policies around how IAM users and groups should be configured and used.  Use the doctrine of least privilege and design a sane access control policy.  Consider a policy where only a few trusted users have full admin access and other users are in a power users group where they can launch instances, but cannot alter AWS console configuration or access.  You might want to give your accounting team limited access to the console as well to review billing and usage.  While you are at it, develop specific policies around the use, distribution and storage of cloud API keys.  Furthermore, consider enforcing two factor authentication.  These kinds of specific use case policies will go a long way for security teams in gaining confidence with the DevOps teams.

    DevOps might be the cool new hotness that is making waves in every kind of organization, but that doesn’t mean we can forget our information security requirements in a DevOps environment.  One of the starting points in ensuring DevOps organizations follow good security practices is to start with a minimum set of security policies.  Make sure you champion these policies and tailor them to your specific organization and use case.  With a little luck and determination even your DevOps teams can follow security best practices.

  • Step One Of DevOps Ignores Technology

    Step One Of DevOps Ignores Technology

    I’ve spent a bunch of time talking with Adrian Cockcroft, former cloud tzar at Neflix, and listening to his various talks and presentations. I’ve also spent a bunch of time looking around the various rooms within which he presents and seen the dazed look of attendees who have rightly assumed that just seen a picture of the way organizations should be. Netflix is unquestionably an exemplar for IT reinvented – where innovation, agility and developer enablement are natural parts of what they do.

    But then I’ve seen these new disciples of the “Netflix way” write a list of the technologies that Cockcroft detailed and attempt to shoehorn those same approaches into their own organization. All to often the result is disappointment, failed projects and a swing back to the traditional rigid ways of doing things.

    These thoughts struck me then I hear about this new masthead and its aim of creating a hub for DevLops thought leaders – surely giving IT practitioners a good place to find the latest tools and techniques will help spread the DevOps gospel?

    Well yes, and no. You see I’m a firm believer in the viewpoint that DevOps has very little to do with products, processes and policies, and instead is a direct result of a particular type of culture. On the one hand you have Netflix, clearly a cultural outlier with its openness, willingness to experiment, embracing of failure-driven iteration and aim of planning for failure. On the other hand you have traditional IT shops that are top-down, driven by a rigid policy framework and consider the “operations department” and the “developer team” as two distinct entities.

    This was a topic that Christian Reilly (he of the sartorial elegance and funny accent tendencies) and I discussed a couple of years ago, before the DevOps hype had really started to hit. In the video, he reflected directly on his organization, Bechtel, as an outlier at one end of the spectrum, while Netflix is an outlier at the other. It’s a somewhat dated video, and talks about public/private cloud, but the themes in terms of organizational readiness hold true for DevOps as well.

    [youtube http://www.youtube.com/watch?v=gVG1Pj4cW-g&hd=1]

    So my aim in my regularly irregular DevOps column in here is to talk about everything but the technology. And where better to start that the first step towards DevOps which is, I believe, all about building cultural bridges rather than making technology choices. It should be a natural concluding that, since DevOps is all about bridging the developer/operations span, that perhaps getting people in a room to talk about their particular drivers and metrics would be a good idea. But alas most people go from talking about DevOps to deploying their choice of automation tool.

    There’s so many things to think about in embarking on this process – over on the ScriptRock blog Alan points out that visibility is a key first step before automation is embarked upon. While I agree with him that visibility is key, for me the money quote comes in his post where he writes:

    Don’t automate what you don’t understand

    But to then suggest that the first step should instead be a visibility tool does little to improve the outcomes – unless both sides fo the organizational divide are in the same room and accepting a common lingua franca and viewpoint on the data – visibility achieves little. So takeaway #1 focus on the culture. There’s lots of ways to build the bridges needed to change the way an organization looks – I’m going to cover some of that stuff over time.

  • Why you can’t buy DevOps in a box

    Why you can’t buy DevOps in a box

    Companies will often believe they can buy their way out of technical debt without making any substantial changes. They often think they can buy DevOps in a box. Well, unfortunately, devops isn’t for sale. For an organization to embrace DevOps it will need to adopt cultural change, adopt tools which enable automation, and change processes to include both dev and ops. I like to think DevOps can be explained simply as operations working together with engineers to get things done faster in an automated and repeatable way.

    Culture

    Organizations large and small have experienced these changes, the challenges encountered along the way, and the light at the end of the tunnel. Each organization requires their own set of tools and processes to embrace DevOps. Cultural change largely depends on the existing dynamic of an organization. In small companies, I have found it needs to come from the ground up and usually starts with adding the DevOps toolchain. In large organizations the value of DevOps needs to be sold to the executive team and then enforced from the top down. This is a daunting challenge for most large companies because it’s hard to quantify the value of DevOps.

    Most organizations only quantify failure when it comes to IT outages. How many outages? For how long and how severe? They often don’t quantify the success from 100% uptime or the resilience of failing over with zero down time. The ability to instrument and prove to an organization the value of applying DevOps is a crucial step to making it happen. DevOps teams efficiently deploy features faster with far less bugs. Also, when outages happen they respond faster minimizing the business impact.

    From developer to operations – one and the same?

    As a developer I have always dabbled lightly in operations. I always wanted to focus on making my code great and let an operations team worry about setting up the production infrastructure. It used to be easy! I could just ftp my files to production, and voila! My app was live and it was time for a beer. Real applications are much more complex. As I evolved my skillset I started to do more and expand my operations knowledge.

    Tools

    During the development phase, the operations staff normally makes sure the development environment is managed and they actively work to set up the test, QA, and Prod environments. This can take a lot of time if automation tools aren’t used. Here are some tools you can use to automate server build and configuration. In another post I will dive through the pros and cons of each tool, but as long as you apply some form of configuration management and treat your infrastructure as code you will be on the right track.

    Process

    From an operational perspective, my first instinct is to understand the application architecture so I can start thinking about the proper deployment model for the infrastructure components. Here are some of my operational questions and considerations for this stage:

    • Are we using a public or private cloud?

    • What is the lead time for spinning up each component and ensuring they comply with my company’s regulations?

    • When do I need to provide a development environment to my dev team, or will they handle it themselves?

    • Does this application perform functions other applications or services can already handle?

    Operations should have high-level visibility into the application and service portfolio. From a development perspective, my first milestone is to make sure the ops team fully understands the application and what it takes to deploy it to a pre-production environment. This is where we the developers sync with the product and ops team and make sure we are aligned.

    Planning for the ops team:

    • What tools will we use for deployment and configuration management?

    • How will we automate the deployment process and does the ops team understand the manual steps?

    • How will we integrate our builds with our continuous integration server?

    • How will we automate the provisioning of new environments?

    • Capacity Planning – Do we know the expected production load?

    Once developers have built unit and functional tests we need to ensure the tests are running after every commit and don’t allow regressions in our promoted environments. In theory, developers should do this before they commit any code, but often times problems don’t show up until you have production traffic running under production infrastructure. The goal of this step is really to simulate everything that can go wrong and find out what happens and how to remediate.

    Combining the power of developers and operations

    The focus on the developer/operations collaboration enables a new approach to managing the complexity of real world operations. I believe the operations complexity breaks down into a few main categories: infrastructure automation, configuration management, deployment automation, log management, performance management, and monitoring.

    In my next post I will walk through each one of these complexities and address how you can get your organization inline.

  • Ops is dead, long live DevOps

    Ops is dead, long live DevOps

    Yes, that’s a bold statement, but if the vibe at the recent AWS re:Invent conference was any indication there are a number of methodologies, technologies, and companies working to make that a reality. This article isn’t focused on whether this is a good or bad thing (my personal belief is that ops will always be a critical part of the IT delivery process). Regardless, it is critical for IT operations personnel to understand the implications of this tectonic shift. As Marc Andreessen, founder of the venture capital firm Andreessen Horowitz, has said, “software is eating the world.” As a result, ops professionals must have the ability to automate tasks that used to be managed manually. Whether or not you think that is the death of ops as we know it I will leave to you, but it does mean we will live in a world where DevOps will offer distinct advantages.

    Before we dive into what this change means to ops personnel, let’s review the underlying assumptions. The fundamental driver at the a high level has been the DevOps movement. Yes, DevOps is quickly becoming the buzzword dejour, but the underlying principles are lasting and changing IT to its core. DevOps is about integrating the various functions that it takes to release software or applications: software development, quality assurance, operations, and security among many others and rapid iteration. The process of integrating these tasks is accomplished through new processes and automation. Different functional areas such as development and operations are brought into the process earlier, and their processes are automated to the extent that they can be. This results in fast cycle times, increased quality, and hopefully increased innovation for the business. All of this is accomplished through software – at each functional level.

    Some technologies that have enabled this massive shift include Infrastructure as a Service (IaaS), Platform as a Service (PaaS), automated configuration management, server management tools, virtualization, and software defined networking. IaaS and PaaS offerings are eliminating the need for organizations to procure, provision, and deploy servers. Further, this class of technology is also dramatically reducing the need to deal with the network layer. This allows organizations to concentrate on their applications and development, and focus less on their infrastructure. The amount of time and effort spent on server and network configuration is minimized. Configuration management solutions are enabling the automation of server setup. An organization’s technology stack can be pushed, pulled, cloned and copied versus manual configuration thereby saving time and adding consistency. Top performing organizations employing DevOps go a step further and automate the scaling of capacity based on utilization. To reduce day-to-day operational activities and costs, organizations are also automating the management of servers, including users access, patches, security, and other areas through both traditional written code and SaaS solutions. Another area changing the game for ops personnel is software defined networking (SDN), which is effectively morphing data center and hardware functions into software programs.

    In short, DevOps and associated innovative new technologies are driving a paradigm shift in the way operations is done. No longer are ops personnel required to manually provision, configure, and manage the network, servers, and application stack. Combining technologies from organizations such as AWS, Rackspace, and SoftLayer along with configuration tools such as Chef and Puppet drives the provisioning and configuration of applications through software automation rather than manual effort. Once applications are deployed, the underlying infrastructure can be managed with tools such as my own company, New Relic, this author’s own JumpCloud, and others. Ops personnel have more companies than ever focusing on helping them automate the tasks that they need to deliver applications for their organizations.

    The result from all of this effort is that ops as we know it won’t exist in the future. The network operations wizard that could manually troubleshoot routing and switching issues, provision servers quickly, and tweak configurations will be tasked with automating those tasks through software. Some of that automation will be through code that ops person writes herself, some delivered by third party providers, and other portions delivered by open source or SaaS-based solutions. But, don’t shed tears for the forced obsolescence of the IT ops people. They will still exist and perhaps be more important and powerful than ever. In fact, because of DevOps ops personnel will in essence be set free from mundane, repetitive “busy work” to perform higher value tasks as well as managing over the software and systems at work. The black magic of enabling different layers of the stack to function together will be less about art and more about science, or rather good software. Solutions will be engineered to work together and be tied together through code. Caveat: yes, service providers and some large enterprises will  still run their own data centers and networks, but the same reasons that DevOps works in IaaS or PaaS will apply to private data centers or private clouds as well.

    If all of this trends toward being true as we see it happening today, what are the opportunities for ops personnel to be at the forefront of this shift? Beyond embracing DevOps – which many operations personnel are doing already – learning to code is the most critical step that ops professionals can take. Many ops professionals already know how to script which is absolutely a minimum, but many will need to go further. Embrace the DevOps movement and push further towards the development side by learning how to code. There are any number of amazing services to teach us how to code or code better. Take the step. It may be the difference between a vestige of the old school versus driving the new world in your organization. The second significant step you can take is to automate your job. Leverage the very best automation tools to code your network, server, and application infrastructure. Not only will you spend less time on these tasks manually, but you will increase availability, reliability, and performance. In most organizations, that’s more than a gold star. Finally, make the mental shift that operations in the future will be less art and more science. DevOps is driving action through data, and the technology stack is responding to this demand with more telemetry than ever. Problems are now solved through understanding performance, availability, and other metrics. And code.

    DevOps is the future. Automation is key. Code is the answer. Developers and ops are in it together for the good of their organizations. Ops pros that embrace this will win! Ops is Dead, Long Live DevOps!

  • So, you wanna DevOps?

    So, you wanna DevOps?

    How do you start your career in DevOps? How do I shift my team to use those approaches? This is a question I run into more and more frequently. Either while speaking to candidates, or among colleagues looking to move their IT operations into a more devops aligned culture.

    I want to look at that challenge from the perspective of a System Administrator. I’ve been an SA for 16 years now, and was able to fumble my way into being a pretty decent Devops Engineer. Much improved from fumbling, there are some clear objectives you can focus on that will get you there quicker.

    What isn’t devops?

    First, let’s talk about some of the differences between conventional SA and a DE. Many of these differences are direct results in the difference between conventional IT or Technical Operations and devops. A traditional SA role would involve

    • sole or shared on-call responsibilities only within the SA/NA teams

    • scripting for administration (bash, perl), very CLI based

    • silo’d from development, and possibly application support groups

    • heavy focus on stability and process

    • cultivating the walled garden

    Great DE’s share a similar skillset with SA’s – strong understanding of OS and networking concepts, deep familiarity with hardware or resource planning, diagnosis of performance bottlenecks, and instrumentation and tuning against servers or systems. In addition, DE’s

    • share on-call responsibilities with developers and overlap duties

    • utilize more robust languages (python, php, ruby), utilize web and database frameworks

    • embedded with day to day operations of development and application, actively contributing

    • heavy focus on innovation and agility

    • flat rejection of walled garden mentality

    There’s clearly a skillset component – in devops we’re writing code to admin and maintain systems, while implicitly acknowledge that systems are part of a broader ecosystem. I’m just as likely to address a file in an object storage system (s3, openstack) as I am on a file backed system. When I perform a system level action, I’m doing so because of a database flag, not a cronjob. I’m sending the result of a job to an API, not a filesystem log.

    But perhaps more importantly there is a mindset component – in devops we’re friends with the developers. I say that tongue in cheek, but am quite sure everyone reading this knows what I mean. The thing I’m most looking for in a devops engineer is their ability to relate to, and share responsibility with developers. I’m looking for developers who are excited to share responsibility with operations and contribute to the systems their code runs on. I think this is the most marked contrast between old world IT and devops – they integration of two traditionally warring tribes.

    So how do you start the transition?

    Code. Write code. Not shell, not perl. Perl is code, so is shell. Choose a language, it doesn’t matter much which, and write a basic CRUD app that you can interact with via a web browser. In my experience this kind of coding is rare in traditional SA focused folks, but absolutely critical for a DE. As part of your project, schedule at least 2 hours to feed the trolls about whether python is better than ruby.

    Puppet or chef. Stop everything you are doing and pick one of these up. Make yourself take an existing system and install apache on it using one of these tools. Configure apache using the tool. Make changes only in puppet or chef. As one recent AWS tips post said, disable ssh. In 2014 if you are not using puppet or chef you are terribly handicapped.

    Get into the cloud. Set up an EC2, Google Compute, or Azure environment. Start and stop instances only with API calls. Make sure that once started, the instances are applying a puppet recipe. You can have a functional playground for free, I’ve had a personal goof off area in ec2 for a couple years now. $18/month or so is a small investment in your career. Set yourself some basic tasks, and implement them in a cloud environment. 

    DevOps is here to stay, and is a massively impactful methodology for quick iteration against systems and code. Whether you’re at a startup, or an enterprise, or looking to shift… adding some basic devops tools to your bag o tricks will be time well spent!

  • DevOps’ Chicken Or Egg Origins

    DevOps’ Chicken Or Egg Origins

    When acolytes explain the DevOps movement to the uninitiated, they usually run into a chicken and egg line of questioning. What is DevOps, really? Is it a culture? Is it a pattern of processes? Or is it a technology? What comes first?

    The ratios of the perceived importance of each of those elements may shift depending on who you talk to in the DevOps community. But the inevitable answer to these questions is usually that DevOps depends on at least a little of all of them, but probably on culture above all else.

    “I think it’s first and foremost cultural. The development team is a tribe. So is operations.  They have norms.  They have things they value.  They have things that they reward and admonish,” says Josh Corman, chief technology officer for Sonatype, a component lifecycle management firm. “Developers are incentivized to cause change.  And ops people are incentivized to prevent change, to promote stability.  So they were natural enemies.  And when they work it out and align to each other’s tools and incentives, they’re able to go faster.”

    Therein lie the historical roots of DevOps, agrees Andrew Storms, senior director of DevOps at cloud security firm CloudPassage, who explains that it is an answer to the blame game that inevitably occurs in the old model.

    “I think that existing corporations or companies or groups are probably doing DevOps in response to existing pain,” he says, explaining that in previous non-DevOps shops he worked at, it was inevitable that development would write a piece of code, throw it over the fence and let ops catch the heat when trouble brewed. “The question is, why is the person at the end of the train always feeling the pain for something that originated at the beginning of the train? To me, DevOps says, ‘There is no train.’ There’s no more division of who’s in trouble for anything. Everybody is on the same darned team.”

    It’s for this very reason that Jez Humble, a principal for the consultancy ThoughtWorks and co-author of Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation, says that the fundamental ingredient to successful DevOps is trust.

    “You need to trust each other. Developers can’t just assume the ops people are all dumb and don’t know anything about code or are just command line monkeys. And the ops people can’t assume the developers don’t know anything about how systems work in real life,” he says. “These assumptions make it all a big finger-pointing exercise.”

    A lack of trust is the underlying roadblock to better teamwork between dev and ops, but it can be difficult to achieve within enterprises.

    “In small organizations people trust each other because they can see each other and work with each other and must learn to trust to get anything done,” he says. “But as organizations scale you lose that human contact.”

    It’s replaced instead with brittle and bureaucratic processes that only feed into those distinct walls between each tribe. However, as organizations feel the pain that’s inflicted by those bureaucracies that have evolved so far in IT and by the walls thrown up as a result, they face a choice. Engender that trust and a culture that allows  dev and ops to simplify processes and tear down the walls, or face further roadblocks that will hurt the business.

    From there, it is a matter of establishing behavioral and process patterns that are a hallmark of DevOps cooperation.

    “I think ‘patterns’ is the right word for it. People get hung up on this textbook or technical definition of DevOps.  And while there are certain common processes and things like Kanban boards and Scrums, and sometimes it’s a hybrid of Agile, sometimes not that ,” Corman says, “what matters the most is an attitude to fail fast and iterate, to continuously look for ways to increase flow, to automate, instrument and find and root out weakness. And to be responsive to new ideas of the business, or new ideas of your install base, or new market pressure.”

    According to Corman, the beauty of DevOps is that it’s a ‘jump ball’ for IT, so there are no set processes that make or break whether a shop is DevOps or not.

    “It’s in a state of flux, and the cement hasn’t dried.  It’s a singular opportunity to redefine the way we engage with the business and drive value, the likes of which we will not see again for 30 years,” he says. “It’s very, very rare.”

    And while few within the DevOps community would say it is defined by technology, it certainly is enabled by it.

    “To me it is more a cultural shift than a tool shift, but that tools shift presents a huge change in the way IT runs,” Storms says, explaining that instrumentation and orchestration tools like Puppet and Chef makes it possible to achieve continuous improvement and fast iterations. “So does it come about through the cultural change or the technical change? I think, actually, neither. I think it happens at the same time. That’s the force multiplier, that two plus two equals five kind of thing.”

    But like tectonic shifts, changes in IT happen sometimes all of a sudden, but mostly in increments.

    “One part of it is focusing on what we can do every day to make things a little better for everybody. You have to proceed incrementally and accept that it’s going to take time,” Humble says. “You can’t say, ‘Well, we’re going to adopt Agile now across the whole organization—go!’ There has to be enthusiasm from the people on the ground to do this stuff and then you have to work together though a process of experimentation and trial and error to work out how to make it work in practice, probably with a small group of people initially.”

    Even with that force multiplier, Storms says, you’ll still see positive steps forward by establishing an element of DevOps one at a time.

    “Even if you’re in a difficult situation where you can’t have the force multiplier of culture and technology changes at the same time, just one or the other is still going to be a big win,” he says. “Let’s say you just took the one piece of ‘Let’s move our dev and ops teams together and make them be friends,’ you don’t necessarily have to orchestrate and instrument like crazy to win. But you can also do the opposite and just picked the orchestration piece and say, ‘Hey, ops people, stop doing things by hand.’ You send them to Chef conferences and make them learn Ruby and guess what? Their skill sets just went through the roof.”