devopsy.kinsta.cloud

Tag: devops

  • Moving Security To the Left In a DevOps World

    Moving Security To the Left In a DevOps World

    Moving security to the left has become a coined phrase meant to describe the process of getting the security team involved earlier in a process.  Most typically, the phrase is used in conjunction with IT or software development projects. One of the top suggestions for ensuring security in a DevOps world is to move security to the left in the process tool chain. But what exactly or how exactly can you move security to the left?

    Grab that Open Seat

    The new DevOps process pipeline created an opening for a seat at the table for security. Prior to DevOps, the development team owned the entire pipeline from plan to release.  The Ops team would historically receive the release from over the wall and then be responsible for the deployment and operating the software inside the production environment.  With the Ops team joining forces with Development in the process pipeline, that has opened a proverbial seat at the table. What I’ve started to witness are security teams taking advantage of the musical chairs to quickly grab the open spot.

    DevOpsProcessPipeline

     

    Taking a seat at the end of the development pipeline is not enough.  Too many security teams are still being relegated to positions too late in the process.  While running pen tests and security code reviews after a release is better then nothing, its still not ideal.  Security needs to find ways to add value to the process so they can act as a force for positive change instead of the after thought.

    Threat Vector Analysis

    Threat vector analysis is part science and part art and part trying to guess uncertainty. What is certain, however, is that most developers would rather be spending their time writing new functionality instead of trying to understand how an attacker could be break their code. The security team could offer a tremendous service to an organization by offering their expertise in this area.

    Continuous Integration Security Testing

    Continuous integration is a key component of DevOps. The security team can leverage the development chain to insert early yet important risk control tests. Those tests could be for example static or dynamic code analysis. Other tests could be simple yet known vulnerability checks in included libraries. The OWASP introduction to testing guide lists a plethora of application security tests that should be checked often and early.

    Author Micro Security Services

    Think about how great it would be if there was a group who specialized in authoring and delivering security related services into your application?  Proponents of services models generally promote splitting the development team up into groups that are organized around business capabilities.  For example, there are separate development groups for UI, middleware and DB.  What’s missing in most of those models is who is responsible for security services such as authentication, authorization and audit. Security has an opportunity just waiting for them to add incredible value while also ensuring the security services used pass muster.

    Summary

    Where we once had Ops fighting their way to be part of the entire development process, we now have security trying to do the same. Ops managed to garner their way into the club by coming to the table with invaluable skills and resources. Now its up to security to present their creative sweet spot in order to secure their own seat at the table.

    Further Reading

    Andi Mann on Ensuring security and managing risk in enterprise DevOps

  • Enterprising DevOps in Banking sector

    Enterprising DevOps in Banking sector

    “Shift left” is common in project life cycle management where the risks are mitigated early in the cycle by involving QA team early in the planning & development phase. In the  IT support systems, shift left empowers support staffs with enough tools, processes & knowledge base so that the issues get resolved at a point closest to the customer. The purpose of this process is to create a leaner organization and increase customer satisfaction. DevOps is a continuous shift left that helps in high velocity product delivery and enhance the quality of the deliverables.

    Why DevOps is crucial to Banking Sector?

    a.       Multi-channel delivery

    Traditionally banks have been offering products & services that were physically distributed through branches/ATMs in a brick-and-mortar fashion. But with the rapid increase in the use of smartphones & tablets, the banks are challenged to transform their distribution channels and trend towards drive-to-digital. The study from eMarketer.com shows the distribution of banking channels used by various US Internet users. The distribution of banking channels increases the need for multi-channel interactions, a consistent UX across different channels like mobile/internet/TV banking, ATMs etc and an integrated channel to offer a seamless user experience. This in turn requires a transformation in the way the applications are developed & deployed, while adhering to high level of security & compliance. Now considering that each of the banking product is managed by a separate division with its own operations team, offering an integrated delivery channel is extremely difficult.

    b.      Security updates and hot fixes

    It is not long since the outbreak of the Heart Bleed vulnerability. It not only impacted the websites but also the banking applications using OpenSSL technology. Hackers could impersonate the banking services and steal user credentials. Banking Applications using OpenSSL were patched. Similarly, Zero-day-attacks pose a serious risk to banking applications as the hackers take advantage of the time before the vulnerability is publicly announced and a patch is available & applied. In early Feb 2014, a zero-day-attack by name IE “Operations SnowMan” [ CVE-2014-0322] could steal banking credential if the user uses IE 9 or 10. DDoS (Distributed Denial-Of-Service) & Man in the browser (MitB) attacks are some of the other top vulnerabilities that can threaten the security of the banking applications. This urges the banks to test & release the hotfixes / patches at the earliest without breaking the existing functionalities and most often make live patches while the customers are still using the banking applications.

    c.       Regulatory and legislative pressures

    US Patriot Act (Providing Appropriate Tools Required to Intercept and Obstruct Terrorism Act of 2001) has changed the way the banks identify their potential customers. All US banks and other oversees banks that have their correspondent account in US need to get certified under this act. As stated here the banks need to have teams & processes in place to ensure that it complies with these regulatory requirements. This mandates a risk management framework and a reporting mechanism. This increases process overheads, which inturn slow down innovation and product delivery.

    DevOps to the rescue

    DevOps philosophy is similar to Toyota’s Kanban / JIT (Just-In-Time) production. Banks are familiar with Agile & XP processes for product development, ITIL/ITSM for Operational efficiency, yet they are not JIT production ready. Let us pick the two major aspects that gear up the banks to be DevOps ready.

    a.       Organizational Structure in Banks

    We all acknowledge the fact that there is a huge gap between R&D and IT Operations. But, bridging the gap between R&D & IT can be complex based on the magnitude of the size of the organization, geographical distribution, compliances and processes. In 2011, Citi bank initiated a multi-billion dollar project named “Project Rainbow” to consolidate its IT platforms across different product portfolios like mortgages, credit portfolios etc. The focus was to move away from product-centric strategy to a more customer centric strategy and improve communication between the teams. In similar terms, RBI [Reserve Bank of India] recommends aligning the IT Development, Quality Assurance & Operations. The above picture is a recommendation of the organizational structure by RBI. This greatly improves the collaboration within the teams and help them move towards a common goal.

    b.      Automation

    Banking business processes are typically split as back office, mid office and front office processes. Back office processes are those that do not require direct customer interaction but involve a lot of data processing. Typically, banks have around 300 – 500 back office automation processes, some of them are repeatable and highly structured. These back office processes like HR management, administrative tasks & IT operations are good candidates for optimization and cost reduction. devops Mid office processes provide the necessary decision support system. Analytical reports, collaboration tools that foster communication and process unstructured data fall under this category. IT systems monitoring and reporting automation can identify the systems usage and reduce IT sprawls. Front office process are customer facing and requires constant up to date information from the mid and back office tools for quick response times. When these front, mid and back office processes are automated and integrated, it greatly increases the velocity of product delivery and increase customer satisfaction.

    Success stories like Loyld, show that banks can reduce cost and operational complexity by automating server configurations and release automation. Automating software testing can greatly improve defect containment and testing efficiency. Thus the banks can make consistent software deliveries across multiple delivery channels, apply hot fixes quickly or accommodate new regulatory needs, once they are DevOps ready.

  • My Little Operations: DevOps is Magic

    My Little Operations: DevOps is Magic

    Bear with me, it will all make sense in the end.

     

    If you aren’t familiar with My Little Pony: Friendship is Magic (because you don’t have a child who watches it over and over and … well, over)  let me sum up for you:

    There’s a pony who knows something horrible is coming. She wants to stop it. She’s instead sent by management to make friends, which seems to her to be ridiculous in the face of a looming catastrophe. When that catastrophe happens, she and her new “friends” (her coworkers) set off to find the elements of harmony – the only thing that can save the world. But lo and behold, evil destroys the elements. Or so it appears. See, it turns out that the friends are the embodiment of the elements of harmony and through their friendship they create the most powerful element of them all: magic. Together, they are strong enough to defeat the evil and restore harmony to the data center. Err, I mean, to the pony world.

    If we take this and apply it to operations, you should start seeing some common themes. Primarily, you might note that, like magic, the ultimate goal of DevOps – agility – can only be “achieved” by the collaborative efforts of the other elements of operational harmony, namely predictability, repeatability, composability, consistency and stability. It isn’t a tangible “thing” you can buy, it doesn’t come in a can, it can’t be delivered by Amazon.

    These elements combine and, when practiced cooperatively, create a new element: agility. Which is a significant goal of DevOps operational harmonybased on a variety of surveys. Oh, it’s always stated a different way: we want to improve the time to market, increase frequency of deploys, reduce downtime due to errors, etc… but in general, these all fall under the need to enable the same level of “agility” in operations (and the network, but that’s a different discussion) as is present in application development today. Fast and frequent change to match application (and business) changes. Agility.

    But agility, like magic, can only be achieved through friendship (or as we’ll call it, collaboration). That’s because agility requires a variety of separate concerns (qualities) to come together in concert before it can be realized:

    Agility (DevOps)

    • Predictability (time to deploy, important for planning)
    • Repeatability (of process to other applications)
    • Composability (reuse of steps in other processes)
    • Consistency (important for maintainability and trust in process)
    • Stability ( of the overall infrastructure )

    That’s why it’s important not to look at DevOps as a set of tools. There are multiple facets to DevOps that can be enabled by tools and enhanced by tools and even monitored by tools, but DevOps is not the tools themselves any more than the elements of harmony were the gems the Little Ponies sought. The real magic of DevOps is the agility created by collaboratively working together to bring each of the elements into operations. Without stability, have you achieved agility? Or is it really chaos?

    If you can’t repeat the process (apply it to other applications) have you really achieved agility or just strictly codified a single deployment? If the process isn’t consistent, and gives different results have you realized an agile environment or anarchistic one?

    These elements are necessary to achieve real agility of operations, and those elements can only come about in a cooperative environment in which they all work together.

    DevOps, so it seems, is magic after all.

     

  • Collaboration in Progress: Making DevOps a Reality

    Collaboration in Progress: Making DevOps a Reality

    Everything on your company’s DevOps checklist is accounted for. It’s got the vision and the talent. It’s identified the problems and mapped out a solution. It’s got a culture that allows the flexibility that such a plan requires. Now’s the time to put the plan into action and let the benefits begin.

    The truth is that it may not be that simple. A great plan won’t mean anything unless everyone understands how to achieve it. And whatever a company’s DevOps strategy consists of, collaboration is the key to making it a reality.

    Collaboration is at the heart of DevOps. It’s even conveyed in the phrase itself. No matter the size of your company, this new process of teamwork and transparency relies on combining strengths between Development and Operations to achieve goals that otherwise wouldn’t be possible.

    Breaking Silos with a Full-Circle Approach

    With one department responsible for writing code and another responsible for implementing it, silos can naturally form as each group focuses on its job. This kind of structure is what commonly gives rise to software deployments taking longer than they should and even little or no communication taking place between the two departments.

    DevOps has come about as an antidote to these kinds of siloed situations in which inefficiencies and a lack of teamwork are the norm. Yet for DevOps to take off, both Development and Operations have to participate fully and understand that collaboration requires equal cooperation from both sides. Such collaboration has to take place not just at certain stages, but throughout the life of a project. When each team is invested in the effort by providing feedback, helping identify future problems and continually learning from day-to-day deployments, a company has a great chance of seeing benefits from these new collaborative efforts. Without complete buy-in from everyone, the process will never get off the ground.

    The Evolution of Collaboration

    Just as DevOps is about moving beyond the assembly line process of software development, the concept of collaboration itself is evolving. DevOps-style teamwork can consist of more than breaking down the silos between different teams. In an agile infrastructure, team members can wear different hats and swap responsibilities depending on the demands of the day. Instead of a scenario in which people work on single specialized tasks without any interaction with the larger group, you’ll have contributors who can play multiple roles within a team and understand how this boosts productivity for the entire company, rather than just a single department. It’s possible that a team member can function as a developer one day and then work on deployments another. Through DevOps, an organization can advance beyond one-dimensional departments and one-dimensional contributions.

    Real-World DevOps Teamwork

    For many within the IT industry, DevOps is already a working reality. Breaking silos and becoming more agile are goals for companies that know how increased collaboration can help their business. A good source that lays out some first-hand experiences of DevOps practitioners can be found in the recently published Rackspace e-book, The DevOps Mindset: Real-World Insights from Tech Leaders. It features interviews with IT experts who discuss what led them to adopt a DevOps approach and recommendations for those interested in a new agile approach to software deployment.

    When discussing the importance of agile teamwork, James Kenigsberg of 2U remarks that, “To achieve true DevOps collaboration, you need your employees to really think and act as one, not just be merged together in name only.” As a provider of online university programs, 2U faces a huge demand for quick, dependable deployments and underwent a transition to increase teamwork at the company. “We had to flip the model,” Kenigsberg explains, “from the idea of personal heroics and siloed responsibilities to real, cross-functional collaboration.”

    Jim Kimball of fund administrator HedgeServ reiterates this cross-departmental approach. “Repeatability and scalability are our business,” he explains. “The capabilities that DevOps gives us directly enable this by encouraging collaboration within the organization to come up with better solutions.”

    Forging a Partnership Between Dev and Ops

    Assembling a skilled and talented group that will be the envy of your industry is the ideal every company would like to have. If you’re able to gather a team that’s the IT equivalent of the X-Men, it’s not enough to just assemble that talent and assign them siloed roles. Without frequent teamwork, not only will company goals become difficult to achieve, but solving problems will be nearly impossible.

    When Development and Operations see themselves as working in partnership with each other, the foundation for greater collaboration will be in place, and company goals, such as greater speed to market, increased productivity and an all-around agile performance, will be within reach. With the right mentality, DevOps can be an evolutionary step in company functionality and reach new levels of efficiency that otherwise wouldn’t be achievable.

  • Provisioning versus Configuration Example

    Provisioning versus Configuration Example

    A few weeks back I wrote about the difference between configuration and provisioning noting, primarily, that there are differences between the two tasks. It remains an important distinction to make because it’s really where the rubber meets the road (or the app meets the network) where it becomes important. As the infrastructure/network side of the house is where we’re concentrating our efforts to apply DevOps, it’s necessary to dig a bit deeper on this topic.

    So today we’ll examine  couple of concrete examples of the difference – and why it’s important.

    Your application is being deployed and it needs, wonder of wonders, a load balancer. Cause, elasticity. Scale. You know, cloudy type stuffs.

    So you grab that golden image and launch that virtual machine with Whatever Your Favorite Load Balancing Thing might be (HA Proxy, Nginx, F5, etc…). Voila! You’re ready to go, right?

    Not by a long shot. You’ve provisioned a load balancing service, yes. But is it configured? Probably not. In addition to the basic networking needed (IP addresses, VLANs, etc…) there are also a whole bunch of very application specific things that must be configured. There’s no “golden” load balancing service that can be generically applied to your application. Period. At a minimum the virtual server (that represents your application to the rest of the world) needs two things: an IP address and a pool of app instances to select from.

    simplified-lb

     

    That means there’s some configuration that must occur after the provisioning. There’s no single LB image that will work for all applications, period. Even if all other settings are the same (LB algorithm, L7 app routing rules, TCP options, etc..) you still have to tell it which pool of real servers to select from and what IP address to use so clients can connect.

    This is true for pretty much all application services (infrastructure) that support an application deployment. Because there’s no single golden image that just “works” for everything and no way to use dynamic configuration options for everything (DHCP for IP addresses is one thing, but there’s no “dynamic load balancing algorithm selection” service out there yet) there remain a set of options that must be configured after provisioning occurs. Consider application security like a web app firewall (WAF). It has to be configured to protect a specific application. It has to learn the URIs and the data expected to be exchanged so it recognizes when some interaction is out of bounds and potentially dangerous. That’s unique to the application, and thus it requires some specific configuration after provisioning.

    It’s important to note this distinction lest we get caught up in believing that we can simply create a golden image of all our network infrastructure and services and put them into production with the touch of a button.

    This little reality also helps to understand why it doesn’t really matter whether the network service in question is launched on custom hardware or some server off the shelf. The bulk of the work when automating the deployment of network services is in the configuration, not the provisioning.

    While you can minimize the amount of post-provisioning configuration necessary, you can’t eliminate it entirely. That’s why API-enabled, programmable infrastructure is a critical piece of the larger DevOps picture. It is the API, programmable templates and similar capabilities that enable the configuration to be automated and orchestrated so as to not be the bottleneck in the deployment process.

  • The DevOps Litmus Test: Is Your Company’s Culture Ready to Get Agile?

    The DevOps Litmus Test: Is Your Company’s Culture Ready to Get Agile?

    New ideas don’t come to those who are afraid of breaking the rules. Improving something usually means you’ll have to push the status quo aside and try something that hasn’t been done before. What may be a standard operating procedure will have to make way for a brand-new approach with its own set of rules and goals.

    This applies directly to DevOps, which requires changes in how departments operate, how employees interact and how problems get solved. DevOps may be an idea your company is considering, but if the culture isn’t compatible with it, DevOps will never become an active part of your day-to-day operations.

    If DevOps Is the Engine, Culture Is the Fuel

    Let’s say two employees who don’t normally work together wind up taking an unprecedented approach to solving a problem. To their surprise, this new way of working together is incredibly efficient and a big improvement over their usual methods. The two recommend this kind of collaboration to others in their respective departments when it comes to spotting problems, fixing them and preparing for future ones.

    That’s the moment when you’ll know how receptive your company is to DevOps. This kind of cultural litmus test determines whether department silos can be broken down and new avenues of teamwork can happen. No matter the company or industry, its culture is the foundation for DevOps adoption. When that culture fits, DevOps is an ideal vehicle for improved productivity, with culture being the fuel moving it forward.

    Getting the Right Cultural Fit

    In order for DevOps to happen, you need the right conditions, similar to how certain plants only grow in the right ecosystem. Team members must be able to interact in ways they may not have in the past. The right culture is one that allows flexibility, spontaneity and resourcefulness to take place.

    On the flip side, DevOps won’t work in a rigid environment where departments are siloed and methods can’t be questioned. DevOps has a slim chance of taking off in a company that wants to hold onto previously established rules about divisions between departments. Anyone who stands up to say, “That’s not how we do things,” will be a hindrance to progress. Greater benefits are in reach, but to make them a reality, a company and its employees have to be willing to make some changes and allow agile collaborations to flourish.

    DevOps Culture in the Real World

    For those who have already put DevOps in action, culture was, and continues to be, a central issue. Household names like Twitter and Netflix may be known for their DevOps practices, but many lesser-known businesses have found that their cultures identify with DevOps collaborations. The recently published e-book from Rackspace, The DevOps Mindset: Real-World Insights from Tech Leaders features interviews with a variety of DevOps practitioners. The importance of culture is a common theme throughout these discussions.

    These firsthand accounts emphasize how encouraging new ways of solving problems depends on the proper culture. As James Kenigsberg of 2U explains, “You don’t have to be in Silicon Valley to embrace the DevOps culture; anyone can do it anywhere with the right people.” Bharat Krish, CIO of HBO Latin America, also emphasizes culture as an essential factor. “When I think of a DevOps culture,” Krish says, “for me it’s building a foundation where development can happen in a more agile fashion.”

    Start with the Biggest DevOps Requirement

    Knowing that DevOps benefits one company doesn’t necessarily mean it will automatically benefit another. Company processes, structure and communication standards all play a part in determining whether DevOps is right for your business. DevOps isn’t a product that can simply be plugged in – it requires a new way of thinking about agile software deployment.

    Culture has to be at the center of a DevOps adoption, and assessing your company’s culture is a crucial first step in determining whether DevOps will be a good fit for your organization. As long as your company is flexible enough to allow some cross-departmental collaboration and silos to be broken, then prepare for DevOps to take you to new levels of productivity. You’ll find that breaking the rules isn’t such a bad thing.

  • Specialists vs. Generalists for Enterprise DevOps

    Specialists vs. Generalists for Enterprise DevOps

    Welcome back to the devopsy.kinsta.cloud series, ‘Enterprise DevOps Q&A.’ I have a backlog of Q&A in store, but don’t let that stop you sending in more! If I can’t answer them myself, I will try to find someone who can.

    For this post, I was recently asked about the challenge that large enterprises are facing over whether to use generalists or specialists to drive their DevOps transformation…

    Q. Does a DevOps environment prefer generalists or specialists to succeed?

    This question has ricocheted for a while. Google CIO Ben Fried – who spent 14 years in IT at a large enterprise (Morgan Stanley) – spoke as long ago as 2011 about how ‘Generalists, Not Specialists, Will Scale the Web.’ Some webscale businesses like Netflix even espouse a singular ‘NoOps‘ skillset, although they continued to have separate specialists … and maybe still do.

    However, in large enterprises, I have seen DevOps work well with both specialists and generalists.

    The Case for Generalists

    A generalist who can take on more duties from design to deployment is especially productive when application alignment moves to a local or vertical approach, as I have discussed before. Such generalists can move code through the SDLC faster, with greater integrity and fewer opportunities to drop the ball. In such structures, there are fewer handoffs from ideation to production, so there is less chance something will slip through the cracks.

    Local and vertical organizational structures tend to be less complex, with fewer products, fewer services, fewer applications, and fewer technologies involved, so such teams can better emulate smaller startup-sized organizations. DevOps generalists in these structures, for example, will not have to learn the vagaries of 5 different provisioning tools, because they all use just one tool. Generalists will often do better in such structures.

    The Case for Specialists

    On the other hand, in a large enterprise you might have not only 5 different provisioning tools, but also 4 test automation tools, 3 middleware systems, 5 databases, and much more – housed in a dozen different departments! It is perhaps impossible then for a generalist to have the expertise required to be proficient in all of these processes and technologies. As anecdotal evidence, look at any web site or mobile app ‘designed’ by a developer – a good proportion of them are almost unusable.

    In the enterprise even specialists like DBAs or security admins need to specialize within their field. An Oracle DBA, for example, has specialized knowledge that a Hadoop DBA does not. Big enterprises adopting big DevOps need specialists who know specific differences between DB2, Oracle, MySQL, and Hadoop. And as I have posted before, it would be nice to standardize on one, but such a dogmatic approach is simply unrealistic.

    The Case for ‘T-Shaped’ People

    That said, as with many conventions in Enterprise DevOps, a best practice seems to be in finding balance. Build up leaders who can truly collaborate across disciplines with experts in other areas, while still retaining a depth of skills and expertise in their specific field. This is the idea of ‘T-shaped’ people – aka ‘Generalizing Specialists’ or ‘Master Generalists’.

    This seeming oxymorons are actually useful archetypes to apply to DevOps staffing, especially when supported by expertise from specialists from the PMO, architecture, development, DBA, security, testing, QA, pre-prod, and operations teams. This is the best of both words – generalists when you want them; specialists when you need them.

    The Case for Pragmatism

    Perhaps more importantly though, as Donald Rumsfeld once famously noted, “You go to war with the Army you have … not the Army you might want.” In a large enterprise the staff you have will likely be specialists, regardless of the staff you might want, so this will be your starting point.

    Fortunately, building a collaborative cross-functional team from separate groups of specialists, rather than a single DevOps department, is as much of a DevOps best practice as seems to exist. Take for example a large enterprise like National Australia Bank, which has have taken a DevOps approach that started with specialists but ultimately resulted in a ‘DevOps Team.’ On the other hand, organizations like DirecTV and Union Bank started their DevOps journeys by helping dev, test, and QA to streamline within their specialization, as much as they looked to streamline the collaboration between teams of specialists.

    The Case for Another Case?

    Did you handle your DevOps staffing differently? Is there another way to reconcile the differences between generalists and specialists? Please feel free to add to the discussion in the comments, I would love to hear more about your approach to enterprise DevOps staffing.

    And as always, if you have questions you’d like to address about Enterprise DevOps, let me know. Leave them in the comments below; hit me up on Twitter at @AndiMann; or send me an e-mail to me at andi.mann@ca.com.

  • Keep Calm and Refactor Operations

    Keep Calm and Refactor Operations

    Developers have been refactoring forever to gain efficiencies and avoid mistakes. Let’s apply more than just theory to ops to do the same.

    keep-calm-and-refactor-operatorsThere’s about a million articles, blogs, tweets and infographics available today that bespeak of DevOps as this or that or the other thing. But one core truth remains: DevOps in part attempts to apply methodologies like agile to operations in order to gain efficiencies and avoid mistakes. We need the efficiencies because the number of apps that need to be deployed, updated, migrated, and maintained is increasing, not decreasing, and there simply isn’t enough time in the day (or during the monthly change window) to get it all done.

    We need to eliminate mistakes because they are, according to data from initialstate, very costly. It takes 30x longer to fix one bug (an error) than it does to write one line of code. Anyone who has ever spent time chasing a wild pointer will immediately say that’s too low an estimate, but anecdotally I’m willing to go with that figure. It is an average, after all. To put that in perspective, the same data indicates that developers must find and fix 100 product bugs per 1000 lines of code. That’s a 10% error rate.

    I’ll let you ponder for a moment what that means to DevOps and the desire to treat infrastructure as code.

     

     

    Exactly. You probably want to avoid that kind of error rate as you go about automating and orchestrating provisioning, configuration and general management of all that infrastructure. Just a few errors can effectively negate time gained by automation. So how do you avoid that?

    One of the ways to avoid some of that is refactoring.

    Refactoring is most commonly used to extract pieces of code (script) that repeat over and over into composable functions (in a procedural language) or methods (in an object-oriented language) or services (in a micro-service or SOA-based system). It’s a primitive form of abstraction that serves to produce more manageable, extensible code bases.

    Basically, you’re taking a set of actions that you repeatedly perform and turning them into a reusable, repeatable “thing”. In the case of operations that “thing” is probably a script.

    Say, for example, that every time you deploy an application you’re also deploying a load balancing service because, of course, you’ll want it scalable and/or reliable. You always perform the same set of steps to complete this task with only a few changes – usually IP addresses, host names, or other network and app-specific settings.

    But you’re doing it manually, or writing a new script every time. Manually making changes are, in most cases, where errors are introduced. Fat fingers are a common cause, especially when you’re not paying close attention because, well, boring! You’ve done this fifty million times and quite frankly, you’re sick to death of the tediousness of it all. refactor all the things

    We’ve all done it, don’t pretend you haven’t both thought it was boring and made a mistake. You’re in good company, trust me.

    What developers do when this is the case is refactor the common code and reuse it across the system. Those bits that get changed are parameterized and everything else stays the same. Even DBAs do this, they just call the resulting code a stored procedure.

    The result of such efforts are self-contained automated tasks that, when used to orchestrate a process, enable composable operations. You string together a set of these scripts and voila! Provisioning complete, minus the fat fingered errors or missed manual steps.

    Refactoring code is much more complex than this, of course, as would also be the case within operations, but the general premise here is that the practical application of DevOps includes the process of refactoring operations. It’s about restructuring it into more composable, logical and discrete tasks that can be automated and orchestrated as a means to achieve more manageable and extensible operations. It’s about identifying those steps in a process that are repeatable and can be refactored into a single, shared executable task.

    When starting an initiative that applies DevOps it is likely you’re going to have to prove the time you’re taking is worth the investment. One of the ways to do that is by refactoring those operations that are continually used and reused to enable a more consistent, predictable and repeatable provisioning and deployment process. By choosing those steps that are most often repeated and refactoring them into their own composable scripts, you can gain efficiencies as well as reduce errors (and eliminate the time they require to find and fix).

  • DevOps cuts out the unnecessary middle-man

    DevOps cuts out the unnecessary middle-man

    How many IT people does it take to produce an application? Actually, let me rephrase that. How many IT people does it take to produce an application effectively and efficiently? For many organizations, the answer to the second question is less than the answer to the first because the traditional IT hierarchy includes layers of middle-manager decision makers that weigh everything down.

    One of the things that DevOps helps companies accomplish is to eliminate unnecessary bureaucratic obstacles so the people trying to get things done can just do what they know how to do. The traditional IT model reminds me a little of putting a police officer in the middle of an intersection to direct traffic when there are already stoplights or stop signs present. Every licensed driver already knows the proper protocols and procedures for driving collaboratively and sharing the road so that everyone gets safely through the intersection. The police officer just disrupts an otherwise smooth process, and wastes resources that could be put to better use elsewhere—like maybe pulling over that car going 75mph in a 45mph zone, but I digress.

    For some reason, it is a fairly normal, natural evolution for a business to add layers of middle-management decision makers as it grows. Sometimes those roles are a function of wanting to promote people when there is really nowhere to promote them to, and the role makes sense at the time. In fact, the initial creation of many such roles is actually intended to help the business operate more efficiently. An upper manager starts to get too much on his plate, and delegates power to intermediaries so that decisions can be made at a lower level, and the upper manager’s lack of availability doesn’t become a roadblock to progress.

    Over time, though, middle managers become a cancer on the organization. The need for the role typically fades away, and you’re left with people in unnecessary roles going out of their way to justify their own existence by asserting themselves and making sure everything goes through them. When big companies like Microsoft or General Motors decide to pare down the company, middle managers are generally first on the chopping block because nobody can even explain why they’re there in the first place.

    That’s where DevOps comes in. By its very nature, DevOps enables teams to function more autonomously and take responsibility for their own work. It accomplishes the same goal that originally justified creating middle-manager roles—it moves decision-making power to a lower level so that scarce upper managers don’t become obstacles for productivity. DevOps does it by bringing decision-making power down to the trenches where those involved understand what they’re doing and why they’re doing it, though, rather than adding an extra layer of red tape that ultimately ends up adding, rather than removing obstacles.

    Like drivers approaching an intersection, the people doing the actual work have a firm grasp on what’s supposed to happen. They understand how things work, and what needs to happen to get things done. The traditional middle-manager hierarchy is like the unnecessary police officer “directing” traffic, while a DevOps approach treats the drivers like sentient adults, and assumes they all know what a stop sign or a green light mean without being told.

    In the end, it may add to our employment challenges. The US is already experiencing a shift away from manufacturing jobs as a result of technology and automation making workers obsolete. If we also decide collectively that middle managers are obsolete, we end up with more people without a job. That is unfortunate, and it is a problem that will need to be solved, but businesses can’t solve it by creating unnecessary roles just to give people something to do.

    For younger organizations that have embraced DevOps concepts since inception, DevOps should help prevent the cancer of middle managers. If teams and individuals are given responsibility for themselves, there is no need to create an additional babysitter role to police them. For established organizations that already have middle managers, eliminating those roles creates a natural vacuum where DevOps concepts can flourish if allowed.

    Regardless of how organizations get there, the result of embracing DevOps is that the middle-man is removed from the equation so development occurs more effectively, and more efficiently with fewer people.

  • The DevOps Diet (Getting Lean)

    The DevOps Diet (Getting Lean)

    IT has gotten fat – fat silos, fat processes, fat procedures.  Adopting DevOps is a great opportunity to look at some of our unhealthy habits and identify where we can eliminate unproductive waste from our process diets.  DevOps will ultimately teach IT how to exercise more with less effort, trim the fat and get lean.

    According to lean principles, the seven main areas of waste are:

    • Defects –  variations from requirements that result in interruptions and re-work
    • Overproduction – delivering something more or before it is required
    • Inventory  –  carrying excess raw materials, work in progress (WIP) or finished goods
    • Over-processing – doing more work than is required
    • Motion – moving people or equipment more than is required
    • Transportation – moving products from one location to another
    • Waiting – doing nothing or moving slowly while waiting on a previous step

    What are the greatest sources of waste in your IT organization?  Are defects being regularly passed downstream? Are ITSM processes overproducing through complexity and bureaucracy?  Are Agile teams  dealing with excessive work in progress and less finished product?  Are unresolved bottlenecks resulting in frequent delays while waiting for something to get done or someone to be available?  Could automation reduce “motion sickness” and make transportation nonstop? These questions will hopefully facilitate dialog between Dev and Ops about analyzing and streamlining existing workflows.

    While waste is fattening, it is not necessarily ineffective.  Projects and other tasks are still getting done and therefore some areas of waste may be difficult to recognize.  To become leaner, start simple by identifying and eliminating one unhealthy habit from your IT diet.   Engage stakeholders from Dev, Ops, customers and suppliers (if appropriate).  Discuss ways to improve the work with healthier habits.  Agree on what to do next or what not to do anymore.

    Every diet should be supported by recipes for success.  The upcoming DevOps Cookbook by Gene Kim, et al. will provide more tangible guidance and recipes for healthy IT habits.  More on that in future blogs.

    Most of all,  identifying and eliminating waste through Lean DevOps is a great opportunity for collaboration.  After all,  it’s always better to diet with a friend.