Category: Enterprise DevOps

  • Enterprising DevOps in Banking sector

    Enterprising DevOps in Banking sector

    “Shift left” is common in project life cycle management where the risks are mitigated early in the cycle by involving QA team early in the planning & development phase. In the  IT support systems, shift left empowers support staffs with enough tools, processes & knowledge base so that the issues get resolved at a point closest to the customer. The purpose of this process is to create a leaner organization and increase customer satisfaction. DevOps is a continuous shift left that helps in high velocity product delivery and enhance the quality of the deliverables.

    Why DevOps is crucial to Banking Sector?

    a.       Multi-channel delivery

    Traditionally banks have been offering products & services that were physically distributed through branches/ATMs in a brick-and-mortar fashion. But with the rapid increase in the use of smartphones & tablets, the banks are challenged to transform their distribution channels and trend towards drive-to-digital. The study from eMarketer.com shows the distribution of banking channels used by various US Internet users. The distribution of banking channels increases the need for multi-channel interactions, a consistent UX across different channels like mobile/internet/TV banking, ATMs etc and an integrated channel to offer a seamless user experience. This in turn requires a transformation in the way the applications are developed & deployed, while adhering to high level of security & compliance. Now considering that each of the banking product is managed by a separate division with its own operations team, offering an integrated delivery channel is extremely difficult.

    b.      Security updates and hot fixes

    It is not long since the outbreak of the Heart Bleed vulnerability. It not only impacted the websites but also the banking applications using OpenSSL technology. Hackers could impersonate the banking services and steal user credentials. Banking Applications using OpenSSL were patched. Similarly, Zero-day-attacks pose a serious risk to banking applications as the hackers take advantage of the time before the vulnerability is publicly announced and a patch is available & applied. In early Feb 2014, a zero-day-attack by name IE “Operations SnowMan” [ CVE-2014-0322] could steal banking credential if the user uses IE 9 or 10. DDoS (Distributed Denial-Of-Service) & Man in the browser (MitB) attacks are some of the other top vulnerabilities that can threaten the security of the banking applications. This urges the banks to test & release the hotfixes / patches at the earliest without breaking the existing functionalities and most often make live patches while the customers are still using the banking applications.

    c.       Regulatory and legislative pressures

    US Patriot Act (Providing Appropriate Tools Required to Intercept and Obstruct Terrorism Act of 2001) has changed the way the banks identify their potential customers. All US banks and other oversees banks that have their correspondent account in US need to get certified under this act. As stated here the banks need to have teams & processes in place to ensure that it complies with these regulatory requirements. This mandates a risk management framework and a reporting mechanism. This increases process overheads, which inturn slow down innovation and product delivery.

    DevOps to the rescue

    DevOps philosophy is similar to Toyota’s Kanban / JIT (Just-In-Time) production. Banks are familiar with Agile & XP processes for product development, ITIL/ITSM for Operational efficiency, yet they are not JIT production ready. Let us pick the two major aspects that gear up the banks to be DevOps ready.

    a.       Organizational Structure in Banks

    We all acknowledge the fact that there is a huge gap between R&D and IT Operations. But, bridging the gap between R&D & IT can be complex based on the magnitude of the size of the organization, geographical distribution, compliances and processes. In 2011, Citi bank initiated a multi-billion dollar project named “Project Rainbow” to consolidate its IT platforms across different product portfolios like mortgages, credit portfolios etc. The focus was to move away from product-centric strategy to a more customer centric strategy and improve communication between the teams. In similar terms, RBI [Reserve Bank of India] recommends aligning the IT Development, Quality Assurance & Operations. The above picture is a recommendation of the organizational structure by RBI. This greatly improves the collaboration within the teams and help them move towards a common goal.

    b.      Automation

    Banking business processes are typically split as back office, mid office and front office processes. Back office processes are those that do not require direct customer interaction but involve a lot of data processing. Typically, banks have around 300 – 500 back office automation processes, some of them are repeatable and highly structured. These back office processes like HR management, administrative tasks & IT operations are good candidates for optimization and cost reduction. devops Mid office processes provide the necessary decision support system. Analytical reports, collaboration tools that foster communication and process unstructured data fall under this category. IT systems monitoring and reporting automation can identify the systems usage and reduce IT sprawls. Front office process are customer facing and requires constant up to date information from the mid and back office tools for quick response times. When these front, mid and back office processes are automated and integrated, it greatly increases the velocity of product delivery and increase customer satisfaction.

    Success stories like Loyld, show that banks can reduce cost and operational complexity by automating server configurations and release automation. Automating software testing can greatly improve defect containment and testing efficiency. Thus the banks can make consistent software deliveries across multiple delivery channels, apply hot fixes quickly or accommodate new regulatory needs, once they are DevOps ready.

  • Ensuring security and managing risk in enterprise DevOps

    Ensuring security and managing risk in enterprise DevOps

    Today’s question in this recurring series comes from a developer in a typical large enterprise – risk averse, tightly controlled, highly secured – a major challenge to any DevOps transformation:

    Q. I work in a large government department that is tight on security. We have strict policies on separation of duties, can’t share users IDs or passwords, struggle to share code, can’t use AWS or Azure, audit everything, and so on. How can DevOps work in my environment?

    If DevOps is about collaboration and removing bottlenecks, then it must include security, from planning to coding to testing to operations; building, delivering, and running applications that are secure by design. Unfortunately, for many enterprises security is a major roadblock in the application delivery lifecycle.

    Yet security does not need to be a DevOps bottleneck. Indeed, these specific issues are solved problems. While I do not boast of a deep security expertise, I can address the solutions based on my experience (and some further reading – see below):

    Maintain separation of duties

    Large enterprises must maintain a separation of duties, to prevent (for example) a rogue developer from adding a back door to new code and putting it into production themselves, with no checks and balances. But staff can still have the access they need when they need it – and no more. When roles change, such as developers accessing the operations ‘flight deck,’ or operators being seconded to development, identity management and access controls can provide the right access automatically, as needed, audited and policy-compliant.

    Protect privileged user IDs and passwords

    For large enterprises it is never acceptable to share passwords, especially to privileged accounts like cloud control panel admins. However, privileged identity management allows staff to ‘check-out’ privileged IDs from a secure pool, use them as needed (e.g. for break-fix), then check them back in when they are done. As policy dictates, access can be subject to additional approvals, time limits, access limits, keystroke logging, session capture, etc. This allows on-demand access to sensitive IDs while maintaining stringent audit and control.

    Provide secure cross-team interfaces

    Part of DevOps is sharing – processes, skills, knowledge … but also reusing code and services (or microservices) through shared APIs, whether public or proprietary. This sharing accelerates delivery and reduces costs, but can also introduce risk. To prevent unintended consequences when sharing code and services, API security can enable sharing both internally and externally, but with controls like user authentication (e.g. OAuth), granular access controls, volume/rate limiting, blocking and more.

    Enable secure on-demand cloud access

    Most enterprise staff cannot just fire up an AWS or Azure instance on the corporate card. Even applying for ‘approved’ access can be a multi-week process. Instead, security pros can set up automated self-service provisioning to ‘known good’ or ‘enterprise-grade’ services, pre-approved by security, HR, finance, etc. This helps to maintain necessary audit and control while enabling DevOps staff to quickly and easily adopt cloud and other services.

    Use automation to maintain security and governance

    Automation is a powerful way to reduce risk, ensure compliance, and maintain governance. For example, release automation allows developers to deploy new code without having personal access to production systems (the tool gets the access, not the individual). It can also restrict access to such systems through ‘known-good processes,’ to ensure deployment complies with policy. Tracking who initiates deployment maintains that essential audit trail for governance and reporting. Test automation, data management, process automation, configuration management and other automation tools all provide similar security benefits throughout the application delivery lifecycle.

    Have a security culture

    Of course, there is more to secure DevOps than technology solutions. Just as DevOps requires a new culture of collaboration, good governance ultimately requires a culture of security. Security is everyone’s responsibility at every step of the service delivery lifecycle. Security is still a specialist role, so security pros should have a place in agile planning, standup and triage meetings, but good governance is everyone’s role. Dev, ops, test, QA and others all need to step to the security challenge; and security pros should share their requirements, knowledge, best practices and tools with the rest of the DevOps team.

    Further Reading

    In compiling this post, I leaned on many others who have helped to establish the concepts of DevOpsSec / SecDevOps and Rugged DevOps. A great starting point for more specialist detail on this topic is the SecDevOps content here on devopsy.kinsta.cloud, particularly the work of Rich Mogull (@rmogull) and Andrew Storms (@st0rmz). For an early intro, check the Gartner note, DevOpsSec: Creating the Agile Triangle and the seminal work of James Wickett (@wickett) on Rugged DevOps. For a more recent thought, try the Threatstack blog, and Edward Haletky (@texiwill) for a good end-to-end view. Finally, for a great resource specific to DevOps auditability, check the Gene Kim (@RealGeneKim) project, the DevOps Audit Defense Toolkit.

     

  • My Little Operations: DevOps is Magic

    My Little Operations: DevOps is Magic

    Bear with me, it will all make sense in the end.

     

    If you aren’t familiar with My Little Pony: Friendship is Magic (because you don’t have a child who watches it over and over and … well, over)  let me sum up for you:

    There’s a pony who knows something horrible is coming. She wants to stop it. She’s instead sent by management to make friends, which seems to her to be ridiculous in the face of a looming catastrophe. When that catastrophe happens, she and her new “friends” (her coworkers) set off to find the elements of harmony – the only thing that can save the world. But lo and behold, evil destroys the elements. Or so it appears. See, it turns out that the friends are the embodiment of the elements of harmony and through their friendship they create the most powerful element of them all: magic. Together, they are strong enough to defeat the evil and restore harmony to the data center. Err, I mean, to the pony world.

    If we take this and apply it to operations, you should start seeing some common themes. Primarily, you might note that, like magic, the ultimate goal of DevOps – agility – can only be “achieved” by the collaborative efforts of the other elements of operational harmony, namely predictability, repeatability, composability, consistency and stability. It isn’t a tangible “thing” you can buy, it doesn’t come in a can, it can’t be delivered by Amazon.

    These elements combine and, when practiced cooperatively, create a new element: agility. Which is a significant goal of DevOps operational harmonybased on a variety of surveys. Oh, it’s always stated a different way: we want to improve the time to market, increase frequency of deploys, reduce downtime due to errors, etc… but in general, these all fall under the need to enable the same level of “agility” in operations (and the network, but that’s a different discussion) as is present in application development today. Fast and frequent change to match application (and business) changes. Agility.

    But agility, like magic, can only be achieved through friendship (or as we’ll call it, collaboration). That’s because agility requires a variety of separate concerns (qualities) to come together in concert before it can be realized:

    Agility (DevOps)

    • Predictability (time to deploy, important for planning)
    • Repeatability (of process to other applications)
    • Composability (reuse of steps in other processes)
    • Consistency (important for maintainability and trust in process)
    • Stability ( of the overall infrastructure )

    That’s why it’s important not to look at DevOps as a set of tools. There are multiple facets to DevOps that can be enabled by tools and enhanced by tools and even monitored by tools, but DevOps is not the tools themselves any more than the elements of harmony were the gems the Little Ponies sought. The real magic of DevOps is the agility created by collaboratively working together to bring each of the elements into operations. Without stability, have you achieved agility? Or is it really chaos?

    If you can’t repeat the process (apply it to other applications) have you really achieved agility or just strictly codified a single deployment? If the process isn’t consistent, and gives different results have you realized an agile environment or anarchistic one?

    These elements are necessary to achieve real agility of operations, and those elements can only come about in a cooperative environment in which they all work together.

    DevOps, so it seems, is magic after all.

     

  • Encouraging a ‘big picture’ view of enterprise DevOps

    Encouraging a ‘big picture’ view of enterprise DevOps

    Welcome to the latest post in a series where I answer some of the many questions on how to approach an Enterprise DevOps transformation at scale.

    Human issues continue to pop up – not surprising, as people and culture are at the heart of DevOps. This next question asks how to encourage a ‘big picture’ view across a complex org:

    Q: Keeping an eye on the bigger picture is getting harder with larger scale operations. Do you have any tips to feel each person in the team responsible for this?

    In larger enterprises IT is often positioned as a factory that services the whole business. Software is created to meet specifications, often with no real connection to larger business goals. Fixes are prioritized by development, releases are controlled by operations, and the business units are expected to be happy with whatever they are given. Specialists get tunnel vision, and barely understand their impact on business goals like “accelerate customer acquisition” and “penetrate new markets.”

    The ‘generalist vs. specialist’ question from my previous Q&A (along with the ensuing debate on LinkedIn) keenly informs this challenge. Ultimately the type of people in your org – specialists or generalists, system thinkers or silo thinkers, I-shaped or T-shaped, goats or sheep – will have a huge impact on how well they collaborate and share responsibilities. But here are some ideas on how to encourage a shared responsibility for the ‘big picture’ of IT service delivery.

    Take a refresher course on corporate strategy

    Start by ensuring everyone, on every team, knows the corporate objectives. IT staff often don’t connect with corporate goals simply because they don’t know them. For large public enterprises, the annual report and/or recent SEC filings are a great place to start. Listen to earnings calls, read press releases, and follow other executive messages. Understanding corporate goals as a team is critical to sharing responsibility for the bigger picture.

    (Re-)Discover your business unit objectives and contributions

    Invite the business unit(s) you support to present to IT their role in the business and their objectives. Ask them to describe how their goals align with corporate strategy, how they contribute to corporate goals, and (if they can) how IT helps them achieve these goals. Then workshop across teams on how every role in the IT value chain contributes to specific business outcomes. This will help to establish each team member’s shared responsibility for those objectives, and their responsibility for each other.

    Formalize shared goals and measure results against them

    Reflect shared responsibilities by establishing and measuring formal performance objectives for individual, team, business and corporate goals. For example, make operators responsible for infrastructure stability and on-time release; make developers responsible for code quality and break-fix; make DBAs responsible for schema design and transaction throughput; make everyone responsible for speed, quality, cost and security. Make individuals accountable to each other, their business, and the company through shared goals and metrics.

    Bring people together with shared process

    Map out the application delivery workflow so everyone can see clearly the shared responsibilities in an end-to-end process that contributes to business objectives. Do not make these process maps rigid; rather, encourage everyone to continuously look for improvements to the process flow. Make them highly visible too, to remind everyone of their shared responsibility in the end-to-end service delivery process, rather than as a reference guide to be slavishly followed.

    Bring people together with shared technology

    Provide shared technology to help establish common language, foster communication, and streamline team interactions. Look for scalable, multi-site, multi-user capabilities for automation and management in a shared toolchain. Adopt common social software to enable fast and easy communications among and across teams. Use common tools, screens, and reports to monitor performance against key individual, team, business, and corporate metrics.

    Align staff intentionally with key business processes

    Where possible align end-to-end (or ‘full stack’) IT teams to business areas rather than grouping specialist teams to service the whole company. Closer alignment helps build a connection between the business and service delivery teams, with a shared responsibility for business outcomes. This ‘artisan’ approach can be hard to enact in a ‘factory’ structure, but it can be adopted incrementally (by geo, by BU, by product line, by project) to make it easier.

    More ideas?

    These are a start, but as with all my tips and answers, definitely not the final word. Please feel free to add your ideas in the comments on how to foster a shared feeling of responsibility for the big picture in a large, complex environment. Or you can tweet your comments to @AndiMann. Or take them to your favorite social site and discuss this article there. I’d love to see any other ideas you have, or chat with you about these ideas wherever you log in.

  • Provisioning versus Configuration Example

    Provisioning versus Configuration Example

    A few weeks back I wrote about the difference between configuration and provisioning noting, primarily, that there are differences between the two tasks. It remains an important distinction to make because it’s really where the rubber meets the road (or the app meets the network) where it becomes important. As the infrastructure/network side of the house is where we’re concentrating our efforts to apply DevOps, it’s necessary to dig a bit deeper on this topic.

    So today we’ll examine  couple of concrete examples of the difference – and why it’s important.

    Your application is being deployed and it needs, wonder of wonders, a load balancer. Cause, elasticity. Scale. You know, cloudy type stuffs.

    So you grab that golden image and launch that virtual machine with Whatever Your Favorite Load Balancing Thing might be (HA Proxy, Nginx, F5, etc…). Voila! You’re ready to go, right?

    Not by a long shot. You’ve provisioned a load balancing service, yes. But is it configured? Probably not. In addition to the basic networking needed (IP addresses, VLANs, etc…) there are also a whole bunch of very application specific things that must be configured. There’s no “golden” load balancing service that can be generically applied to your application. Period. At a minimum the virtual server (that represents your application to the rest of the world) needs two things: an IP address and a pool of app instances to select from.

    simplified-lb

     

    That means there’s some configuration that must occur after the provisioning. There’s no single LB image that will work for all applications, period. Even if all other settings are the same (LB algorithm, L7 app routing rules, TCP options, etc..) you still have to tell it which pool of real servers to select from and what IP address to use so clients can connect.

    This is true for pretty much all application services (infrastructure) that support an application deployment. Because there’s no single golden image that just “works” for everything and no way to use dynamic configuration options for everything (DHCP for IP addresses is one thing, but there’s no “dynamic load balancing algorithm selection” service out there yet) there remain a set of options that must be configured after provisioning occurs. Consider application security like a web app firewall (WAF). It has to be configured to protect a specific application. It has to learn the URIs and the data expected to be exchanged so it recognizes when some interaction is out of bounds and potentially dangerous. That’s unique to the application, and thus it requires some specific configuration after provisioning.

    It’s important to note this distinction lest we get caught up in believing that we can simply create a golden image of all our network infrastructure and services and put them into production with the touch of a button.

    This little reality also helps to understand why it doesn’t really matter whether the network service in question is launched on custom hardware or some server off the shelf. The bulk of the work when automating the deployment of network services is in the configuration, not the provisioning.

    While you can minimize the amount of post-provisioning configuration necessary, you can’t eliminate it entirely. That’s why API-enabled, programmable infrastructure is a critical piece of the larger DevOps picture. It is the API, programmable templates and similar capabilities that enable the configuration to be automated and orchestrated so as to not be the bottleneck in the deployment process.

  • Being Agile is all about controlling the flow

    Being Agile is all about controlling the flow

    For businesses, Agile is the new fashionable initiative as was E-business was in the 2000’s, Six Sigma was in the 90’s and Total Quality Management (TQM) was in the 80’s. If we look back across the spectrum of these initiatives, we need to examine how many of these initiatives met the goals the business was looking for and, more importantly, if they were fully adopted, remembering that many were derailed, did they deliver the value the business was seeking?

    Of course we cannot answer this question with generalities—it’s a spectrum. For some businesses, these initiatives were red herrings, some did not follow through to completion, but certain changes adopted as part of these initiatives had a very positive impact, while others succeeded wildly. It’s a Harvard Business Review case study as to why a business ended up at the point on the spectrum that they did.

    Clearly, one fundamental aspect that is consistent across all these initiatives is the ability for a business to embrace change. In a previous blog, “Cloud Computing, DevOps and Business Velocity,” I addressed the issue of business velocity—the rate at which an organization can embrace a particular disruptive technology change. Underestimating your business’ ability to adopt a new technology or process will most likely have a strong negative impact on the outcome. In a recent conversation with a client, the non-IT business areas has been very successful at adopting lean practices, but IT short-circuited shortly after starting. It didn’t outright fail, but after a strong start, they took a break from which they never started back up. Without a post-mortem, we don’t know why this specific initiative fell flat, but we can intimate that if something is difficult and feels unnatural, we’re more easily able to focus our attention elsewhere—think dieting and exercise programs in your own life.

    This seems to be the decade of Agility. Businesses want to be more dynamic and fluid. They want the organization to be prepared to respond to market changes more quickly, introduce new products faster, and respond to problems that occur more rapidly. To accomplish this goal, the business will have to let go of some of the ways that they do things today. They will have to re-engineer processes to be more compose-able and componentized so that they are less static. For example, a manufacturer goes through a lot of planning to deliver a new product to market. They go through steps for design, prototype, test, source, manufacture and distribute. The entire cycle may take upwards of a year and is highly automated once it reaches the manufacture stage. Tweaking the product between batches would cause  significant delays and pain because the process is static. It effectively amounts to restarting the entire cycle. So, what if they discovered that by simply elongating the body by 2 cm the product was far more useful their intended audience? In this case, agility might equate to being able to complete the design-to-manufacture process faster.

    In the previous example, this latter step of being able to tweak the product is called continuous improvement and is a key element of fostering agility. If a product isn’t selling well in the market, many companies will just abandon the product completely due to the costs associated with making the product viable. The reason being that most manufacturers consider that each product is a project and when it reaches the manufacturing stage it moves under the control of operations. That is, the creative team that includes marketing, finance, and other stakeholders view their position as “completed” with regard to getting that product to market and they then move onto other projects. With this mindset, there’s no room for the group to review the process, think about how it could have been done better and then iterate with the improvements incorporated. Hence, they just keep taking their “this is the way we do it” to the next project.

    Being agile means being able to control the flow. There’s no way to change how products come to market if there’s a culture of accepting finality once a product has reached a certain point in the cycle.

    The other aspect of controlling the flow is managing how products and services enter and exit the development pipeline.  I spend a lot of my time helping IT organizations unlock their potential to deliver by helping them to understand that there’s only so much that can be done to improve managing the work-in-progress (WIP). The only way to make gains beyond a certain point is to enable transparency with those that are responsible for inserting work into the pipeline or who are demanding that work completes and exits the pipeline at a certain time.

    The following two figures express well the reason transparency is a requirement. I believe most people would agree that managing production is a far easier task in Figure 1 than Figure 2. In Figure 1, there is one input and one expected output. Managing WIP for this pipeline is a matter of optimizing just the process for completing that one deliverable. In Figure 2, there are many more variables at work. There could be contention for resources within the pipeline to support the multiple inputs.

    wip

     

     

     

    wip2

     

    Likewise, there may be a request to reprioritize what gets completed when to meet certain business goals. Perhaps a particular report is required urgently to support a key business decision. I personally dealt with this very scenario when the business I was working for years ago was suddenly engaging in acquiring another company. All focus was transferred to producing the needed financial statements and everything else was put on the backlog. In that particular case, the CEO and CFO were the customer, so no one questioned the delays to their projects, but for many IT organizations reprioritization requests are never that clear making it very difficult to manage the control of the flow. Hence, transparency and communication between those consuming the services of the pipeline and those who own the control of the flow of the pipeline is an imperative for agility.

    Like TQM, Six Sigma, and E-Business, Agility will require changes to the fundamental way that a business operates. An analogy of this process can be as benign as a person recovering from knee surgery or as complex as recovering from a stroke depending upon what the business wants to accomplish and how they are organized currently. Along the way, people will most likely be the biggest hurdles to success.

    When change of this magnitude occurs, employees tend to become very myopic of their own situation. To mitigate this risk, it is critical that changes are reviewed with employees prior to change, during implementation and after the change is enacted. Management must gauge if employees are struggling because the complexity of the change or because they personally don’t agree with the change. The former can be overcome with education and support while the latter should be dealt with quickly through removal.

    Businesses can succeed with their Agility initiatives given they identify and mitigate key risk factors prior to starting and are willing to commit to looking critically at the way they have been operating the business. In agile software development, developers are trained to think about ways their code can break and then write unit tests to ensure that the software is sustainable in light of these possibilities. Creating an agile business means looking at ways to ensure sustainability in light of market changes, new competitors, supplier changes, and changing customer demands.

  • Specialists vs. Generalists for Enterprise DevOps

    Specialists vs. Generalists for Enterprise DevOps

    Welcome back to the devopsy.kinsta.cloud series, ‘Enterprise DevOps Q&A.’ I have a backlog of Q&A in store, but don’t let that stop you sending in more! If I can’t answer them myself, I will try to find someone who can.

    For this post, I was recently asked about the challenge that large enterprises are facing over whether to use generalists or specialists to drive their DevOps transformation…

    Q. Does a DevOps environment prefer generalists or specialists to succeed?

    This question has ricocheted for a while. Google CIO Ben Fried – who spent 14 years in IT at a large enterprise (Morgan Stanley) – spoke as long ago as 2011 about how ‘Generalists, Not Specialists, Will Scale the Web.’ Some webscale businesses like Netflix even espouse a singular ‘NoOps‘ skillset, although they continued to have separate specialists … and maybe still do.

    However, in large enterprises, I have seen DevOps work well with both specialists and generalists.

    The Case for Generalists

    A generalist who can take on more duties from design to deployment is especially productive when application alignment moves to a local or vertical approach, as I have discussed before. Such generalists can move code through the SDLC faster, with greater integrity and fewer opportunities to drop the ball. In such structures, there are fewer handoffs from ideation to production, so there is less chance something will slip through the cracks.

    Local and vertical organizational structures tend to be less complex, with fewer products, fewer services, fewer applications, and fewer technologies involved, so such teams can better emulate smaller startup-sized organizations. DevOps generalists in these structures, for example, will not have to learn the vagaries of 5 different provisioning tools, because they all use just one tool. Generalists will often do better in such structures.

    The Case for Specialists

    On the other hand, in a large enterprise you might have not only 5 different provisioning tools, but also 4 test automation tools, 3 middleware systems, 5 databases, and much more – housed in a dozen different departments! It is perhaps impossible then for a generalist to have the expertise required to be proficient in all of these processes and technologies. As anecdotal evidence, look at any web site or mobile app ‘designed’ by a developer – a good proportion of them are almost unusable.

    In the enterprise even specialists like DBAs or security admins need to specialize within their field. An Oracle DBA, for example, has specialized knowledge that a Hadoop DBA does not. Big enterprises adopting big DevOps need specialists who know specific differences between DB2, Oracle, MySQL, and Hadoop. And as I have posted before, it would be nice to standardize on one, but such a dogmatic approach is simply unrealistic.

    The Case for ‘T-Shaped’ People

    That said, as with many conventions in Enterprise DevOps, a best practice seems to be in finding balance. Build up leaders who can truly collaborate across disciplines with experts in other areas, while still retaining a depth of skills and expertise in their specific field. This is the idea of ‘T-shaped’ people – aka ‘Generalizing Specialists’ or ‘Master Generalists’.

    This seeming oxymorons are actually useful archetypes to apply to DevOps staffing, especially when supported by expertise from specialists from the PMO, architecture, development, DBA, security, testing, QA, pre-prod, and operations teams. This is the best of both words – generalists when you want them; specialists when you need them.

    The Case for Pragmatism

    Perhaps more importantly though, as Donald Rumsfeld once famously noted, “You go to war with the Army you have … not the Army you might want.” In a large enterprise the staff you have will likely be specialists, regardless of the staff you might want, so this will be your starting point.

    Fortunately, building a collaborative cross-functional team from separate groups of specialists, rather than a single DevOps department, is as much of a DevOps best practice as seems to exist. Take for example a large enterprise like National Australia Bank, which has have taken a DevOps approach that started with specialists but ultimately resulted in a ‘DevOps Team.’ On the other hand, organizations like DirecTV and Union Bank started their DevOps journeys by helping dev, test, and QA to streamline within their specialization, as much as they looked to streamline the collaboration between teams of specialists.

    The Case for Another Case?

    Did you handle your DevOps staffing differently? Is there another way to reconcile the differences between generalists and specialists? Please feel free to add to the discussion in the comments, I would love to hear more about your approach to enterprise DevOps staffing.

    And as always, if you have questions you’d like to address about Enterprise DevOps, let me know. Leave them in the comments below; hit me up on Twitter at @AndiMann; or send me an e-mail to me at andi.mann@ca.com.

  • Making DevOps Work in Complex Enterprise Environments

    Making DevOps Work in Complex Enterprise Environments

    Welcome back to the devopsy.kinsta.cloud series, ‘Enterprise DevOps Q&A’. Remember, if you have specific questions about Enterprise DevOps, let me know and I will try to answer them for you.

    Today’s question is:

    Q. In large enterprises, there are always environmental differences. From public cloud to internal hosting, from virtual desktops to legacy and mainframe – it is not possible to have full consistency between environments. Do you have any suggestions for approaching this gap?

    The Challenge of Enterprise Complexity

    This is a huge problem in larger enterprises, especially as DevOps practices grow beyond just isolated departmental applications. Architects have one system for prototyping; developers have a different system to code on; test and QA have their own unique ‘standard’ systems; while operations have untouchable systems for production.

    Add to this the complexity of backending into CICS or IMS applications, accessing data from DB2 or ADABAS on iSeries, or automating WebSphere deployment onto HP/UX; plus licensing for an Oracle database on AWS or an SAP client on Azure; and meeting compliance requirements for data privacy and operational security. This enterprise complexity and environmental inconsistency so often means new code is not properly tested until it hits production, resulting in unacceptable bug counts and an average annual downtime cost of almost $700K per incident!

    No Snowflakes

    Many will say that you cannot perpetuate ‘unique snowflakes’, so you must standardize environments. This dogma is easy to preach – and in principle is a best practice – but in reality it is too simplistic for most enterprises. Disparate environments often exist for good reasons (e.g. to accommodate cost, skillset, throughput, integration, or other technical requirements). In any case, you cannot simply run roughshod over established architectures. So standardize if and when you can, but realize that this path is unlikely to solve the problem.

    Abstract Platforms

    When you cannot fully standardize, you can still approach consistency by abstracting unique platforms. For example:

    Automate Complexity

    Automation can also help manage enterprise complexity to lessen the impact of environmental inconsistency. For example, solutions for enterprise-grade configuration automation or release automation can both streamline the handoff between dev and ops teams, where environmental inconsistency causes significant issues. Or adopting test automation technologies like service virtualization that can streamline development, test, QA, and other use pre-prod use cases, resulting in

    “shorter test times, increased productivity, and better production quality” by enabling complex environments to be consistently replicated on demand.

    Tool Selection Matters

    Be careful though. DevOps is not all about tools, so automation is not a complete answer; and for enterprises in particular, tool selection is harder too. For more complex enterprise environments, you will likely need to look beyond the trendy open source/webscale/startup tools. For example:

    •  Chef and Salt both claim support for Linux, Windows, and Solaris, but not AIX or HP/UX
    • Docker claims support for Linux and Windows desktops, but not for UNIX or Windows Server
    • Ansible only documents support for Windows control machines in a recent beta version
    • Most cloud services only support x86 architectures, and then only Windows or Linux
    • There is limited specific support for Mobile DevOps beyond Jenkins and some blogware
    • Only traditional large vendors support traditional large systems like IBM i or System z

    And, it must be noted, many ‘enterprise’ tools don’t support newer environments like MacOS, Azure, or Google; or don’t integrate well with a diverse toolchain.

    So when selecting automation and other solutions for your DevOps toolchain, make sure they support your environment, from mobile to mainframe, whether on-premises, hosted, or in the cloud.

    What is Your Solution?

    These are some solutions for the consistency gaps large enterprises are facing, but are just scratching the surface. I would love to hear your solutions for handling enterprise complexity within a DevOps approach. Or if you have any Enterprise DevOps questions, let me know and I will try to answer them. You can leave your ideas or questions in the comments below; hit me up on Twitter at @AndiMann; or send me an e-mail to me at andi.mann@ca.com.

  • Keep Calm and Refactor Operations

    Keep Calm and Refactor Operations

    Developers have been refactoring forever to gain efficiencies and avoid mistakes. Let’s apply more than just theory to ops to do the same.

    keep-calm-and-refactor-operatorsThere’s about a million articles, blogs, tweets and infographics available today that bespeak of DevOps as this or that or the other thing. But one core truth remains: DevOps in part attempts to apply methodologies like agile to operations in order to gain efficiencies and avoid mistakes. We need the efficiencies because the number of apps that need to be deployed, updated, migrated, and maintained is increasing, not decreasing, and there simply isn’t enough time in the day (or during the monthly change window) to get it all done.

    We need to eliminate mistakes because they are, according to data from initialstate, very costly. It takes 30x longer to fix one bug (an error) than it does to write one line of code. Anyone who has ever spent time chasing a wild pointer will immediately say that’s too low an estimate, but anecdotally I’m willing to go with that figure. It is an average, after all. To put that in perspective, the same data indicates that developers must find and fix 100 product bugs per 1000 lines of code. That’s a 10% error rate.

    I’ll let you ponder for a moment what that means to DevOps and the desire to treat infrastructure as code.

     

     

    Exactly. You probably want to avoid that kind of error rate as you go about automating and orchestrating provisioning, configuration and general management of all that infrastructure. Just a few errors can effectively negate time gained by automation. So how do you avoid that?

    One of the ways to avoid some of that is refactoring.

    Refactoring is most commonly used to extract pieces of code (script) that repeat over and over into composable functions (in a procedural language) or methods (in an object-oriented language) or services (in a micro-service or SOA-based system). It’s a primitive form of abstraction that serves to produce more manageable, extensible code bases.

    Basically, you’re taking a set of actions that you repeatedly perform and turning them into a reusable, repeatable “thing”. In the case of operations that “thing” is probably a script.

    Say, for example, that every time you deploy an application you’re also deploying a load balancing service because, of course, you’ll want it scalable and/or reliable. You always perform the same set of steps to complete this task with only a few changes – usually IP addresses, host names, or other network and app-specific settings.

    But you’re doing it manually, or writing a new script every time. Manually making changes are, in most cases, where errors are introduced. Fat fingers are a common cause, especially when you’re not paying close attention because, well, boring! You’ve done this fifty million times and quite frankly, you’re sick to death of the tediousness of it all. refactor all the things

    We’ve all done it, don’t pretend you haven’t both thought it was boring and made a mistake. You’re in good company, trust me.

    What developers do when this is the case is refactor the common code and reuse it across the system. Those bits that get changed are parameterized and everything else stays the same. Even DBAs do this, they just call the resulting code a stored procedure.

    The result of such efforts are self-contained automated tasks that, when used to orchestrate a process, enable composable operations. You string together a set of these scripts and voila! Provisioning complete, minus the fat fingered errors or missed manual steps.

    Refactoring code is much more complex than this, of course, as would also be the case within operations, but the general premise here is that the practical application of DevOps includes the process of refactoring operations. It’s about restructuring it into more composable, logical and discrete tasks that can be automated and orchestrated as a means to achieve more manageable and extensible operations. It’s about identifying those steps in a process that are repeatable and can be refactored into a single, shared executable task.

    When starting an initiative that applies DevOps it is likely you’re going to have to prove the time you’re taking is worth the investment. One of the ways to do that is by refactoring those operations that are continually used and reused to enable a more consistent, predictable and repeatable provisioning and deployment process. By choosing those steps that are most often repeated and refactoring them into their own composable scripts, you can gain efficiencies as well as reduce errors (and eliminate the time they require to find and fix).

  • Culture Debt

    Culture Debt

    There has been a lot of talk lately about IT’s accumulation of “technical debt’.  Technical debt describes necessary work from a prior deployment that was deferred in favor of other tasks.   As technical debt grows, the ability to make future changes is hindered because of a backlog that was not addressed.

     

    I have recently come to recognize that IT suffers from another type of debt – culture debt.  IT is a young industry that grew rapidly  while facing constant pressure to bring technical innovation to the market. Culture considerations were deferred  in favor of building and deploying products and services.  IT’s silo culture grew organically out of the need for diversifed sets of specialized experience and expertise.

     

    While cultural debt was accumulating,  IT’s complexities were increasing.  The single platform mainframe vanished in favor of multi-platform servers.   Production applications grew exponentially. The IT supply chain went from a single location department  to a network that spans multiple geographies and organizations. Although IT’s silos were also acknowledged to be  IT’s constraints, converging different processes, frameworks,  vocabularies and customs was not going to be easy.  And so the cultural debt grew.

     

    Like all debts, payment is eventually due.  For IT, the due date is  today. Cultural debt is having a tangible impact on the bottom line by hindering IT’s ability to meet the pace of deployment that the business now requires. Every bottleneck in the workflow between Dev and Ops affects the entire system of supply and demand.  The business has zero tolerance for missed deadlines, poor quality code or fragile applications.  A paradigm shift is required in order to improve IT’s consistency and speed.

     

    The good news is that an increasing number of individuals and organizations are looking to pay down cultural debt by embracing a DevOps approach. Case studies are emerging that demonstrate proof of concept – silos are breaking down and people are talking to each other. Tools are enabling automated tasks and deployments are happening faster with fewer defects.  Dev and Ops are starting to embrace a common set of practices and accountabilities.  Best of all, DevOps is stemming the rate of future technical and cultural debt  – and that’s a win for everyone.