devopsy.kinsta.cloud

Author: Andi Mann

  • Ensuring security and managing risk in enterprise DevOps

    Ensuring security and managing risk in enterprise DevOps

    Today’s question in this recurring series comes from a developer in a typical large enterprise – risk averse, tightly controlled, highly secured – a major challenge to any DevOps transformation:

    Q. I work in a large government department that is tight on security. We have strict policies on separation of duties, can’t share users IDs or passwords, struggle to share code, can’t use AWS or Azure, audit everything, and so on. How can DevOps work in my environment?

    If DevOps is about collaboration and removing bottlenecks, then it must include security, from planning to coding to testing to operations; building, delivering, and running applications that are secure by design. Unfortunately, for many enterprises security is a major roadblock in the application delivery lifecycle.

    Yet security does not need to be a DevOps bottleneck. Indeed, these specific issues are solved problems. While I do not boast of a deep security expertise, I can address the solutions based on my experience (and some further reading – see below):

    Maintain separation of duties

    Large enterprises must maintain a separation of duties, to prevent (for example) a rogue developer from adding a back door to new code and putting it into production themselves, with no checks and balances. But staff can still have the access they need when they need it – and no more. When roles change, such as developers accessing the operations ‘flight deck,’ or operators being seconded to development, identity management and access controls can provide the right access automatically, as needed, audited and policy-compliant.

    Protect privileged user IDs and passwords

    For large enterprises it is never acceptable to share passwords, especially to privileged accounts like cloud control panel admins. However, privileged identity management allows staff to ‘check-out’ privileged IDs from a secure pool, use them as needed (e.g. for break-fix), then check them back in when they are done. As policy dictates, access can be subject to additional approvals, time limits, access limits, keystroke logging, session capture, etc. This allows on-demand access to sensitive IDs while maintaining stringent audit and control.

    Provide secure cross-team interfaces

    Part of DevOps is sharing – processes, skills, knowledge … but also reusing code and services (or microservices) through shared APIs, whether public or proprietary. This sharing accelerates delivery and reduces costs, but can also introduce risk. To prevent unintended consequences when sharing code and services, API security can enable sharing both internally and externally, but with controls like user authentication (e.g. OAuth), granular access controls, volume/rate limiting, blocking and more.

    Enable secure on-demand cloud access

    Most enterprise staff cannot just fire up an AWS or Azure instance on the corporate card. Even applying for ‘approved’ access can be a multi-week process. Instead, security pros can set up automated self-service provisioning to ‘known good’ or ‘enterprise-grade’ services, pre-approved by security, HR, finance, etc. This helps to maintain necessary audit and control while enabling DevOps staff to quickly and easily adopt cloud and other services.

    Use automation to maintain security and governance

    Automation is a powerful way to reduce risk, ensure compliance, and maintain governance. For example, release automation allows developers to deploy new code without having personal access to production systems (the tool gets the access, not the individual). It can also restrict access to such systems through ‘known-good processes,’ to ensure deployment complies with policy. Tracking who initiates deployment maintains that essential audit trail for governance and reporting. Test automation, data management, process automation, configuration management and other automation tools all provide similar security benefits throughout the application delivery lifecycle.

    Have a security culture

    Of course, there is more to secure DevOps than technology solutions. Just as DevOps requires a new culture of collaboration, good governance ultimately requires a culture of security. Security is everyone’s responsibility at every step of the service delivery lifecycle. Security is still a specialist role, so security pros should have a place in agile planning, standup and triage meetings, but good governance is everyone’s role. Dev, ops, test, QA and others all need to step to the security challenge; and security pros should share their requirements, knowledge, best practices and tools with the rest of the DevOps team.

    Further Reading

    In compiling this post, I leaned on many others who have helped to establish the concepts of DevOpsSec / SecDevOps and Rugged DevOps. A great starting point for more specialist detail on this topic is the SecDevOps content here on devopsy.kinsta.cloud, particularly the work of Rich Mogull (@rmogull) and Andrew Storms (@st0rmz). For an early intro, check the Gartner note, DevOpsSec: Creating the Agile Triangle and the seminal work of James Wickett (@wickett) on Rugged DevOps. For a more recent thought, try the Threatstack blog, and Edward Haletky (@texiwill) for a good end-to-end view. Finally, for a great resource specific to DevOps auditability, check the Gene Kim (@RealGeneKim) project, the DevOps Audit Defense Toolkit.

     

  • Encouraging a ‘big picture’ view of enterprise DevOps

    Encouraging a ‘big picture’ view of enterprise DevOps

    Welcome to the latest post in a series where I answer some of the many questions on how to approach an Enterprise DevOps transformation at scale.

    Human issues continue to pop up – not surprising, as people and culture are at the heart of DevOps. This next question asks how to encourage a ‘big picture’ view across a complex org:

    Q: Keeping an eye on the bigger picture is getting harder with larger scale operations. Do you have any tips to feel each person in the team responsible for this?

    In larger enterprises IT is often positioned as a factory that services the whole business. Software is created to meet specifications, often with no real connection to larger business goals. Fixes are prioritized by development, releases are controlled by operations, and the business units are expected to be happy with whatever they are given. Specialists get tunnel vision, and barely understand their impact on business goals like “accelerate customer acquisition” and “penetrate new markets.”

    The ‘generalist vs. specialist’ question from my previous Q&A (along with the ensuing debate on LinkedIn) keenly informs this challenge. Ultimately the type of people in your org – specialists or generalists, system thinkers or silo thinkers, I-shaped or T-shaped, goats or sheep – will have a huge impact on how well they collaborate and share responsibilities. But here are some ideas on how to encourage a shared responsibility for the ‘big picture’ of IT service delivery.

    Take a refresher course on corporate strategy

    Start by ensuring everyone, on every team, knows the corporate objectives. IT staff often don’t connect with corporate goals simply because they don’t know them. For large public enterprises, the annual report and/or recent SEC filings are a great place to start. Listen to earnings calls, read press releases, and follow other executive messages. Understanding corporate goals as a team is critical to sharing responsibility for the bigger picture.

    (Re-)Discover your business unit objectives and contributions

    Invite the business unit(s) you support to present to IT their role in the business and their objectives. Ask them to describe how their goals align with corporate strategy, how they contribute to corporate goals, and (if they can) how IT helps them achieve these goals. Then workshop across teams on how every role in the IT value chain contributes to specific business outcomes. This will help to establish each team member’s shared responsibility for those objectives, and their responsibility for each other.

    Formalize shared goals and measure results against them

    Reflect shared responsibilities by establishing and measuring formal performance objectives for individual, team, business and corporate goals. For example, make operators responsible for infrastructure stability and on-time release; make developers responsible for code quality and break-fix; make DBAs responsible for schema design and transaction throughput; make everyone responsible for speed, quality, cost and security. Make individuals accountable to each other, their business, and the company through shared goals and metrics.

    Bring people together with shared process

    Map out the application delivery workflow so everyone can see clearly the shared responsibilities in an end-to-end process that contributes to business objectives. Do not make these process maps rigid; rather, encourage everyone to continuously look for improvements to the process flow. Make them highly visible too, to remind everyone of their shared responsibility in the end-to-end service delivery process, rather than as a reference guide to be slavishly followed.

    Bring people together with shared technology

    Provide shared technology to help establish common language, foster communication, and streamline team interactions. Look for scalable, multi-site, multi-user capabilities for automation and management in a shared toolchain. Adopt common social software to enable fast and easy communications among and across teams. Use common tools, screens, and reports to monitor performance against key individual, team, business, and corporate metrics.

    Align staff intentionally with key business processes

    Where possible align end-to-end (or ‘full stack’) IT teams to business areas rather than grouping specialist teams to service the whole company. Closer alignment helps build a connection between the business and service delivery teams, with a shared responsibility for business outcomes. This ‘artisan’ approach can be hard to enact in a ‘factory’ structure, but it can be adopted incrementally (by geo, by BU, by product line, by project) to make it easier.

    More ideas?

    These are a start, but as with all my tips and answers, definitely not the final word. Please feel free to add your ideas in the comments on how to foster a shared feeling of responsibility for the big picture in a large, complex environment. Or you can tweet your comments to @AndiMann. Or take them to your favorite social site and discuss this article there. I’d love to see any other ideas you have, or chat with you about these ideas wherever you log in.

  • Specialists vs. Generalists for Enterprise DevOps

    Specialists vs. Generalists for Enterprise DevOps

    Welcome back to the devopsy.kinsta.cloud series, ‘Enterprise DevOps Q&A.’ I have a backlog of Q&A in store, but don’t let that stop you sending in more! If I can’t answer them myself, I will try to find someone who can.

    For this post, I was recently asked about the challenge that large enterprises are facing over whether to use generalists or specialists to drive their DevOps transformation…

    Q. Does a DevOps environment prefer generalists or specialists to succeed?

    This question has ricocheted for a while. Google CIO Ben Fried – who spent 14 years in IT at a large enterprise (Morgan Stanley) – spoke as long ago as 2011 about how ‘Generalists, Not Specialists, Will Scale the Web.’ Some webscale businesses like Netflix even espouse a singular ‘NoOps‘ skillset, although they continued to have separate specialists … and maybe still do.

    However, in large enterprises, I have seen DevOps work well with both specialists and generalists.

    The Case for Generalists

    A generalist who can take on more duties from design to deployment is especially productive when application alignment moves to a local or vertical approach, as I have discussed before. Such generalists can move code through the SDLC faster, with greater integrity and fewer opportunities to drop the ball. In such structures, there are fewer handoffs from ideation to production, so there is less chance something will slip through the cracks.

    Local and vertical organizational structures tend to be less complex, with fewer products, fewer services, fewer applications, and fewer technologies involved, so such teams can better emulate smaller startup-sized organizations. DevOps generalists in these structures, for example, will not have to learn the vagaries of 5 different provisioning tools, because they all use just one tool. Generalists will often do better in such structures.

    The Case for Specialists

    On the other hand, in a large enterprise you might have not only 5 different provisioning tools, but also 4 test automation tools, 3 middleware systems, 5 databases, and much more – housed in a dozen different departments! It is perhaps impossible then for a generalist to have the expertise required to be proficient in all of these processes and technologies. As anecdotal evidence, look at any web site or mobile app ‘designed’ by a developer – a good proportion of them are almost unusable.

    In the enterprise even specialists like DBAs or security admins need to specialize within their field. An Oracle DBA, for example, has specialized knowledge that a Hadoop DBA does not. Big enterprises adopting big DevOps need specialists who know specific differences between DB2, Oracle, MySQL, and Hadoop. And as I have posted before, it would be nice to standardize on one, but such a dogmatic approach is simply unrealistic.

    The Case for ‘T-Shaped’ People

    That said, as with many conventions in Enterprise DevOps, a best practice seems to be in finding balance. Build up leaders who can truly collaborate across disciplines with experts in other areas, while still retaining a depth of skills and expertise in their specific field. This is the idea of ‘T-shaped’ people – aka ‘Generalizing Specialists’ or ‘Master Generalists’.

    This seeming oxymorons are actually useful archetypes to apply to DevOps staffing, especially when supported by expertise from specialists from the PMO, architecture, development, DBA, security, testing, QA, pre-prod, and operations teams. This is the best of both words – generalists when you want them; specialists when you need them.

    The Case for Pragmatism

    Perhaps more importantly though, as Donald Rumsfeld once famously noted, “You go to war with the Army you have … not the Army you might want.” In a large enterprise the staff you have will likely be specialists, regardless of the staff you might want, so this will be your starting point.

    Fortunately, building a collaborative cross-functional team from separate groups of specialists, rather than a single DevOps department, is as much of a DevOps best practice as seems to exist. Take for example a large enterprise like National Australia Bank, which has have taken a DevOps approach that started with specialists but ultimately resulted in a ‘DevOps Team.’ On the other hand, organizations like DirecTV and Union Bank started their DevOps journeys by helping dev, test, and QA to streamline within their specialization, as much as they looked to streamline the collaboration between teams of specialists.

    The Case for Another Case?

    Did you handle your DevOps staffing differently? Is there another way to reconcile the differences between generalists and specialists? Please feel free to add to the discussion in the comments, I would love to hear more about your approach to enterprise DevOps staffing.

    And as always, if you have questions you’d like to address about Enterprise DevOps, let me know. Leave them in the comments below; hit me up on Twitter at @AndiMann; or send me an e-mail to me at andi.mann@ca.com.

  • Making DevOps Work in Complex Enterprise Environments

    Making DevOps Work in Complex Enterprise Environments

    Welcome back to the devopsy.kinsta.cloud series, ‘Enterprise DevOps Q&A’. Remember, if you have specific questions about Enterprise DevOps, let me know and I will try to answer them for you.

    Today’s question is:

    Q. In large enterprises, there are always environmental differences. From public cloud to internal hosting, from virtual desktops to legacy and mainframe – it is not possible to have full consistency between environments. Do you have any suggestions for approaching this gap?

    The Challenge of Enterprise Complexity

    This is a huge problem in larger enterprises, especially as DevOps practices grow beyond just isolated departmental applications. Architects have one system for prototyping; developers have a different system to code on; test and QA have their own unique ‘standard’ systems; while operations have untouchable systems for production.

    Add to this the complexity of backending into CICS or IMS applications, accessing data from DB2 or ADABAS on iSeries, or automating WebSphere deployment onto HP/UX; plus licensing for an Oracle database on AWS or an SAP client on Azure; and meeting compliance requirements for data privacy and operational security. This enterprise complexity and environmental inconsistency so often means new code is not properly tested until it hits production, resulting in unacceptable bug counts and an average annual downtime cost of almost $700K per incident!

    No Snowflakes

    Many will say that you cannot perpetuate ‘unique snowflakes’, so you must standardize environments. This dogma is easy to preach – and in principle is a best practice – but in reality it is too simplistic for most enterprises. Disparate environments often exist for good reasons (e.g. to accommodate cost, skillset, throughput, integration, or other technical requirements). In any case, you cannot simply run roughshod over established architectures. So standardize if and when you can, but realize that this path is unlikely to solve the problem.

    Abstract Platforms

    When you cannot fully standardize, you can still approach consistency by abstracting unique platforms. For example:

    Automate Complexity

    Automation can also help manage enterprise complexity to lessen the impact of environmental inconsistency. For example, solutions for enterprise-grade configuration automation or release automation can both streamline the handoff between dev and ops teams, where environmental inconsistency causes significant issues. Or adopting test automation technologies like service virtualization that can streamline development, test, QA, and other use pre-prod use cases, resulting in

    “shorter test times, increased productivity, and better production quality” by enabling complex environments to be consistently replicated on demand.

    Tool Selection Matters

    Be careful though. DevOps is not all about tools, so automation is not a complete answer; and for enterprises in particular, tool selection is harder too. For more complex enterprise environments, you will likely need to look beyond the trendy open source/webscale/startup tools. For example:

    •  Chef and Salt both claim support for Linux, Windows, and Solaris, but not AIX or HP/UX
    • Docker claims support for Linux and Windows desktops, but not for UNIX or Windows Server
    • Ansible only documents support for Windows control machines in a recent beta version
    • Most cloud services only support x86 architectures, and then only Windows or Linux
    • There is limited specific support for Mobile DevOps beyond Jenkins and some blogware
    • Only traditional large vendors support traditional large systems like IBM i or System z

    And, it must be noted, many ‘enterprise’ tools don’t support newer environments like MacOS, Azure, or Google; or don’t integrate well with a diverse toolchain.

    So when selecting automation and other solutions for your DevOps toolchain, make sure they support your environment, from mobile to mainframe, whether on-premises, hosted, or in the cloud.

    What is Your Solution?

    These are some solutions for the consistency gaps large enterprises are facing, but are just scratching the surface. I would love to hear your solutions for handling enterprise complexity within a DevOps approach. Or if you have any Enterprise DevOps questions, let me know and I will try to answer them. You can leave your ideas or questions in the comments below; hit me up on Twitter at @AndiMann; or send me an e-mail to me at andi.mann@ca.com.

  • Enabling Collaboration Across Diverse Cultures

    Enabling Collaboration Across Diverse Cultures

    Welcome back to this series, ‘Enterprise DevOps Q&A.’ In my last post, I talked about breaking down the barriers between dev and ops. Now, I want to address another, very typical question from a global enterprise about managing cultural differences in a globally dispersed enterprise.

    Q. Being part of a large org, our biggest issue seems to be a global culture clash. How can we address the issues of geographic location and culture that are a big obstacle for us?

    Large multi-national enterprises, like CA Technologies too, often have major data centers in multiple geographies in North America, EMEA, Asia and Latin America. Despite so much data center consolidation and cloud migration, cross-geo cultural issues remain a major issue.

    For example, in Israel the holy day is Saturday, while in the US and Europe it is Sunday, so it can be hard to lock down a release window. In Japan it is common to work long, late, and on the weekends, while by contrast some French labor agreements specifically limit the workday, so ‘crunch time’ may not have the same meaning in both locales. In the USA, staff may only get ten days annual vacation, and not even take all of it, while much of Europe takes all of July or August off, so consistent teaming can be a challenge.

    To help you address these geo-cultural challenges, here are some approaches I have seen my customers adopt including:

    A local approach

    A local approach, such as co-locating devs and ops to service business users in a common geographic or cultural area allows teams to adopt DevOps practices within cultural norms. This may mean imposing a difficult structural change, but it can also be very effective. See my last Q&A for more on this approach.

    A ‘core team’ approach

    Some businesses have assigned one or two team members from each cultural area to represent their teams in a common virtual or ‘core’ DevOps transformation team. Representatives on the core team can translate (sometimes literally) for their ‘home’ teams, and bring unique culture back to the core team. This can raise process/hierarchy barriers, but when done well can also help to break them down.

    A vertical approach

    I have seen a handful of global businesses implement DevOps vertically, with devs and ops teams, often geo-located, acting as service providers to individual business units, rather than the whole company. This is best suited for organizations that are already organized by business unit, but it ensures teams have common focus on specific and shared business problems.

    A ‘known good process’ approach

    Speed the flow of software delivery by focusing on refining and documenting specific processes all teams can rely on.  Each geographical or cultural team can define and share their own ‘known good process’ (cf. the Visible Ops handbooks), and share the interfaces to those processes, so they can continue to optimize their own activity, but work seamlessly with other teams at handoff points.

    A process automation approach

    Even better is if each team can encode their processes in automation to make it possible for anyone, anywhere, using any language, to execute those processes remotely. Even small improvements matter, so even task automation can help get closer to a cross-cultural, ‘enterprise-wide,’ DevOps approach.

    A ‘pace-layered’ approach

    Different business services have differing levels of acceptable risk and tolerance to change. Mobile app dev, for example, is more tolerant of risk than, say, accounts receivable. Rather than changing the world (literally!), instead build a DevOps approach into a specific service delivery team, even in a single geography. This ‘pace-layered’ approach is working very well for some of the world’s largest enterprises.

    A tactical approach

    Similarly, you do not need to fix all your service delivery problems at once. In a global enterprise, pick one big winner at a time instead. Identify constraints in the test process, a roadblock in QA, a provisioning delay, a problematic release process, a monitoring gap, or a broken feedback loop, and connect cross-cultural staffs from similar functional teams to solve specific shared issues.

    A technology-assisted approach

    Regardless of the approach, technology can assist. Global tools with multiple language support can enable common experience and terminology. Telecommuting can give regional staff a virtual presence in team meetings. At CA Technologies, we are trialing remote-controlled robotics to give staff a physical avatar in meetings from anywhere across the globe.

    Other approaches?

    As with most of DevOps, there is no definitive prescription, and I am sure there are many other approaches that can work. If you have ideas or experience to share, please post in the comments below, and let us all know how you have or could address these cultural and geographic barriers.

    And remember, if you have questions about Enterprise DevOps, let me know and I will try to answer them for you. If I can’t answer them myself, I will find someone who can! Please leave your questions in the comments below; hit me up on Twitter at @AndiMann; or send me an e-mail to me at andi.mann@ca.com.

  • Removing the Wall Between Dev and Ops

    Removing the Wall Between Dev and Ops

    Welcome to the new devopsy.kinsta.cloud Q&A series on Enterprise DevOps. In the first post in this series, I answered five key concerns for adopting a DevOps approach in a large enterprise. Now, I want to address a more specific question from an admin at a large enterprise about the silos and barriers between dev and ops …

    Q. How do you work with customers to remove the imaginary wall between dev and ops to create that DevOps environment?

    This is perhaps the core of a DevOps transformation – dev and ops teams working closely together toward common goals, rather than in isolated silos. I have seen several methods have substantive impacts on these entrenched silos, so I will take you through some of the more effective approaches.

    Familiarization

    In many organizations, dev and ops barely know each other – a huge barrier to the empathy at the centre of devops. Breaking this wall down may be as simple as getting both departments together for some social activities. Some successful approaches I have seen include:

    • Sponsor a one-off or regular shared non-work team activity, like paintball, a pool tournament, team picnic, LAN game, or cooking classes. Bring both teams together in a non-workplace environment to work together. Remember to mix up the teams – you do not want to reinforce any us vs them mentality;
    • Schedule sessions for team members to present to each other in short sessions (think lightning talks like Ignite or TedX) beyond technology to any areas of interest – like ‘My Sports Team,’ ‘Steampunk in the 21st Century,’ ‘Restoring Train Carriages,’ or ‘Strategies for TF2 Domination.’ Share common interests to create empathy for each other as people, not just functionaries;
    • Initiate informal knowledge sharing, with regular ‘lunch-and-learn’ or ‘brown bag’ sessions, where team members can explain what their role is, why they do it, how they do it. Offer a ‘Day in the Life…’ presentation, a Q&A session, or a whiteboard over pizza and sodas and your technical staff will come out like vultures, ravenous for … knowledge!

    Co-Location

    Telecommuting has many benefits, but standup meetings are hard to achieve across five time zones, ten locations and six languages. People communicate better in person, so physically co-locate teams as we at CA and others have done, to enable them to connect and talk in person, not just via e-mail or IM.

    Geo Alignment

    In multi-national enterprises central developers often push code to global operations, reinforcing the silo mentality. Align local devs and ops with local business users instead to facilitate customer-centric agile development, and enable smaller, closer, more agile teams with more immediate shared goals.

    Secondment

    Consider seconding team members into the other team e.g. for one day a week, one week a month, or one month in six. When ops learn firsthand the basics of their dev counterparts, and vice versa, they will better internalize how to work together, and the impact of each role on the overall process.

    Restructuring

    Some enterprises have combined dev and ops into one team. While often seen as an ‘anti-pattern,’ for some organizations this is working well. Others have adopted a flexible org structure with hierarchical management for administration (reviews, goals, training, etc.), but loose teaming for functional duties.

    Shared Technologies

    Beyond cultural and attitudinal shift, technology can also help. A shared toolchain for the end-to-end service delivery lifecycle can bring teams together too. Common tools for planning, testing, release, monitoring, or triage will help break down barriers with a common terminology and shared experience.

    Other Approaches?

    While there are no silver bullets, these are some ways I have seen enterprises break down these silos. But perhaps you have seen or been a part of other approaches? If so, other readers and I would love to know more, so please take the time to leave a note in the comments below, or drop me a line directly.

    And remember, if you have questions about Enterprise DevOps, let me know and I will try to answer them for you. If I can’t answer them myself, I will find someone who can! Please leave your questions in the comments below; hit me up on Twitter at @AndiMann; or send me an e-mail to me at andi.mann@ca.com.

  • Five Top Tips for DevOps At Scale

    Five Top Tips for DevOps At Scale

    Since I published my blog explaining the topic of ‘Big DevOps’ late last year, the concept of Enterprise DevOps has exploded, with commentary, experience, and advice (some good, some awful) popping up from many different quarters. Fortunately, many pundits have now come to agree that application delivery in a traditional enterprise is very different from in a typical web-scale/startup environment.

    As a result, a growing body of work is filling a significant and important gap with unique insight into Enterprise DevOps, from many excellent commentators and practitioners who understand how large enterprises are different, and how to address those different challenges.

    As Enterprise DevOps takes off, I seem to field questions about it on an almost daily basis, so I decided to contribute to the body of work with a webinar explaining my answers to five of the most frequent challenges I hear about DevOps at scale. It was titled ‘Big DevOps: Five Top Tips When Implementing at Scale’ and you can catch the full replay on YouTube.

    The top five issues I chose to address are below:

    TopFiveIssuesDevops

    My five ‘Big DevOps’ tips for overcoming these issues center mainly on cultural change, because DevOps is primarily a cultural and attitudinal shift. However, it is also undoubtedly true that certain technologies can play a large part in enabling that change, especially in the more complex world of larger enterprises.

    With that in mind, my top five tips to address these challenges are as follows:

    FiveAnswersDevOps

    • Facilitate a culture of teaming and collaboration – this is at the core of devops in any environment, large or small, but especially for large organizations with bright lines between teams, this is essential to overcoming the tyranny of complex and dispersed silos
    • Establish agile development as a shared discipline – agile development is brings code to market faster, but can result in just filling the clogged pipes of PMO, HR, training, security, and other departments faster. ‘Agile’ needs to be common throughout the SDLC – in dev, ops, and beyond
    • Automate relentlessly to enable rapid devops response – automation allows faster execution throughout the system, enabling downstream processes to keep up with speed of DevOps. Automation also adds audit and governance, especially important to larger organizations
    • Push smaller releases faster, measure and remediate impact – large organizations especially are can avoid big errors by avoiding big releases. Adopt faster, smaller cycles with closer scrutiny to achieve a DevOps approach while managing the risk of faster cycles and fewer controls
    • Manage user identities to enable shared access – in smaller organizations it might be acceptable to give out root access like candy, but not for large enterprises. Identity and access management can enable a new DevOps approach that is safe, secure, auditable, and well-governed

    While these tips may not be the perfect answer for every organization, I have seen how these approaches have made the journey to DevOps easier, safer, and more effective for big enterprises. I hope they help you on your journey.

    If so, you will want to stay tuned to devopsy.kinsta.cloud. Since starting work on Enterprise DevOps concepts, I have collected many questions on how large organizations can best succeed in their DevOps transformation. I will be sharing these questions along with my answers here on devopsy.kinsta.cloud in a regular series, to give everyone the chance to understand how large enterprises can overcome the challenges of adopting DevOps at scale.

    So keep your eyes out for my posts over the coming days, weeks, and months in the brand new occasional series, ‘Enterprise DevOps Q&A’. I hope you will keep reading, and maybe even help me out by discussing your experiences and ideas in the comments below.

    Finally if you have any questions about Big DevOps, Enterprise DevOps, or anything to do with adopting a DevOps approach at scale, let me know in the comments below too. I would love to hear from you, and answer your questions if I can, or source guest opinions from the DevOps community.

    Meanwhile, please stay tuned for more Enterprise DevOps Q&A. I am excited to continue the discussion on what it takes to transform application delivery at scale.

  • Five Top Tips for Cloud and DevOps Automation

    Five Top Tips for Cloud and DevOps Automation

    A while back, I overheard someone asking why many of the ‘cloudy people’ have moved on to DevOps.

    One reason is certainly that cloud is at the heart of DevOps. Many of the grounding principles of DevOps – rapid iteration, agile development, automated testing, continuous delivery and continuous integration – are barely even possible without cloud. However, I think it runs even deeper than that. DevOps and Cloud are both fundamentally enabled by the same technology I have been working with my entire career – automation.

    When IT leaders get excited about cloud, it is mainly because of automation. Sure, cloud providers get excited about essential characteristics like broad network access and resource pooling, because that is how they maximize reach and minimize costs. Similarly, CFOs get excited about the measured service, because that means they only pay for what they use.

    However, according to a recent Luth Research study, most IT leaders prioritize the higher-order values enabled by on-demand self-service and rapid elasticity – faster innovation, superior performance, better scalability and higher resiliency. These capabilities are deeply rooted in automation – of provisioning, deployment, configuration, scale-up/scale-out and other processes.

    Similarly, DevOps – beyond a certain point – also builds on a foundation of automation. As Rackspace CTO John Engates noted in his recent column in this journal, “DevOps is more than a set of automation tools.” He is right, of course. You can achieve a great deal through cultural and structural changes – better collaboration, shared goals, team building, process improvements, etc.

    However, tools for release automation, configuration automation, process automation, test automation, service virtualization and provisioning – just to name a few – are adding tremendous value with a rapidly maturing DevOps automation toolchain. Indeed, in recent research from Vanson-Bourne, over half of senior IT decision-makers worldwide rate automation as the most important DevOps component – above such putative stalwarts as agile development, collaborative teaming and process alignment.

    Looking at the history of enterprise automation, and considering my own experience, I believe there are many common principles that you can apply to both cloud and DevOps. Here are just five of my top tips:

    Don’t Automate Bad Processes

    Automating bad process just makes bad things happen faster. This seems obvious, but I often see automation that only encodes (and accelerates) a bad existing practice. To avoid a high-speed train wreck, first identify, catalog, document, review and refine key manual processes to make sure you are automating the right activities, at the right time, in the right way. Solve for simplicity, speed and throughput, even before you apply automation. Remove friction across silo boundaries – departmental, technological and disciplinary. Start simple, be inquisitive, and iterate methodically. Then make sure you automate deeply, thoroughly and efficiently.

    Cross-skill Existing Resources

    Great people are hard to find and salaries for cloud- and devops-relevant skills are amongst the highest in IT. This explains why, according to the same Vanson Bourne research, over 70 percent of DevOps leaders are retraining existing personnel, while only 50 percent are hiring new resources. Look for staff with a history of automation in their resume, through multiple waves of technical change, rather than specialists in one automation technology or product. Find the IT equivalent of a “good athlete” rather than a “position player”.

    Security Cannot Be an Afterthought

    Automation systems and the staff responsible for them have always needed highly privileged (and frequently shared) user IDs, whether automating a mainframe IPL or scripting a production code release. This is an audit red flag, and can result in high-impact incidents, whether accidental or malicious. Identity and access management provides essential audit and control, while giving users (including automation tools and APIs) easy access to the resources they need (locally and in the cloud), when they need them.

    Document, Measure, and Backup

    If you do it right, automation quickly becomes pervasive and mission-critical. Without documentation, you create inscrutable systems, tribal knowledge, the hero and other ‘anti-patterns‘ in your Cloud or DevOps machine. Without measurement, you cannot know if (or when) complex autonomous systems run amok or stop working. Without backups, critical systems are prone to irrecoverable failure. Cloud and DevOps teams must value and prove traditional disciplines like documentation, measurement and backup – painful as they may be.

    Build Awareness into Automation

    As automation becomes mission-critical, it must be resilient to errors, failures, exceptions and other unexpected conditions, and still deliver a correct result. Remember when ‘autonomic computing’ was a thing, when we dreamt of ‘self-aware’ systems? Similarly, concepts like ‘desired state’ and ‘idempotence’ go back at least as far as mainframe and server automation. Similar concepts apply to cloud computing and DevOps too. In both, it is critical that automation routines are aware – of their own state and of the surrounding environment – so that they can test for and accommodate unexpected conditions, and still deliver the expected results.

    Given the depth of history in automation, and the many parallels with modern computing, some may claim that cloud is nothing new, that we have been doing that since the mainframe; while others will say something similar about DevOps. I think they are wrong, but there are definitely parallels, and George Santayana has a point – we should be at least informed by history, if we are going to try to write the future, while avoiding the mistakes of the past.