devopsy.kinsta.cloud

Tag: sdlc

The Role of a Traditional NOC in the new DevOps World
This new question in the devopsy.kinsta.cloud Enterprise DevOps Q&A series comes from a large business, though not what many would recognize as a traditional or legacy enterprise. This organization, like many high-growth businesses, is in a relatively new market segment that only exists due to the rise of cloud computing and digital media. In their business, the network (especially access to and from their public cloud service) is critical to the value they provide their customers.

So it is no surprise then that, as they contemplate their transformation to an agile environment built on DevOps principles, they are asking me about the role of the Network Operations Center (NOC).
1. What Role Does a NOC Play in the DevOps World?
Despite coming from a ‘new tech’ business, this is another of the questions that is typical in a large enterprise. While smaller startups and even many web-scale businesses may have an Application Support team, not many run their own NOC, and even fewer separate out these roles from the rest of the operations team.

First, some things do not change in a DevOps world. Fundamentally, the NOC is still tasked with maintaining high quality production networks and systems, detecting potential problems as soon as possible, triaging those problems to find root cause, and remediating problems as fast as they can.

However, the NOC team will gain additional activities and support a broader interlock beyond just the production environment, especially as networking becomes more embedded in application delivery through capabilities like software-defined networks and cloud computing. They will work more closely with teams like app dev, SQA, pre-prod and release; they will provide input to design and development; and they will contribute earlier in the software cycle. This is the logical outcome of a DevOps approach that emphasizes the need to work more collaboratively across the entire end-to-end SDLC.

So you should consider bringing these network specialists into:
- development planning, to help architects and engineers to build applications with operable networking by design, including software-defined networks that help make a truly idempotent application
- testing and QA, so they can provide engineers and SQA teams with tests and analysis that accurately reflect the reality of the prod network, rather than leaving network testing to after production deployment
- dev/test support (where they may already be acting), to ensure that the environments that dev/test rely on to create new capabilities are just as stable and reliable as the systems they will eventually be deployed to
- release and deployment, to ensure they have input into determining and defining ‘known good values’ for production monitoring and for problem diagnosis, triage, and escalation – before release, not just after
- expanding interoperability and self-service, by exposing ‘known-good’ deployment, triage, and (re-)configuration capabilities as automated software-defined processes (or APIs) that dev/test can use without opening a new service ticket
- a broader role in ensuring the end-to-end quality of the network, alongside traditional operations/infrastructure support as part of a new role commonly referred to as a ‘Site Reliability Engineer’ (SRE)
Because DevOps is about improving collaboration, remember too that you should also bring dev, test, QA, etc. into the NOC, and even rotate them onto support, so they have an understanding of the core operations of a production network, how their code acts in a real environment, and how their work affects other teams. This will help to build the empathy between teams that is a core tenet of DevOps.

You may be concerned that asking your NOC to work more upstream is adding a load of work to those teams that they didn’t have before. That may be true, but a significant outcome of Enterprise DevOps in practice is to reduce the ‘busy work’ of such teams, free them up to do more ‘smart work’, and to do it earlier in the software lifecycle.

For example, recent research has shown that DevOps is correlated with improving the quality of applications (especially before they hit production), reducing the amount of time and effort the NOC team spends on problem detection, triage, and remediation. In fact, in this research, ‘reduction in time spent fixing and maintaining applications’ is one of the strongest outcomes, with an average 21% reduction in this metric.

Similarly, standardizing on known-good processes, APIs, configurations etc., and enabling other teams to employ self-service and software-defined networking capabilities will allow the NOC team to spend less time on routine responses to things like servicing provisioning and configuration tickets.

I cannot guarantee they will not be a zero-sum game, especially in the short-term, but over time a DevOps approach should actually spare workload from the NOC team, freeing them up to ‘shift left,’ moving their talents earlier in the SDLC, as they communicate, collaborate, and integrate more closely with upstream teams like dev, test and QA.
February 4, 2015
Top Skills for Enterprise DevOps
There is one question I hear almost every time I talk with my large enterprise customers about DevOps:
1. What skills do my people need to be successful with DevOps?
Talk of DevOps almost immediately focuses on culture – like having empathy for fellow workers, being flexible and adaptable, seeking continuous improvement, building relationships, etc. However, while critically important in DevOps, culture is an outcome, not an input; and such attributes are mostly either innate or acquired slowly. Culture cannot easily be taught.

At the other end of the spectrum, we often look at technologies – agile development, coding languages, virtualization/cloud platforms, management tools, etc. Again, these can be important, but while learning these skills might make you a better dev or op, they may not make you a DevOps practitioner.

Like most transformations, DevOps will need new people, process and technology skills, so in this post I take a look at a handful of the most important, pragmatic, and trainable skills in all three areas for practitioners looking to adopt DevOps, especially in large and more complex organizations.

People Skills
- Communication skills – while this may not come easily to typically introverted IT staff, both written and verbal communications, within and across the organization, and even with customers, are essential to a strong DevOps approach. This skill is learnable from formal courses, groups like Toastmasters, direct mentoring, social activities, or informal sessions like ‘lunch and learn.’ For extra credit, jump on stage for open mic night at a local comedy club!
- Business alignment – much of a dev or ops life is spent embedded in the technology of software delivery, rather than the business need. This can cause many issues, and is something DevOps tries to address. Understanding the business strategy, goals and priorities is essential to DevOps, helps drive and easily trainable. A ‘big picture view’ helps everyone collaborate on common objectives to deliver high-value business outcomes – a key tenet of DevOps
- Workplace history – “an appreciation for history, both technical and organizational”* will help staff understand the critical antecedents of DevOps, so they can internalize its principles not just rote learn its actions. For example, understanding hierarchical and network structures, why we adopted open work environments (and why it hasn’t always worked), and the antecedents of DevOps in things like The Toyota Way will help staff not just to act predictably in known circumstances, but adapt appropriately to new ones. (*Thanks to @garethr for input on this!)
Process Skills
- Agile development – it may be possible to take a DevOps approach in a waterfall environment, but it is almost definitely less effective. DevOps arose out of the Agile movement, and they pair so well that it is sometimes referred to as ‘agile for infrastructure.’ Teaching on one or more specific agile approaches (Scrum, Kanban, Kaizen, etc.) will give staff an approach and mindset informed by agile practices to help teams work together to improve flow, delivery and quality
- Lean approaches – ‘lean’ theories born in the manufacturing world (see ‘history’ above) give a pragmatic foundation for applied DevOps, not just new-age ideals. Concepts like Just-In-Time, Toyota Production System, Design for Manufacture, Theories of Constraint and Waste can be learned from original texts, courseware and many business courses. Some are also revised in modern IT-specific texts like The Phoenix Project or Implementing Lean Software Development
- Process reengineering – DevOps is as much about refining and improving ‘flow’ in the SDLC as it is about encouraging a sharing culture, so classic learning about how to effect change in business processes is invaluable to early stage DevOps adoption. Understanding how and why to analyze needs, identify problems, and effect change in software processes, organizational structures, teaming etc. is essential to DevOps and a long-established body of knowledge
Technology Skills
- Coding/scripting – in its current iteration there is a lot of focus on helping operators to understand coding languages; and teaching developers to use script-driven operations tools. From JavaScript (eek!) to C# or .NET, any coding or scripting skills will help both dev and ops to have empathy for each other’s jobs, and help to develop the elusive ‘T-shaped people’ that are so well-suited to a DevOps approach
- IT Automation – Enterprise DevOps especially will likely involve (require?) a raft of technologies in an automated tool chain. Because DevOps is concerned with end-to-end flow, such skills need to cover the whole SDLC. Look to provide training in Portfolio Management, Agile Project Management, Source Code Management, Version Control, Test Automation, Test Data Management, Service Virtualization, Continuous Integration, Release Automation, Cloud Provisioning, Configuration Management, Process Automation, Performance Monitoring and Event Management, whether through formal training or informal knowledge transfer
- Workflow mapping – smaller organizations might get away with pen and paper, or even a whiteboard/flipchart, but to document, share, analyze and rationalize the SDLC for large, complex, enterprise applications, you will need to learn to use workflow mapping tools. This will allow team members across the org (and the world) to take ownership of the end-to-end process, and start to collaborate on continual improvement
This is of course a (very) short list, but should give you a starting point for developing essential skills for enterprise DevOps practitioners. As always, please feel free to drop a comment and tell me what I missed, what is more important, or even what I got wrong. I am always interested in discussing ideas and learning new things!
January 5, 2015
Two Ways DevOps Unlocks the True Potential of Agile

In a recent post on InformationWeek, Erik Weber, a professional Scrum trainer and Agile expert, explores four signs that you’re doing Agile development wrong. Upon finishing the article, I was reminded that Agile in theory is often very different from Agile in practice, and when that gap exists within IT, it can have massive consequences on both development operations and the business.

At the risk of stealing some of Mr. Weber’s thunder, I’m going to spoil the first two signs he discusses – namely, “failure to ‘get done’” and “failure to launch” – because I believe they are common challenges for many IT groups. The good news is, organizations can close their gaps and realize the true value of Agile development by incorporating DevOps processes and tools into their systems development life cycle (SDLC).

Getting to “Done” Faster

“If it takes you longer than a few weeks to turn an idea into a production-ready feature, you are not Agile – full stop,” says Weber, and I would agree. The whole point of Agile was to move away from the slow, monolithic development methodologies of the past by breaking the process into smaller chunks and having developers, testers and business development people working concurrently to speed new features to market. Most Agile practitioners would say they’re doing just that, but many are still not “getting to done” as fast as they would like. Why is that?

According to Weber, the biggest problem is that many IT groups leverage Agile techniques for development, but then do all of the testing at the end. Obviously, this creates numerous problems that can delay project completion. As we all know, the later bugs and glitches appear in the SDLC, the longer it takes to fix them. When that happens, costs go up, delivery pushes out and customers are unhappy.

The DevOps principle to employ here is getting the operations team involved in testing earlier in the SDLC – essentially shifting their work left – and the tool that can help is service virtualization. By simulating constrained or unavailable systems across the SDLC, service virtualization enables developers, testers and performance teams to work in parallel for faster delivery and higher application quality and reliability. With DevOps, they can realize truly parallel Agile development without constraints – and ultimately get to “done” faster.

From “Failure to Launch” to Continuous Delivery

“If you haven’t actually released your software to production – and on to customers – for three or more months, you are missing the big picture,” argues Weber. He believes that many product managers and executives hold back official product releases because they “must have all the features,” and when this happens, they forfeit the ability to both get new features into customers’ hands on a regular basis and react more nimbly to their wants and needs.

There are numerous reasons why the “all features” mindset persists, but a good way to prove its obsolescence is by incorporating the DevOps principle of continuous delivery into the development process. Continuous delivery means always being in a ready state for pushing applications and services through the SDLC and into customers’ hands – and release automation is the tool that helps organizations reach that state.

Release automation enables continuous delivery by automating complex, multi-tier release deployments through orchestration and promotion of applications from development through production. As a result, visibility improves across an organization’s deployment chain, giving it the flexibility and agility to continually release new features to market and react quickly to changing market demands. Once an IT organization reaches a state of continuous delivery, it becomes easier for executives and product managers to pull the trigger on a given release – because they know that the next one isn’t far behind.

Agile is as Agile Does

Weber offers four signs of ineffective Agile development in his article, but I would argue that there is a unified idea running through them all: It’s not enough to employ Agile practices; you have to follow through until you meet your Agile goals. Ask yourself, “Are we really delivering releases to market faster, at a higher quality and at lower cost to the business?” If the answer is “no,” the right DevOps methodologies and tools might be just what you need to put these failures behind you and finally realize the true value of Agile development.

June 4, 2014
DevOps and The Goal

Reading, “The Goal” was a pivotal event in my career. More than a decade ago a friend of mine who was an operations professor at Georgia Tech told me it was his favorite book. We were discussing the challenges of getting work done in a corporate environment. I was grumbling about leading a web dev team in a company that did not embrace the software development discipline. Already a fan of Tom DeMarco and Timothy Lister’s, “Peopleware”, and being very vocal about the lack of ability to focus whether due to environmental distractions or the constantly changing priorities and schedules, I was receptive to the message in “The Goal”. It gave me a vocabulary and a roadmap for improving our throughput issues.

Our throughput problems were a textbook case for development disaster. When I joined the project, we had three source code repositories, from none of which could we build production and when we did produce a build it would work in pre-production environments but not production (this just slightly predates the widespread acceptance of continuous integration). We also had the classic inconsistent release cycles that would lengthen based upon insistence by the business owners to include more features. Looking back, it is not surprising in the least that we were not successful.

Not surprisingly, these issues led to a lack of trust with the business. Overcoming the technical problems was straightforward using brute-force tactics. We addressed the source code problem by recruiting a senior developer and empowering him to make improvements. We overcame the build quality issue by using a primitive form of agile and enforcing knowledge transfer; the operations engineer sat between two developers during the build and release process and he was the only one allowed touch the keyboard. Our tactics were basic, and they worked.

The bigger challenge facing us was getting our business owners to accept a different SDLC model. We were saddled with a classic waterfall process, each phase being dependent upon the last with a lot of idle time built-in at each phase gate. The shortcomings of the waterfall process were amplified because the business lacked confidence about when, if ever, they would get a feature released to production. This lack of confidence led to a constant rearranging of the priorities to get the most important feature du jour in the next release. The constantly changing priorities created even more idle-time and a lot of work-in-progress (WIP) which Goldratt’s Theory of Constraints (TOC) tells us is at the root of inefficiency.

Using TOC as a guide, we redefined our SDLC from requirements through QA. We focused on developing a process that de-coupled the phases, reduced idle-time and work-in-progress (WIP). De-coupling the phases meant the work could queue between each phase allowing the teams, (requirements, development, QA, and operations) to vary independently; they didn’t have to wait on the team before or after to complete their work which reduced idle-time. We then focused on improving the slowest areas, which seemed to be requirements and deployments.

We convinced our business owners to trial a monthly release cycle (this was a long time ago, even before continuous integration was widely accepted!). The monthly release cycle helped give the business confidence that they would see their features in production at an expected pace and it allowed requirements to cycle on schedule independent of development and QA. Some features would get done in a month and some would span several months. Over a period of three months we got into a rhythm that worked for about 14 months until organizational focus changed and we focused elsewhere.

Reading “The Phoenix Project” reminded me about those days and the challenges Goldratt helped us overcome. I really liked “The Phoenix Project” and many of the situations rang true. I have since moved from development to operations (twice!) which gives me a unique perspective as I see both sides. I see DevOps as the embodiment of Goldratt’s ideas in IT; reducing work-in-progress, reducing cycle-time and always focusing on the most constrained processes. Add a consistent release schedule and it looks a lot like DevOps. Everyone has their own ideas about what is DevOps and how best to implement, I suggest we follow Goldratt’s advice and focus our efforts where they will have the most benefit.

March 21, 2014