devopsy.kinsta.cloud

Category: Enterprise DevOps

DevOps is a Risky Business
I spend a considerable amount of time with clients discussing DevOps and the potential value it can bring to them, but there is one theme that always resonates: the ability of a DevOps approach to eliminate risk. While many of us understand the benefits in terms of being faster (improved productivity), cheaper (reduced cost) and better (improved quality), I find that a conversation based around risk, especially when talking to the business or department heads, usually gets straight to their concerns in a language they understand.

IBM’s definition of DevOps is not simply about connecting development and operations; it is fundamentally about providing a capability that: 1) allows lines of business to connect with their customers (which may be internal employees) quickly, and 2) facilitates rapid feedback from their customers so relevant adjustments can be made. DevOps, therefore, embraces the philosophy of iterative (and incremental) development and delivery, whereby solutions are delivered through a series of incremental releases.

By ensuring all aspects of a solution (requirements, designs, code, executables, tests, etc.) evolve over time as each increment is completed, DevOps is fundamentally focused on addressing risks including:
- The risk of not knowing the requirements (have you ever tried to write down and freeze the requirements for a mobile app?)
- The risk of not knowing the solution (especially when it’s new, you’re defining a new architecture or you’re embracing new technologies)
- The risk of not knowing the effort (especially when the solution is novel for the organization)
- The risk of not delivering on time (when the pace of the team, for example, is an unknown)
- The risk of not providing the right level of quality (and the impact this may have on customer sentiment)
- The risk of adopting inappropriate practices (meaning that the approach being taken by the team is not working for them)
- The risk of having an ineffective delivery environment (including automation through the use of delivery tools)
I could go on. There are so many risks that can be addressed simply by delivering working software iteratively, that it’s surprising how many organizations continue to fall back to a “waterfall” mentality—even in high-risk environments where much is unknown.

The tack I’ve taken here originates from my time at Rational Software Corp. (which IBM acquired in 2003), where we would emphasize the need to deliver software iteratively and progress was measured based on the results achieved (i.e., working software) rather that the documents produced. My favorite graphic from that time is the one shown below, which shows how a modern approach changes the risk profile associated with a delivery project, since it focuses on removing risk as early as possible in the life cycle.

In summary, incremental and continuous delivery evident in a DevOps approach allows us to call out risk early and determine its resolution. DevOps often is used along with “lean startup” thinking (and its build-measure-learn cycle) and monikers such as “fail fast” since these are, at their heart, also focused on addressing risk.

Take the DevOps self-assessment to find out where are you on your DevOps journey

About the Author / Peter Eeles

Peter Eeles is an Executive IT Architect in the DevOps Global Tiger Team, IBM Hybrid Cloud, where he helps organizations improve their IT delivery capability in order to drive tangible business value. He has a particular interest in the adoption of architecture, agile, DevOps and cloud as innovation enablers. In the IT industry since 1985, Peter comes from a delivery background and was previously chief architect of IBM Rational’s Worldwide Solution Delivery organization, where he led many large-scale transformations in a variety of industries. Peter has written many papers and several books, including “The Process of Software Architecting” (2009), “Building J2EE Applications with the Rational Unified Process” (2002), and “Building Business Objects” (1998). He runs his own website at www.architecting.co.uk.

Connect with Peter on LinkedIn and Twitter.
May 19, 2016
Dying for Intelligent Calendar Awareness

What would we die for in an integrated DevOps tooling solution—something that would make our operations partners stand up and cheer? Intelligent calendar awareness.

The most fundamental construct of any release orchestrations (RO) tooling software is the calendar. RO software, no matter who the vendor is, uses a form of the calendar, to outline what tasks will get performed in what order from beginning to end. Better RO software allows for parallel tasking, but all RO software uses a calendar. So the most fundamental use of a calendar construct is focused on “when” tasks are performed and completed. Adding intelligence to a calendar construct adds three more dimensions or characteristics to this baseline feature.

The Intelligence of Ownership

First is the concept of ownership. Instead of just marking a calendar with the dates of specific tasks, or clusters of tasks, the addition of ownership marks the calendar with “events” that are “owned” by specific teams of people intended to perform groups of actions or tasks. As an example, instead of just marking the calendar that on April 27 this year, with the specific tasks we need to run a mini-release to upgrade our Oracle database from version “x” to version “y.” With the addition of ownership, we could mark this date as running an “event” of an Oracle database platform upgrade. This new event would be “owned” by the database update team.

The block of time the team outlines for the Oracle upgrade “event” would then take on two additional characteristics for teams in general: inclusionary and exclusionary implications. As an example, the time set aside for the tasks to upgrade the Oracle platform becomes inclusionary for the owning team. Members of this team would be allowed to schedule tasks during this block of time, ideally intended to facilitate this goal of upgrading the platform.

Other teams are (by default) excluded from performing actions during this block of time so they don’t impact what the database team is attempting to accomplish. When other teams pull up their respective calendars to schedule some new task or event, the April 27 block of time is greyed out or presented as unavailable because the database team owns an event listed in that block of time. For the database team the time is inclusionary, for all the other teams the block of time is considered exclusionary.

The benefits to you the customer are immediately visible. By adding ownership to the calendar construct, events could be scheduled in such a way that teams do not inadvertently “step on” each other (i.e., attempt to schedule tasks during the same window of time that would adversely affect the other). It also allows you the customer, to view a dashboard that shows the “kinds” of events that are scheduled throughout the week, month or quarter, and who is doing what.

To avoid being too strict with this idea, the RO tooling vendor need only add concept of granting exceptions to requesting teams by the owning team for events that would not impact their efforts. So as an example, if a different team wanted to work on tooling the SQL Server platform during the same block of time that Oracle platform was being updated, and the servers were different anyway, there is no reason why that team could not schedule a similar event of its own at the same time. If ownership is the only concept added in the RO tool, an exception would have to be granted by the Oracle team, but this would be costly in terms of administration. It is for this reason we introduce the second main construct in the intelligence of calendar management.

The Intelligence of Component Awareness

The second main concept of intelligent calendar management is component level awareness. This dimension of awareness is based on identifying specific “technologies” as resources or components that are part of the environment you operate at your company. It doesn’t have to be a ground-up exercise, and if you prefer one of those, you could use a standard discovery tool to keep it up to date. But the more basic idea is to simply identify the fundamental technology components you use in your world.

So as an example, I could identify that this company uses Oracle databases (perhaps also tracking the version if there are multiples of it), SQL Server databases, the type of servers it uses, the operating systems, the middleware, perhaps a mainframe platform and the network fundamental technologies, etc. It is highly likely that there are groups of people who are uniquely identified in this company that manage these specific types of technologies. This makes the associations easy for event ownership. The Oracle guys become associated with the Oracle technology components but not with the SQL databases, etc.

Having teams associated with the components they manage allows for better parallel event scheduling. But this is only the first level of intelligence. The next level is to track, by application, the underlying technology components they utilize. This process is done during the build and deploy automation anyway, so the effort is nothing more than what is done today. The beauty of having this spider web map, or this map of technology component dependencies, is that it adds a new level of capability to scheduling in our calendar.

At this point, scheduling an Oracle event does not preclude the SQL team from scheduling another one during the same time window, since those two technology types don’t conflict with each other. Both teams would see open calendars on the same day due to this intelligent calendar awareness of component interdependency, or lack thereof. This dimension of intelligence precludes having to grant needless exceptions to owned events. The only requests for exceptions would come from users or teams that actually would impact the upgrade event, and they could be evaluated for legitimate need and potential real impact.

Let’s return to our Oracle platform upgrade event our mythical company scheduled for April 27. With the addition of component-level awareness, when this event is scheduled by the Oracle database team, every application that uses Oracle is “automatically” notified that this time window is now off-limits for its development or usage activities. This limitation could include automatic notification to application development teams that they could not deploy new code during this window. Or the notification could be expanded to notify users of the impacted application(s) that this list would be down during this window of time. Applications that do not use Oracle would remain unaffected and not notified.

Imagine what this kind of automatic notification could do for both development and the user base of the technologies you operate, all coordinated automatically through the RO tooling you use today in DevOps. Then think about the possibilities from here. The Oracle team could look at a block of time again and examine whether scheduling on April 27 would have a high impact on other teams, or not, before it finalizes it. If the impact to others is high, perhaps the team picks a different date. The possibilities become endless.

The Intelligence of Scope

The third and final architectural dimension to add to calendar intelligence is the concept of scope. Scope is a simple hierarchy that you could define about your company by logical business subdefinition. As an example, you could outline the scope you define as: global (entire company), business unit (listed by name), product unit (listed by name) or departments (listed by name). Just using these kinds of simple scope definitions begins to insert an impact assessment into the event scheduling that takes place.

As an example, events could be scheduled with a global scope, owned by the change management governance body, that excludes a 10-day window because of a corporate audit by outside regulators. Now, when any development team attempts to schedule deployments or releases, for example, their calendars are automatically greyed out during this 10-day window because of the corporate governance event. Exceptions would only be granted by that governance body. In this case, this prevents accidental changes occurring during the middle of an audit.

If the example cited above is too broad, and the mythical company only needs to limit an event to a particular product line or further down to a particular department, the “scope” characteristic allows them to do so. This final dimension to intelligent calendar management does require a higher level of sophistication from the RO tooling vendor and a higher upfront investment in the administrative costs of providing DevOps services. But for an enterprise-class installation, this is an investment that can take the chaos of corporate communication where it comes to the impact of change to a systemic nirvana of communication on demand.

The Impact to Your Bottom Line

So why outline an architecture that most RO vendors have yet to implement? If it is candy I created that you can’t have yet, why discuss it? Two reasons: First, when you begin to examine the cost of running your DevOps set of services, you will find that, like many other enterprise-class efforts, it is the administration costs that are heavy at the outset and increase over time. The effort it takes to manage “communication” and avoid having interdependent technologies “step on” each other is no easy task, particularly when we have moved from weekly deployment windows to potentially daily deploys to production across large app portfolios or complex legacy hardware architectures.

Second, for those of you who still maintain data centers, consider how much time you “waste” if your Ops teams continue to require a 12-hour to 48-hour window over the weekends to make updates to their own platforms. Going to an external cloud provider is a way out of this, but for organizations that are unable to fully do so, the weekend downtime remains a significant issue. DevOps will create desire in the Dev teams to want to use this traditionally Ops window for their own purposes as well. Once DevOps has been fully embraced, there is as much a need to automate communications and intelligently ensure we do not step on each other as there ever was. DevOps accelerates this need, as it accelerates the speed of innovation itself.

The proper tooling, or rather the tooling that is thinking about intelligent architectures that will greatly reduce your costs and your overhead and ensure consistency in your world—these are the features (and the thinking) you need your tooling vendor to embrace. The alternative is having to pay your labor base to manage it another way. If your leadership and your vendor partners are not considering these issues, they should be. If you need help, feel free to contact me (I am still looking at the moment). The goal of DevOps to automate process should not just be about moving code; it should extend to how we administer, communicate and inject governance as well. Intelligent architectures accomplish these goals, and any vendor could be there first, if they wanted to.

To continue the conversation, feel free to contact me.

May 16, 2016
Preserving Bottom-Up Innovation in DevOps

Balancing top-down control and oversight with bottom-up innovation and autonomy

The driving force behind DevOps adoption in the enterprise is the idea that it will enable faster and more frequent delivery of better quality software. If all goes well, DevOps should result in higher customer satisfaction and lower costs, but the journey to get there can be challenging. Unfortunately, only a minority of organizations that attempt to implement DevOps actually fully deploy it.

There are many barriers to overcome before organizations can realize the benefits of DevOps. One of the main challenges is retaining sufficient top-down control and oversight without hampering bottom-up innovation.

From Start-Up to Enterprise

The root of this problem lies in the fact that many ideas in DevOps were developed in environments with small teams and flexible organizational structures. In such setups, where everyone can see what everyone else is doing, visibility and control seldom are an issue. Adapting this approach to an organization with hundreds of geographically distributed teams, in industries with a lot of regulatory baggage and diverse legacy systems, is an entirely different prospect.

How do enterprise organizations find a structure that enables individual teams to unleash their innovation and maintain a feeling of responsibility without losing oversight and control? If DevOps teams are not empowered to make their own decisions and free to move swiftly, then they’ll struggle to realize the benefits that drive adoption in the first place.

Walking the Tightrope

Individual DevOps teams should be free to make local decisions, but these need to be in alignment with business goals at the management level.

You don’t want to hamper the speed and reliability that automation brings, or the innovation that can come from autonomous decision-making, so you need to find a way to balance tools, processes and team structure to maintain control and visibility. Avoid flying under the radar; instead, aim for maximum transparency in what your DevOps teams are doing. Set goals and measure, because visibility for management will prove that the new approach is working, build confidence and secure more top-down investment.

Connecting the Dots

A large enterprise will have a number of DevOps teams, but they shouldn’t be completely isolated. Requirements such as audit trails, security access logs and compliance controls must be met by all of these teams. There’s little sense in making each individual team responsible for solving the same problems over and over again. Offer a set of standards and tools, but be careful not to shut down choice.

That same logic applies to competence development, so offer your DevOps teams access to best practices and enable them to learn from each other. The size and structure of an enterprise organization does present unique challenges to DevOps adoption, but there’s also opportunity here to benefit from the scale by sharing and collaborating. Sharing and standardizing the right things will eradicate redundancies and free your DevOps teams to focus on improving their products and innovating, so the business derives maximum value from their efforts.

Finding What Works

The right answer will be different for each organization, but there are some things everyone should keep in mind when building or shopping for tooling. It needs to support a large, diverse environment. Look for support for the platforms you use and consider extensibility, so you can add the functionality you need.

A key factor here for orchestration tools—especially those that define and execute your delivery pipelines—is that they must be accessible for management. You shouldn’t need to be a technical expert to dip in and get an overview, or compile a report that provides the insight you need. Ensure you collect diverse metrics and present information in a digestible format so you can find bottlenecks and identify successes and failures and really deliver the best experience possible for the end user.

About the Author / Andrew Phillips

Andrew Phillips is VP of DevOps Strategy for XebiaLabs, provider of software for continuous delivery and DevOps. He is a cloud, service delivery and automation expert and has been part of the shift to more automated application delivery platforms. He contributes to a number of open source projects including Apache jclouds, the leading cloud library, and is a co-organizer of the DynamicInfraDays container community events.

May 11, 2016
Embracing a DevOps Culture, Part 2
In Part 1 of this series, we discussed the fact that more organizations are looking to embrace the “DevOps culture.” However, unlike ITIL, implementing DevOps into an organization does not happen overnight. DevOps is more a philosophy than a procedure, and choosing the right leadership team is as important as choosing the right tools.

In Part 2 we look at other important elements of organizations successfully embracing the DevOps culture.

Without Proper Collaboration, DevOps Initiatives Will Fail

A good test to determine if your organization has truly embraced the DevOps culture is how well your IT department responds when a major incident occurs in the product environment. For example, an e-commerce company’s dramatically slow website, a hospital’s unresponsive electronic medical record (EMR) system, and a malfunctioning airline reservation system. In these chaotic situations, every minute counts and all Dev and Ops hands need to be on deck to keep your organization afloat.

When incidents occur, it’s critical that your organization’s Dev and Ops teams work together to resolve the problem and restore failed service as quickly and efficiently as possible.

Incident Response: The Critical Incident Team

An international company with global operations may have network operations centers (NOCs) in the United States and the U.K. that support more than 50 mission-critical business applications, development centers in China and the United States and a service desk in the United States. When an incident occurs, the company’s service desk is flooded with alerts, ranging from various IT monitoring tools and customer phone calls to open tickets from online users and inquiries from stakeholders. When that happens, the only concern of the service desk is to fix the issue—not tracking down a department head on the other side of the world that may be responsible for the problem.

DevOps should enable an organization to assemble its critical incident team in only a few minutes. The Ops team should know exactly what team(s) to contact for a given incident, and who from those teams is assigned to application support. They should know who is on call based on the time zone and day of the week, and have the correct phone numbers easily accessible.

Next, the team should review the technical details of the issue provided by the incident manager. The universally used DevOps toolset enables everyone to speak the same language, quickly identify the root cause, put a remediation plan together to have approved by the emergency cab, deploy the fix, test it out and confirm service is back up and running.

IT Communication: The Glue That Holds Dev and Ops

To be successful, organizations need to be able to respond as quickly as possible during a major IT incident to limit the negative impact on the business. This is achieved when Dev and Ops works together to create a central communication and collaboration center to ensure proper communication with key stakeholders.

An organization’s ability to effectively mitigate the impact of an IT issue relies heavily on its ability to access and communicate critical information, and to ensure the right people can analyze it and initiate the appropriate actions to keep the business running smoothly. The communication hub equipped with an IT communication solution will facilitate:
- Reaching out to the right on-call people among all the different teams: infrastructure, server, system administration, middleware, network, DBA, QA, support team, service desk and the application developers.
- Because emails don’t wake up people, they will be able to leverage the IT communication solution to reach out to people via multiple channels until they respond (voice, SMS, email, push notification app, paging, etc.), or automatically escalate to the next resolver on the on-call list.
- Providing the right information so the IT resolvers start investigating the issue, identify the root cause and put a resolution plan together without wasting time.
- Get people to collaborate together using the same telecommunication and collaboration tools whatever the time zone and wherever they may be located.
- Contacting a third-party vendor in case the problem is not attributable to the company but caused by an external piece of software.
- Informing the other departments if the business impact grows so big that it affects the company profitability or reputation, such as a cyberattack leading to a data breach for instance. The CEO, legal and marketing may need to be informed to anticipate the consequences.
- Informing end users or customers to limit the number of incoming calls into the help desk.
The benefits of quick collaboration and not just handoff between the teams don’t even have to be demonstrated. Removing the broken pathways, obsolete delivery methods and redundant platforms will improve significantly the organization’s ability to communicate during critical moments. Becoming more efficient and removing the time wasted will automatically have a huge impact on the mean time to know (MTTK) and, in turn, on the mean to resolve (MTTR), hence minimizing the disastrous impacts on the business and keeping the company’s execs sleep at night.

DevOps Culture Can Happen

Embracing a DevOps culture is not a Big Bang change that happens overnight. This is a journey that will be more or less difficult to pursue based on the existing culture and the history and maturity of the IT organization. Even more importantly, it will depend on the level of commitment from upper management to introduce a new culture. Leaders need to embark on this journey keeping in mind that not only will it require new common tools and new common processes, but also they’ll need to convince people of the value of such transformation. To achieve this goal, management should consider IT communication as a key element, which will be the glue that binds Dev and Ops together. And because there always will be room for continuous improvement, the road to DevOps will have no end.

About the Author/Vincent Geffray

Vincent Geffray is Senior Director of Product Marketing at Everbridge with focus on IT Service Alerting & Communications Automation and IoT.

He has more than 14 years of experience in the information technology business, designing, promoting and selling Enterprise IT Operations Management solutions, including Critical Communications, Application Performance Management, IT Process and Workload Automation. He also has international experience as he started his career in Europe. Vincent holds a Master of Science (Mechanical Engineering and Computer Science) and executive certificates from the MIT – Sloan School of Management.

Vincent’s LinkedIn: www.linkedin.com/in/vgeffray

Vincent’s Twitter: @vgeffray
May 10, 2016
Balance: The Key to Successful DevOps

In the realm of IT buzzwords, DevOps is in a league of its own. The pervasiveness of the word is matched only by the variety of its definitions. Over time, much like the parable of The Blind Men and the Elephant, it has slowly taken shape out of the void, albeit in a backwards fashion. Associated terms such continuous delivery, integrated quality assurance and deployment automation have helped backlight the beast’s silhouette, but there is much more to discover.

Generally, DevOps is defined by how it is used rather than what it is. This is helpful for those already familiar with the amorphous nature of the movement, but can be a confusing introduction for the uninitiated. To achieve a better understanding, it is important to grasp the underlying need driving the shift to a DevOps model.

Currently, the world of IT is barreling toward a future dominated by increasingly specialized and inherently complex automated solutions. Software is taking a front row seat in ways never before possible. (Insert here)-as-a-service offerings and the demand for instantaneous and consistent deployment turnaround is at an all-time high. The days/weeks required to shuffle a solution between development and engineering is a luxury that most companies can no longer afford and is less tolerated by customers. This level of expectation is something that the traditional IT org chart was not designed to cope with. Tech companies everywhere have felt the strain of this increasing momentum, with many at a loss of how to handle it. Demanding that teams work harder, faster, longer will only get so far. And inevitably, increased intensity results in a rise in mistakes and an inferior product.

Balance: Dev’s Yin to Ops’ Yang

Clearly, change was and is needed. The path from development to operations needs to be streamlined at a fundamental level. However, dissolving the barriers between two longstanding departments is not without its perils. As any IT veteran will tell you, development and operations each have their own cultures, quirks and personalities. But the differences in each help create the perfect balance.

In its purest form, development is the trailblazer, forging into uncharted territory and unearthing hidden gems. Ever the innovator, Dev values progress and change above all. This constant forward momentum is Dev’s greatest asset as well as its Achilles’ heel. The habitual urge to tinker, tweak and create can lead to frivolous or wasted effort. The results of this misguided effort can be seen in retrograde patch releases, hotfixes and other “band-aid” measures needed to undo the damage.

On the other end of the spectrum, the operations mantra can be summarized with the idiom, “If it ain’t broke, don’t fix it.” Pragmatic to the core, stability and consistency are the guiding forces behind Ops. Change requests are usually met with a Treebeard-esque, “Don’t be hasty.” Taken to the extreme, this mindset can quickly lead to an environment running outdated, unpatched software and end-of-life hardware.

On the surface, the differences between these two groups appear to be on the verge of irreconcilability. However, in this diversity the beauty of DevOps emerges as the strengths of each team balances out the weaknesses of their counterpart. Development is free to innovate and create knowing that their efforts will be guided by feedback from operations. Like stabilizers on a rocket, operations is able to focus on maintaining a reliable infrastructure with development providing the forward momentum. Working in unison, this duo forms a unit far greater than the sum of its parts. And with it, a company is left with an agile and stable launch pad from which to tackle the latest wave of IT challenges.

Admittedly, the combining of development and operations into a single cohesive unit is just a piece of the entire DevOps puzzle. However, it is an important piece that is frequently overlooked. Understanding the forces and motivations at work within your teams is a pivotal step on the road to a successful DevOps transition.

About the Author / Jared Perry

Jared Perry is a cloud engineer at iland, where he is responsible for maintaining the virtual infrastructure and a number of disaster recovery solutions. He holds a Bachelor of Applied Science in technology management and a Bachelor of Science in IT administration and management from Texas A&M University.

May 9, 2016
3 Ways to Become a More Agile IT Organization

In 1960, the average life expectancy of a Fortune 500 company was 75 years. Today, the average lifespan is just 15 years and declining further. So what gives? We only have to look as far as the “Uber-effect” to understand why so many companies fail. Uber very simply and quickly disrupted a centuries-old industry with the touch of a button—making cab service part of the On-Demand Economy, where consumers get what they want, when they want it, with no delays. Today, agile companies are winning.

So, ask yourself, is your company a disruptor or a disruptee? The reality is, most companies are the latter, with IT organizations under more pressure today than ever before. Whether it’s the fear of being “ubered” by the competition, or business demands to pick up the pace, IT organizations face a balancing act of ensuring the operation of core applications and being innovative. But the truth is, innovation and maintaining the efficiencies of legacy systems can co-exist.

While the prospect of competing with born-on-the-web companies such as Uber or Airbnb might seem daunting, there are a few things you can do to become a more agile IT organization:

Automate manual processes

How long does it take your organization to deliver a simple, new feature in an application? Two days? A week? A quarter? Traditional approaches to software delivery are too slow for today’s speed of business. These often-manual processes are fragmented, error-prone and time-consuming. A recent Forrester Total Economic Impact study revealed organizations that automated their app deployments experienced an increase in speed-to-market and reduced development cycle times up to 75 percent.

Take a look at your delivery pipeline and assess where you’re testing and deployments could be automated. Are developers able to easily deploy applications to a test environment, or are they bogged down with unnecessary wait times and manual deployment and configuration steps that are susceptible to errors? If you’re facing a development-testing cycle of two to five days, that’s simply too long. Look to adopt DevOps by automating release and deployment processes wherever you can and aim for continuous delivery, so you can push apps out quickly and be more agile.

Lean and Mean Equals Agile

No company will risk its core systems of record for the sake of innovation. At the same time, you cannot invest in becoming innovative if the delivery and maintenance of your core systems are inefficient. The goal is to make these core systems as lean as possible by automating repetitive tasks, freeing resources that can be directed toward speeding innovation. Today there are cloud technologies and services and API management systems that can help organizations streamline the maintenance and operations of these core systems and expose their services as APIs for consumption by new applications.

Create a multispeed IT process

While you’re making your core systems as lean and agile as possible, create a second layer, where innovation and agility are the key focus. As your core systems are exposed as APIs, you can begin to use their business functions and services to experiment and innovate. Cloud-based development platforms allow organizations to stand up complete development, test and production environments quickly to incubate and deliver new, innovative applications that consume these services from the core systems, delivering new and innovative business capabilities in a fraction of the time traditional methods would take. These innovative apps can be delivered and decoupled from the speed of delivery of the core systems, enabling what is known as multispeed IT.

While the status quo might seem like an easier option, by doing nothing, you risk not only being passed by the competition, but your line of business leaders may bypass you altogether, turning to “shadow-IT” and outside vendors to meet their needs. While this hypergrowth of the tech industry is unprecedented, it is not a fad. The pace of innovation will only get faster and your only options are to fail or adapt and be agile.

About the Author / Sanjeev Sharma

Sanjeev Sharma, CTO and Distinguished Engineer – DevOps Technical Sales and Adoption, IBM Cloud Unit, is a 20-year veteran of the software industry with expertise in DevOps, mobile development and UX, lean and agile transformation, application lifecycle management and software supply chains. He is a DevOps thought leader at IBM and speaks regularly at conferences. He has written several papers and is the author of the “DevOps for Dummies” book.

Connect with Sanjeev LinkedIn | Twitter

May 5, 2016
Humans in the Machine of Release Orchestration

Let’s face it: Most organizations are still struggling to achieve the Nirvana of full gated automation. At first, we automate the core of our DevOps services, the build and the deployment disciplines. Over time, we realize that a templated or framework (or pick your similar terminology) approach toward automation allows for scale as well as predictability, lowering costs. It is here where the lion’s share of variability can be reduced and scalability achieved.

The next culprit to hit the radar is usually test automation, but this tends to be where the rubber meets the road for the level of work it takes to automate process. So generally minimal work gets done, ascribing to the 80:20 rule that 80 percent is quickly attainable and should be sufficient, the remaining 20 percent is just too hard. But having gotten this deep in the DevOps maturity scale, release orchestration (RO) is usually next up in our continuum of services.

The automation of release orchestration is a different animal than the core of build, deployment or even test automation. Release orchestration is more about automating our process of change management. Invariably the spreadsheets or SharePoints of yesteryear are intended to give way to our preferred vendor’s methodology of reinventing the release wheel in their proprietary toolset (albeit with better results expected). But usually this is the wrong approach as well.

Changing Human Thinking About RO Construction

As I continually repeat, DevOps is about thinking differently about “how we innovate.” (If you need more strategic leadership help in this regard let me know, I am looking.) Consider that our spreadsheets and SharePoints, while impressive, constricted us to thinking about release and change management within the architectural constructs or limitations of the spreadsheet and SharePoint tools. They were task-oriented, less interactive, top-to-bottom lists of what we thought we needed to do in any given release. If we were to truly go back to the drawing board and thought about creating a release without those constraints, or just with the idea of parallel tracking, what might we accomplish?

Digesting the release process with a lean mentality breaks down large sections into more manageable subsections. This reduces risk, and increases speed, particularly if it is coupled with parallel execution. If there truly is a sequential dependency in our tasks, let’s call it out, but when we do, we must ask ourselves whether that dependency is a go/no-go for all activities, or for only some of them. If it is the former, so be it; if it is the latter, then perhaps we can adjust our sections to account for this and run more of them in parallel. Just implementing a parallel mindset can radically reduce the time it takes to run a release from end to end. But we must also ask ourselves, What other key features of any preferred release orchestration toolset can alter how we think?

Changing Human Thinking About RO Execution

The other axis upon which to digest release orchestration is in its real-time interactivity. Consider for a moment that some tasks either cannot or will not ever be fully automated. Some tasks will remain in the province of a human to fulfill. For example, (and I cannot imagine why, but humor me), if we needed to install additional hard drives in bare metal servers during a release, the physical installation would always require a human to do so. With old release orchestration technology we must wait for shared spreadsheet files to be updated by the engineer when he/she completes this task (or any other manual task example you can think of).

With new mobile release orchestration technologies, the engineer could mark them complete (or percentage complete) in real time on his/her mobile device while still in the data center, standing in front of the hard drives/server. How much time does this save, in pure walking to his terminal, logging in and updating a file, emailing it to the change manager, merging it with the current one, etc.. A simple swipe and done. The engineer is now able to provide a great deal more information to the team in real time, and with minimal effort. This gives real-time visibility to the release manager such as has never existed before.

It can also grant them a level of direct interactivity. Imagine the hard drives’ physical installation has been completed, but manual testing is failing. If the engineer has no good way to share the information with the team, risk for overall on-time completion cannot be properly assessed. Proper tooling avoids the scenario where change and release managers are waiting on a task, and drill-down activity is unavailable (picture a spinning icon, rather than a percentage complete on specific task item). Add the dimension of red-yellow-green to track the completion of tasks, whether parallel or not, and we have a dashboard that calls our attention to the problems impacting a release in real time. This is far less possible with older RO technologies until somebody checked their watch and started making phone calls.

Changing Human Thinking About RO Valuation

Now every organization dreams of a release containing nothing but virtualized full-stack deployment for every app (even the legacy ones), where everything from bottom to top is automated during the deployment process. However, some of us living in the real world, still have a few database scripts that must be run manually during a production deploy (don’t ask me why). Or pick your poison in the real world you are contending with, often infrastructure and platform tasks are the last to be fully automated.

So during a release, engineers need to be on hand and ready to go when called upon. If however, we now all subscribe to a tooling solution that acts as a common backbone for our release orchestration … the engineers know exactly when to do something, if they can do it early, and when they can go home afterwards. There is the value proposition for the engineer, less time spent doing tedious work during a weekend, when they would otherwise prefer to be at home.

The value proposition to the release manager is real-time insights into all this activity during a release. The automated tasks flow through just as you would expect them to, with up-to-the-minute status of their progress until completion is reached. The remaining manual tasks provide as-good-as-it-gets status updates, also in real time, and serve as the next target of opportunity for automation prior to future release(s).

Changing Human Thinking About RO Perspective

One thing that always seems to be missing from a number of vendor’s offerings I have reviewed in the release orchestration marketplace, is a view from human centricity. Most release orchestration tooling is obsessed with task management, integrating it with calendar scheduling and designing it for maximum efficiency in getting the next step done. If however, our vendor community would embrace the idea of intelligent calendaring a bit more (a future article will discuss this characteristic more in detail), the offerings could form a more human-centric view of the release orchestration process.

Imagine what it would be like for either an engineer or a change manager to log in to the release orchestration toolset and have a personalized dashboard outlining the tasks that need to be approved, worked on or completed to support the all pending releases for that user (based on integrated workflow and intelligent scheduling). An automated red-yellow-green attention draw, for work that may be behind, on schedule or completed early, would greatly improve the human experience of release orchestration. And an integrated departmental or team-focused view of the work effort from manager or employee-centric views would add to job performance critiques—proof of superior, or inferior performances over time.

This represents only a simple recognition that humans should not be stuck in the machine of release orchestration tooling, slave-to-a-machine-centric view of actions to be completed. Instead, a vendor could recalibrate a more human perspective—a subtle alteration in perspective adds an exponential lift to the value of this release orchestration toolset, and it seems to me, any vendor could be there first … if they wanted to.

To continue the conversation, feel free to contact me …

May 4, 2016
Does the Future of DevOps Lie in NoOps?

Why DevOps is more a bi-directional rather than a simple top-down approach

DevOps is one of those buzzwords that nearly everyone wants to discuss these days. However, it often is interpreted as an extension of Development’s influence to the detriment of Operations teams.

So much has changed in the past few years that it is conceivable we will see another major shakeup: Could Operations teams disappear altogether? Or to put it another way, is “NoOps” the future of IT? While DevOps promotes faster delivery of new digital services and a blurring of the lines between Dev and Ops, in reality, is Dev consuming Ops entirely?

Before DevOps was called DevOps, the two disciplines were growing further apart. Agility has been driven by Developers since the turn of the century, with the famous Manifesto for Agile Development. Meanwhile, Operations formalized repositories of good practices into frameworks such as ITIL or COBIT, which were implemented by most large companies in the mid-90s. Those two approaches seem to oppose, as DevOps advocates the acceleration and multiplication of changes and ITIL tends to monitor and control them.

NoOps, or the End of Operations?

If you believe everything you read, then Operations is a dying art. But in practice, Development does not want to be involved with infrastructure, while few organizations would trust developers to manage their data centers, whether physical, virtual or cloud. So breathe easy: DevOps does not mean the end of Operations teams.

On the contrary, Operations remains essential throughout the DevOps approach. Production infrastructure is a complex and delicate stack of technologies, often with a discreet mix of modern architecture and historical applications. It often is impossible for Development to replicate these types of environments. And that is why an application successfully tested in a qualification environment often does not work properly in production. However, it is crucial to validate new developments in a representative replica of the production environment.

So is DevOps doomed to constant failure because of lack of continuity between qualification platforms and production systems? Well, this is probably the case if we merely consider DevOps as a simple top-down approach from Development to Operations. Focusing on producing changes at a higher frequency only shifts and grows a bottleneck that sits in production, without producing the added value expected by the company. As a consequence, DevOps must also be seen as a bottom-up process. In other words, if we consider DevOps, we must also take very serious look at “OpsDev” (OK, it doesn’t flow off the tongue, but you see my point!).

DevOps or OpsDev?

For a successful DevOps approach in practice, Development must position itself as a consumer of turnkey infrastructure environments. Operations then adopts an OpsDev approach and provides infrastructure on demand for all steps of continuous integration, from compilation to qualification, through unit testing.

If DevOps is a radical change in how Development works, OpsDev would also revolutionizing common Operations practices. It comes with more agile “declarative” infrastructures (described and built from source code), an even more sophisticated level of automation and the ability to provide self-service infrastructure for developers.

The adoption of DevOps does not mean that Operations will disappear or will be replaced by Development, but rather the emergence of bi-directional agility that needs to be supported by more sophisticated automation. In fact, DevOps implies OpsDev rather than NoOps. Finally, whatever you consider your release process under the ITIL or DevOps angle, isn’t the end goal exactly the same: delivering zero-default service to customers?

April 28, 2016
The Nirvana of Gated Progression

A highly functioning enterprise-class DevOps set of services has a fundamental goal of full automation. To be honest, having the right DevOps strategist (or visionary) leading this effort is as important to achieving this goal as the combination of all the challenges and topics that must be addressed. So for the sake of argument, will we assume you have one (if not, feel free to contact me—I am looking at the moment 🙂 ).

With key leadership in place, the first target is always the automation of the build and deploy process. This could easily be considered the core of DevOps or its heart. But once this automation has been achieved, the trickier topic of release orchestration begins to emerge. Strategy is to release orchestration what taste is to food: Having the right one does make all the difference.

To progress our code to production, the inherent assumption is that all testing has been completed successfully. Sounds like common sense. But then, how much of that testing occurs with human hands on it? The user acceptance testing (UAT) discipline inherently requires a human at the wheel. But the reality is, for every time a human must interact with the progression of code from the development-class environment through all the testing environments (integration, performance, user acceptance and QA) on its way to production, the momentum of innovation is slowed down—all the way down to human levels.

The Complexity of Testing Types

The first automated testing begins to occur in development or (DEV) class environments, where the programmer is under strict instructions to implement test-driven development. This requires new feature/functional testing as part of the build or deployment automation into DEV. Either the build or the deployment cannot be said to be successful “if” the accompanying tests are not “passed.” This also represents our first “gate.” Sophisticated release orchestration (RO) software can examine “flags” set during the build or deployment process to see if the testing has been completed successfully. If so, the code can progress to the next testing level environment within the software development life cycle (SDLC).

Another useful type of testing completed early in the SDLC (most often in DEV) is the examination of code coverage. Development organizations can begin to examine how much of their code is put to the test, and set arbitrary percentages as a minimum standard before the target application is able to progress. The addition of alternate types of testing occurring in DEV class environments does add to the quality of the innovation effort. However, it brings with it an increased labor effort to automate both progression—and audit—of the version under test.

For instance, feature/function testing may use a single repository to store the results for a given version of the code. I go to my testing repository and I can look up the results from version 1.4 of my application, or version 1.5 of my app, as many as I desire. Should a regulatory official or internal auditor ever ask me to prove the quality of version 1.5 of my app, I need only consult my testing repository and show them how many tests were executed, and how many passed or failed before this version of my app was ever deployed into production.

When you add code coverage testing to the process, I now need to consult two different testing repositories, producing two separate reports to assess the quality of this same version of my app. There is nothing wrong with this, but it does create more work or effort to produce than it would if a single testing repository was used. The problem only increases as we keep increasing the types of tools we use to prove quality. Security testing, for example, (ensuring code does not have inherent security holes) tends to use different tools, or, worse, processes that require human peer observation, which again can add effort to create the automation and the traceability for audit. These are critical types of testing (and/or processes) so they must be done, but they also come with an inherent cost in automation effort.

Small shops may not worry about these kinds of concerns, but large organizations will, particularly if they are still in the process of transitioning from legacy waterfall methods (where controls were high) to new agile methods (where it feels to auditors like the wild, wild west). Compiling a test log that matches the version of code under scrutiny such that it can be reproduced automatically will restore the faith of both internal auditors and regulatory officials who come asking. But the addition of each new type of testing tool will require more effort to maintain the ability to compile those test logs.

The Complexity of Testing Responses

Next, consider the impact of non-binary responses to automation efforts for gated code progression. Does my feature work (yes/no)? This is a binary evaluation, a true/false or yes/no that is easy to set my progression flag for testing response. It is another matter to evaluate the results of my code coverage and ensure this app is above 70 percent or between 70 percent and 90 percent. These kinds of questions are not binary (in other words, there is a real result to compare to, not just a pass/fail), and require further effort to set a binary flag for the gated pass/fail. When the evaluation question does not intend a binary answer at all, such as a grading of 1 to 5 where 1 equals horrible and 5 equals outstanding, I am now most often forcing a human to make an intelligent decision as to whether this code should progress.

Multiple testing repositories require additional automation effort. Multiple kinds of responses require additional automation effort, and the standards by which the gates pass or fail also may change over time, requiring tracking of the goals at the time the tests were taken. Creating independent test automation that can be called from the build process or the deployment process is wise. Associating this test automation (whether scripts or simulations, etc.) with the version of the application code is wise. And working to reduce the number of repositories to examine is wise. Having de-coupled test automation strictly from a build allows it to be re-executed in more advanced testing environments when the need arises.

Charting the Course to Nirvana

RO benefits from as full a level of automation as possible. I will discuss the RO topic more fully in another article, but the foundation of automated change progression relies on a gated history of proving quality. As long as the code under review continues to pass testing, setting progression flags to green or yes or pass, the RO software can continue to progress it along its way to production. Failing code must be sent back to the DEV environments where development teams can address the issues before making its way back through the automated process once again from beginning to end.

If the organization has made the intentional decision to keep humans substantially involved in the testing and evaluation cycle, it must realize a few impacts of that decision. Daily movement to production becomes less likely—most often it takes a week or two for changes to undergo significant human scrutiny (depending on the level of testing automation). The results of human intelligent decisions also must be chronicled in the testing repository to preserve the history for audit. This can lead to a “blame game” kind of thinking when something goes wrong, as it is immediately attached to “who” approved it, instead of to “why” they did so. And obviously the cost of labor is always the highest cost element in any innovation cycle, so buyer beware in this scenario.

On the other hand, “Nirvana” is the state when all testing (except UAT) is fully automated and gated, and provides the foundation for RO software to move code into production as often as it can prove quality. This could mean fully automated releases instead of release weekends. This could mean multiple releases per day into production. This could mean that releases themselves are no longer considered “events,” but by default are now “non-events.” Nirvana does not endanger the lives of change and release managers—it actually gives them one. For a “change” (pardon the pun), they get to discover something they vaguely remember … a life outside of work.

To continue the conversation, feel free to contact me.

April 26, 2016
Infrastructure as Code is SDN for Ops

As we continue to recognize the importance of DevOps—particularly its automation and orchestration aspects—to achieve the agility necessary for continuous deployment of apps, the question of how the term “Infrastructure as Code” (IaC) applies to traditionally network-bound systems and services is bound to crop up more frequently. After all, in addition to core IP-based networking characteristics required for an app to communicate and serve up content, an entire ecosystem of app services that support, secure and scale it must be considered.

IaC is driven by the application of development principles to network infrastructure. Configuration, scripts, templates, profiles and the like are treated as code, in that they should be reviewed, versioned and provisioned from a centralized repository (one hopes on-premises) rather than managed as discrete network elements.

In a recent article in TechBeacon, Boyd Hemphill, director of evangelism at StackEngine, noted, “The basic principle is that operators (admins, system engineers, etc.) should not log in to a new machine and configure it from documentation.”

Christopher Null, CEO of Null Media and author of the article, then wrote, ” … But IaC is a concept that extends beyond simple infrastructure automation. IaC requires applying DevOps practices to automation scripts to ensure they’re free of errors, are able to be redeployed on multiple servers, can be rolled back in case of problems, and can be engaged by both operations and development teams.”

The idea is that you can describe any given service as a set of artifacts and deploy (and redeploy) them at will. So rather than set up a system and configure it directly, its configuration any associated artifacts (such as profiles or templates) are treated as code and stored in a common repository. A controller—the orchestrator—is responsible for distributing those artifacts when necessary. The controller uses an API (or protocol such as OpenFlow) to manage the configuration of services across a distributed infrastructure. It’s decoupling the service definition from the service and its platform.

Read that again. It decouples the service definition from the service and its platform and assigns responsibility for provisioning and configuration to a controller. Kind of like decoupling the control plane from the actual forwarding and switching actions (the data plane) and assigning operational responsibility to an external controller.

At its simplest, treating infrastructure as code is separating configuration from the platform and enabling a software-driven or software-defined approach to provisioning, configuring and managing infrastructure. It’s applying the principles of software-defined networking (SDN) to operations, rather than to devices and systems.

But as Null makes clear in his article, IaC goes beyond just automation and orchestration, adopting the best practices of development with respect to code reviews, versioning and management of that “code” to ensure consistent, predictable and repeatable deployments.

It’s all driven by software, ultimately, and deployed by software, and defined by software but always under the auspices of the operational experts: the engineers and architects.

Treating infrastructure as code is basically applying the architectural principles of SDN to operations itself. It’s SDN for ops.

April 22, 2016