devopsy.kinsta.cloud

Category: Enterprise DevOps

Understanding Mesos and Its Place in DevOps

When building a house, work is done in teams. This work is managed by a foreman, supervisor, or general contractor, but let’s restrict ourselves to the teams doing finish work, and a foreman. Foreman is old fashioned, but general contractor gets long to type over and over. Assume a large enough house to warrant a couple of teams putting in cabinets and trim, hanging doors, etc. A team of two tells the foreman “We’ve finished with the master bedroom”, and he looks at his list and says “Good, go install the trim in the master bathroom.” And the two head off to do just that. About half way through, one of the installers has an unfortunate incident with a nail gun, and nails his hand to the floor. Both inform the foreman that they have to leave – one to go to the hospital, the other to drive.

When the other finishing team comes free, the foreman says “The master bathroom trim isn’t done, go take care of that for me.” And keeps his checklist up to date.

Eventually, the entire house has the finish work completed, simply by the foreman sending teams off to do different parts of the job until all parts are complete.

That’s Mesos, essentially.

The foreman is, in a nutshell, Mesos in this example.

Mesos is a scheduler that knows what jobs have to be done, and what resources are available to do them. The details of getting the job done are not Mesos’ concern, but if the job is not completed, Mesos will send other resources so that it can get done.

There is a lot more involved, of course, but the analogy holds pretty darned well.

Mesos turns a collection of machines into a cluster, simply by placing agents on each machine that know how to (a) report what resources are available for Mesos to dole out, and (b) execute applications for a given environment (called framework in Mesos-speak for a variety of good reasons). So once Mesos is distributed across your servers, it can effectively treat those servers as one master server, and itself as the scheduling algorithm.

It’s never simple. Because IT isn’t simple.

I have an adage for IT though: No good idea is complete until the real world imposes complexity on the simple.

In order to make Mesos fault tolerant, there can be multiple masters, and zookeeper can be used to make certain that if the current master fails, there is a replacement ready to pick up and keep things moving (processing).

In order to keep Mesos focused on scheduling, it does not have complex resource scheduling built in. Instead, it relies upon frameworks asking for resources, and then offering them what fits the requirements. So let’s say Hadoop asks for five CPUs. Mesos may say “Here are three”, then a few minutes later say “Two more!” and Hadoop could tell the executors on those servers to start up tasks. Frameworks can also tell Mesos what they don’t want (filtering), and reject offers of compute resources Mesos makes (in which case those resources are offered to any other frameworks out there).

Of course, those frameworks that know both how to schedule jobs and how to send them to agents are key. Marathon can handle most long running tasks, Hadoop has a framework of its own. So the “subcontractors” are ready to get the job done, for the most part.

So what is the benefit? Mesos doesn’t much care where you install it. Put it on hardware, put it on VMs, put it in a VPC. Using Marathon, you can deploy your apps in containers, meaning you really don’t have to care what the underlying infrastructure is.

What’s the downside? Well, there are a whole lot of tools to learn to make a production environment stable and maintainable, and there are other alternatives out there that we’ll be looking at in the coming weeks. Honestly, most of the “other alternatives” are expansions based upon Mesos (and often Marathon), so this was a good place to start in a series exposing what you need to know to consider if Mesos fits your environment. If you want to start running apps and seeing hardware/VMs as a pool of available resources, Mesos is a good place to start, though the packages we’ll talk about in the coming weeks are more interesting, mostly because they take Mesos’ simple scheduler concept and expand upon it to offer a more complete solution.

If you would like to learn more about Mesos, The Apache Foundation, DigitalOcean, and Mesosphere all have tutorials. I have used the Mesosphere one, and it was pretty good, though on OSX it failed, it worked like a charm on Windows 10.

— Don Macvittie

December 1, 2016
7 Best Practices New MongoDB Users Should Know

Being a developer in today’s technology landscape is tricky. While the proliferation of different cloud services makes it faster and simpler than ever to turn ideas into reality, this rapid development comes at a price. Developer responsibilities have grown to include not only product/feature development, but also other application-critical tasks such as configuring and managing a database deployment, in this case, MongoDB.

While this guide looks at MongoDB databases specifically, the broader (and proverbial) fork in the road for developers is this: Manage a production database deployment yourself or go the database-as-a-service (DBaaS) route. A DBaaS manages the database on a developer’s behalf, so the developer can focus more on feature and product development operations instead of database ops. However, sometimes this isn’t an option. If you’re embarking on a MongoDB journey and don’t use a DBaaS, consider this checklist to ensure you have the best practices down before getting started.

Host Your Application in the Same Data Center as Your Database

Your application infrastructure and your database should be located in the same local network (i.e., data center/cloud region), as it will be the most secure method of deployment and will minimize latency between your application and database. For example, if you are hosting your application in Azure West US, you should also host your database in Azure West US.

When you connect to your database from within the same data center/region, you communicate through your cloud hosting provider’s internal network. All major cloud hosting providers provide a good deal of network security infrastructure to isolate tenants. The hypervisors they use do not allow VMs to read network traffic addressed to other VMs; thus, no other tenant can “sniff” your traffic.

Use a Replica Set for High Availability and Review Deployment Best Practices

MongoDB Replica Sets synchronize data over multiple redundant servers and protect your application from downtime in the face of node failure. Should your primary node fail or become unreachable, your application will automatically fail over to a secondary node, ensuring the availability of your database.

AWS users should take special care to deploy their replica set nodes across separate Availability Zones to protect against zone failures.

Not all workloads have the same requirements, so there are multiple strategies for deploying a replica set. You can find a list of considerations in the MongoDB documentation here.

Host Different Environments on Separate Hardware/VMs

In addition to the production environment, you may be responsible for setting up environments for development and staging applications. Create a separate database deployment for each of your application environments, with each deployment running on its own VM or hardware to protect against resource contention (CPU, RAM, etc.) across your multiple environments.

Enable Database Access Control

MongoDB does not require authentication by default. Protect your database against unauthorized access by enabling database access control. Enabling access control also allows you to limit actions that particular users can perform on the database.

Use an Application Driver Maintained by the MongoDB team

MongoDB is a very popular database technology with a large community behind it. As a result, many third-party drivers have been created. However, third-party drivers may become abandoned or outdated over time. It’s critical to find a driver that is well-maintained and stays up to date with the latest MongoDB features.

If possible, use an official driver by the MongoDB team. You can find a list of drivers that are maintained by the MongoDB team here.

Index Early and Often

Without question, the most common database performance issue is due to improper indexing (or a complete lack thereof). Proper indexing is critical because even one unindexed query is enough to prompt significant performance degradation.

If you’re unsure about indexing best practices, we’ve created a list of indexing tips for improving your MongoDB performance.

Schedule Regular Backups

While replica sets can offer redundancy to protect against system failure, they will not protect you from accidents caused by human error such as dropping a collection or even your database. Backups are crucial for ensuring you do not lose valuable data.

Two common backup strategies are to take filesystem snapshots or to use the mongodump utility. Schedule frequent, recurring backups to keep your data safe!

About the Author / Chris Chang

Chris Chang is a developer advocate at mLab. At mLab, Chris has provided database support to thousands of developers. Previously, Chris has held IT roles at VMware and NetApp. He is interested in developer tools, fitness, and house music. Chris holds a B.S. in Computer Science from the University of California, San Diego. Connect with him on LinkedIn and Twitter.

November 21, 2016
DevOps Ascending: Guardrails, Not Guidelines – DevOps For Digital Leaders

The season of DevOps is upon us, if not only related to widespread adoption, then certainly in terms of industry emphasis.

This week, at the CA World ’16 conference, my colleagues and I will be interacting with thousands of practitioners, all of whom have evolving DevOps narratives. Meanwhile, CA was a founding sponsor of the AllDayDevOps conference, held earlier this week, attracting more than 12,000 registered attendees worldwide.

These events come on the heels of a successful DevOps Enterprise Summit held last week in San Francisco, where many of the industry’s brightest minds gathered to discuss and debate the current state of play. If activity is a reliable indicator of relevance, the growing noise around DevOps adoption has been amplified into a deafening roar.

All of this stands as further evidence that DevOps is no longer an emerging phenomenon, but rather a ubiquitous movement transforming the manner in which nearly every organization builds and operates software.

As a result, the DevOps movement is finally ready to move from initial framing of emerging techniques into the domain of defining best practices. Yet, at the same time, given the fluid nature of DevOps itself and its unique correlation with the individual needs of every business and environment, even this notion of best practices is one that mandates flexibility.

Increasingly, we hear leading experts in the field espouse the need for these practices, but those that serve as “guardrails” that set useful parameters for DevOps adoption and refinement rather than concrete guidelines that dictate specific requirements.

Such is the tone and goal of the new book that my co-authors Kieran Taylor, Peter Waterhouse and I released this week to help practitioners understand common DevOps challenges, methods and milestones.

Titled “DevOps for Digital Leaders,” the book examines the genesis and increasing maturity of DevOps, all with the explicit intention of helping to inform further refinement among today’s organizations. After reading it, I’d argue that any practitioner, from total neophytes to hardened DevOps vets, should walk away with a greater understanding of how to bring today’s best practices to bear within their own work.

Beyond looking at precisely how DevOps has grown in response and parallel to use of agile methodologies in the world of manufacturing thereby transforming the modern software factory, the book offers specific examples of leading adopters that are using these techniques, and related tooling, to re-engineer the very DNA of their business.

From the core competencies of building, testing and deploying applications to continuous monitoring and management of software to help stay attuned to changing customer expectations, we sought to offer a practical guide to DevOps transformation. Other chapters touch on finer points related to key topics including creation of success metrics and measurement of ROI.

In one of the analogies invoked throughout the book, we highlight how DevOps-empowered software factories are not unlike the engineering, design and race day teams present in the world of today’s Formula One racing series.

Pushing themselves and constantly innovating on the fly—clawing for every second in the name of winning the overall title—the comparison fits nicely as we watch this season’s contenders fight it out down to the final race in Abu Dhabi later this month. As with winning customers in today’s applications economy, clearly every little bit counts.

Again, no matter what—or how much—you think you know about DevOps, we’d like to think that there are tactical “learnings” sprinkled throughout the book that will broaden and deepen your understanding.

Download your copy today, and let us help drive you down the road to victory. Alternatively, you can order a hard copy of the book via Amazon, with all proceeds being donated by the authors to fund Foundation for Excellence, which provides STEM scholarships to children in India.

— Aruna Ravichandran

November 16, 2016
Accelerating DevOps with Service Management (and Vice Versa)

One of the great misconceptions of service management is what it means. This also is the reason why so many service management initiatives fail. Consider a call center, for example, with many people answering phones and replying to emails. It’s true that a service desk is one of the functions (and in some organizations, maybe even a cornerstone) of service management. It’s also true that the customer or user is one of the main characters in the story of service management. The very word “service” implies there is a customer at the end of that service.

However, service management is, at the very core, about designing, delivering and redefining services to the customer base—whether those are internal, external or occasional users of the service. User roles may vary over time and users may become part of the supplier chain, and vice versa. A great example of this is knowledge management, wherein users are typically both consumers and producers of information and, thus, the service.

When we take this higher-level view of the service management practice, our eyes open to new possibilities. In particular, we can see beyond the service desk. The scope of service management widens to the end-to-end delivery of the services in question, retaining the focus on the customer.

Finding the Cure: It’s All About the Customer, and Communication

The happy medium and the much sought-after goal of the harmonization of service management and DevOps can be achieved when stakeholders find common drivers. In both worlds, the respective goals focus on the business, delivering value and doing so quickly.

The following sections present actionable recommendations that can be implemented incrementally across the entire organization. They offer a bottom-up approach. In conjunction with implementing them, senior management needs to support the culture change that is a vehicle, a catalyst and a byproduct of the improvements. In this way, the full benefits of the changes can be analyzed and achieved.

Tip 1: Stop playing ping-pong with issues

We’ve all identified an issue with a system, logged it with IT as a ticket and received updates of the progress while the ticket is handed over from team to team—only to discover weeks later that the development team knew about the bug but had de-prioritized it due to lack of interest.

Surely, by the time the final bit of information reaches users, they are less than satisfied with the result. What’s more, the organization has wasted everyone’s time, not least the time of network, storage and other team members who had to prove it wasn’t their fault in the first place. Finally, development team members could have prioritized their work correctly if they had known about the customer need in the first place.

With so many implications in the above scenario, why does this happen? The reason is simple: less than optimum process interfaces and lack of information flow. When you’ve identified the problem, the solution is easy: Include relevant data from the development teams, such as defect information, in your chosen IT service management solution.

Sometimes, the easiest way is to integrate the defects created in development’s chosen tool set to IT service management’s “problem” records. This way there is no need to change the processes per se, as problem records can be linked to issues (or incidents) easily in the service management tool. The integration simply allows for the information to flow between systems and enables a collaborative approach to issue handling.

In this scenario, when the development team opens a defect, it creates a problem record in the service management system, which, if feasible, could be displayed automatically to users on a self-service portal. Thus, users can search known errors before logging a ticket, resulting in time savings across the whole organization.

Tip 2: Communicate bugs and known issues to everyone impacted

But what happens when a user finds a new issue that has emerged due to a bug that has yet to be discovered by developers and testers? Traditionally, the user might log a ticket, which is then analyzed by the IT service team and forwarded to developers for analysis, coding and testing. That might be last of the ticket the user sees, as the feedback loop often is missing.

The benefits of forwarding the defect information from the user to developers are realized only when there is a way to easily track the whole life cycle of the issue regardless of where it has been initiated. Instead of batch importing records from one system to another, the most advantageous way is to mirror the record sets in the two systems.

In this example, the feedback loop between the customer and the service desk to the development team is achieved when synchronizing problems and defects in the two solutions.

Tip 3: Empower the customer by crowdsourcing bugs and new features

One key metric that contributes to employee satisfaction and retention is the ability of employees to do their jobs well. To accomplish this, they need resources. When employees have all the resources they need to do their job well, more often they are happy than not.

We are all dependent on technology; there are few professions that do not require technical resources to perform their jobs. If there is a hiccup in the finance solution, for example, the finance assistant may well be the first to find an issue after a version upgrade. Or, a team-wide brainstorming session may result in some great “if-only-our-analytics-software-could …” ideas of how to improve the efficiency of the team.

But how best to communicate these issues, and capture the many ideas? Possibly, you might even perform real-time impact analysis on which improvement ideas are important to various business processes, and communicate the urgency of each one to the development organization. The inherent problem in many organizations is that users have only one touchpoint to the “DevOps cycle,” most commonly through the IT service management practice. Meanwhile, the rest of the IT organization may use a number of disparate systems to communicate and assign current work tasks that form a basis of shipping new features and functionalities.

Again, it may be useful to look at the tools already employed throughout the organization, and understand the existing processes and practices. Understanding team functions is important as well, though many teams are fluid in their roles and tool sets used. Once relevant tools are identified, look for mutually beneficial interfaces between them to ensure that users’ requests for new features are not forgotten but logged and picked up by the development team.

In this example, you could integrate a tool that supports the back office (such as a defect management tool) and tools that support users (such as requirements management and project management tools). This allows you to effectively crowdsource and track upcoming features and bugs, keeping the whole organization informed.

Tip 4: Don’t forget suppliers

Include the entire software supply chain in the equation. After an era of outsourcing, insourcing, and multisourcing, many organizations have a diverse supply and sourcing strategy. One of the main reasons for this is to manage risk and ensure business continuity.

Thus, the last remark is to ensure that all the above examples work in a multisupplier environment. This typically involves contractual preparation. New frameworks such as service integration and management (SIAM) help identify risk and reward in supplier collaboration. These are useful tools for both IT service management and DevOps practices.

A great example of supply chain integration is an end user reporting an issue. Once it has been reported, it is then centrally analyzed to find the root cause of the issue and the underlying problem is routed to the correct supplier automatically. By synchronizing the information between supplier and customer, not only is each party kept up to date about current issues, but the pipeline and project customer demand are built by integrating incidents, projects and requirements across the software supply chain.

One of the core aims of both service management and DevOps initiatives is to respond to customer demand as quickly and effectively as possible. Real-time synchronization between solutions is merely a technical solution, but one that has true potential to elevate the response speed.

Reflections

Having examined the above four ways to improve the collaboration of DevOps and service management, there are clear solutions to reforming the partnership to drive quicker customer response and satisfaction. But be forewarned: The four recommendations offered are but advice to improve; an organization’s leadership must truly be committed to a culture change to make change happen.

About the Author /Tuuli Bell

Dr. Tuuli Bell is a Partner Account Manager, EMEA at Tasktop. She is a member of the BCS, the Chartered Institute for IT, and the Institute of Physics. For her, happiness is about discovering opportunities to challenge the status quo and about embracing change, be that organizational change, or professional or personal development. Tuuli’s interests outside work include art, science, and nature. Follow her on Twitter or find her on LinkedIn.

November 16, 2016
You’re Not Doing DevOps if You Can’t Pull the Cord

There are many concepts in Lean manufacturing that have applicability with software development “factories.” One example is the Andon Cord.

The Andon Cord concept involves using signals and alerts to indicate that a station on the assembly line has a problem. Alarms are activated manually by workers pulling a physical cord or automatically by the equipment itself.

Developed as one element of the Jidoka quality method and pioneered by Toyota and the Toyota Production System, the Andon Cord empowers workers to notify co-workers and management of quality and process problems across the assembly line—and even stop production.

While pulling the cord to stop production sounds crazy, if a defect is found and can’t immediately be fixed, then pulling the cord means stopping the problem from flowing downstream, where it will be costlier to fix and have a severe impact on the customer experience.

But the point of Andon Cord pulling isn’t only to fix problems. As part of a continuous improvement program, much of the “pulling power” comes from how it helps teams build quality into products and processes. When production stops, improvement begins.

Many would argue that the application of Lean management concepts such as the Andon Cord is more challenging in software engineering. With software, there are many more opportunities for defects, and testing for problems only occurs late in the development cycle. So late, in fact, that in some cases customers end up doing most of the testing.

But as the digital and physical world becomes inseparable, excuses and outdated practices are no longer acceptable. Just as we wouldn’t ask a customer to test air bag deployment in a car, we shouldn’t expect them to test whether a new software-enabled telemetry system is subject to security hijacks.

So having the power stop software development production line to prevent defects, security problems and safety issues during the actual development process makes perfect sense. But how do you start?

Automate Andon with Continuous Testing

In many organizations people are often afraid to speak up about problems. Even if bugs are suspected, teams will cover up problems by coding around them, even fudging tests. Owning up to problems doesn’t come naturally, so defects and crippling technical debt accumulates. DevOps practitioners need their own version of the Andon Cord: fully automated testing.

With DevOps promising software throughput at dizzying rates, quality must accompany the velocity. Fail here and the continuous delivery glass can only be half-empty. This is why the process must incorporate its own automated testing Andon Cord to prevent problems moving downstream. Automation must be comprehensive, providing maximum coverage at every stage in the software pipeline—from requirements definition to full production and including code quality, performance, security, compliance and test data. Then, when a test fails, the system must be capable of stopping everything until the problem isn’t just fixed but can’t happen again.

Eliminate Waste at the Source

If you’re pulling the cord but not making improvements you don’t have an Andon Cord. Consider a factory scenario where machine operators always have to sweep up dust that’s clogging production equipment—obviously, they should be eliminating the source of the dust.

Unfortunately, many software engineering practices involve lots of dust-sweeping.

When developers need access to dependent systems for testing, they might mock up test code. If QA needs test data in a hurry, it could copy production data but fail to mask sensitive customer details. In IT operations, staff console watch in the hope of gleaning signals from all the noise—the list goes on. Lots of sweeping without ever eliminating the source of the problem.

Fortunately, tools with Andon-like capabilities can help. Service virtualization techniques can emulate dependent systems to remove testing constraints, while automatically generating synthetic test data helps eliminate the possibility of costly compliance breaches. Modern performance monitoring systems also play an essential role. By providing performance insights before, during and after every software build, they help cross-functional teams swarm problems when the cord has been pulled by an automated test.

Build the Ultimate Cord – Customer Experience

If an organization considers itself in any way customer-centric, it will be able to “pull the cord” when it detects anything that detracts from the customer experience as delivered across multiple channels—web, mobile, call-center. The winners go further, using these events as fuel for continuous improvement programs.

This all requires some serious smarts, not at the least application analytics to monitor the customer experience in real time. Such advanced systems become the ultimate Andon Cord, since they treat every sub-optimal customer experience (e.g. functional issue, usage behavior, performance problem) as both a defect and opportunity for improvement. Through advanced analytics and integration this improvement could take many forms—such as immediately prescribing code improvements to developers, or support systems notifying customers when a poor experience is about to occur—maybe even (shock-horror) compensating them financially for any inconvenience.

A strong DevOps culture thrives when individuals call out mistakes and inefficiencies. Fail in this respect and organizations risk paying a heavy price, such as broken customer trust and lost business. Seek out automated methods that help you metaphorically pull the cord and keep on pulling it. Only then can teams truly excel in driving real improvements that will help business gain and retain customers.

— Peter Waterhouse

November 7, 2016
Infrastructure’s Impact on Shifting Left

If you lead big honkin’ software development projects at a big honkin’ enterprise, you’re probably doing everything you can to shift QA left. That’s because the earlier you discover defects and shortfalls, the faster and less expensively you can bring great software to market.

Unfortunately, above a certain scale, it takes more than just better processes and a more aggressive culture to shift test and QA left. It also requires a different approach to the infrastructure you use to distribute your builds.

Infrastructure and the Problem of Scale

Most software developers understand the scale issue primarily as one of complexity. The bigger the project, the greater the chance that something won’t work with something else exactly as planned.

But large software projects also canget badly impeded by the sheer physics of build distribution.

Take, for example, a 10GB build that you have to distribute to five different domestic and offshore locations from your primary facility over your company’s 10Mbps MPLS connection. If your two of your locations are domestic and have 5Mbps connections, it will take five hours to get them each build. If the other three are offshore and have 2Mbps connections, it will take them each 12.5 hours to get their copies of the build.

That’s a “time tax” you’ll have to pay again and again as you keep having to ship massive files over the network to your globally distributed test/QA team. Worse yet, the more Agile and iterative you get, the more you keep paying this “time tax” over and over.

So hours expand into days, and days into weeks—constantly frustrating even your best efforts to more aggressively shift left.

Avoiding the Scale-Driven Time Tax

Here’s how you can successfully avoid the time tax that chronically prevents you from shifting your biggest projects further left.

Instead of continuing to use IT infrastructure that was never designed to support time-compressed software development at scale, you can instead implement a hub-and-spoke build distribution system that virtually eliminates network-related delays.

This hub-and-spoke model lets you maintain a “gold copy” of your current build(s) in the cloud—while providing all your remote locations with their own local copies that get continuously and automatically updated with any changes as they occur.

This model eliminates network-related bottlenecks while allowing your geographically dispersed teams to collaborate without tripping over each other’s work. As a result, you can shift left much more aggressively.

Of course, infrastructure decisions typically aren’t made by the same folks who lead software development. So the two groups have to collaborate closely to make sure the network doesn’t impede critical development processes.

But both parties have lots to gain—because hub-and-spoke saves infrastructure owners lots of money in storage, bandwidth costs and network acceleration hardware at the same time as it empowers development to shift further left.

That’s why DevOps leaders and enterprise infrastructure owners need to work together ASAP. The digital success of the business depends on it!

— Barry Phillips

October 28, 2016
The Real ROI of DevOps

Many established enterprises are stuck in reverse. Dragged down by legacy IT systems and sclerotic software workstreams, these businesses simply cannot adapt and get up to speed in the continuously changing, high-velocity global marketplace of the 21st century. Delta Airlines’ recent technology crash, for example, dramatically illustrates how aging and inflexible IT systems can cripple a company in 2016 and lead to millions of dollars in lost revenue, as well as the dissipation of a hard-won reputation among customers.

But an increasing number of organizations are accelerating their cultural transformations to become software-led companies that can consistently deliver new products and services as a result of automated and collaborative workflows. Alaska Airlines, which focuses relentlessly on measuring the innovation velocity of its business, is a powerful case in point.

This is DevOps at work, and it’s how IT must operate to be the business engine of our quicksilver digital economy.

Measuring the return on investment (ROI) from DevOps is critical to convince organizations falling behind that they can still shift course, transform and head to the IT future. Measuring DevOps ROI is also essential for companies that are embracing the industry’s forward movement, so they know if they’re on the right track and can keep turbo-charging their DevOps efforts.

The most proactive approach for measuring DevOps ROI is to gauge the impact of products and services on customers. But, to get there, we must also assess and analyze the efficiency and effectiveness of the software development process.

Which brings us to the one metric that’s truly relevant in business today—velocity. Without velocity, you can’t quickly recognize and remediate problems. More significantly, you’ll fall behind in the marketplace because you can’t innovate fast enough to keep pace with competitors and customer demand.

Enter DevOps: The culture of innovating at velocity.

Of course, that’s easy to say, difficult to measure. The real ROI of DevOps is how rapidly you can ship your new ideas to customers—and, importantly, understand if those ideas meet customer needs.

Many may look at cost savings and headcount as their first DevOps metrics. These are important barometers but don’t measure long-term value back to the business. Two places to start for value-oriented metrics are time to delivery and quality of release.

Measuring time to delivery shows how long it takes to get new product from code checking to production. Ideally, you want to see your time to delivery drop from months to weeks to days, and then literally to hours and minutes.

Quality of release reveals how many bugs and errors are found in each iteration of the product. Companies should look to remedy these immediately and to develop a pre-production environment and workflow of automated testing for issues.

It’s equally important to measure how rapidly you can get feedback from customers and incorporate that into the next development cycle. Development teams must get clear, direct and actionable feedback about whether their products are moving the business forward. For example, did the two-week sprint to build a new landing page for a bank’s website increase customers’ online banking sessions? Did it generate more cross-sell for the bank’s product portfolio? How soon can we see an uptick in customer satisfaction? This is significant information, and the question is, how fast do the developers respond to this feedback?

The bottom line is that DevOps metrics should always connect back to the business; they need to show a measurable improvement in the customer or staff experience. This could be more quality touchpoints with customers, an improved customer satisfaction score (such as a Net Promoter Score) or increased customer activity in areas such as online billing or account management.

“Delivering on the final product is key,” says Adam Mikeal, director of Information Technology at the College of Architecture, Texas A&M University. “Your job’s not done if the customer can’t accomplish what they want. If the customer can’t do it, it doesn’t matter if your section of the code was good. The customer experience is all that matters.”

Another important part of this equation is measuring your team’s efficiency on the road to velocity. To truly ship ideas at speed, and positively impact your business and its customers, you also need to understand how smoothly your teams are working—in addition to how quickly they’re delivering.

Banks Cash In on DevOps

Westpac Bank in New Zealand fully grasps this imperative, for example, and it enacted DevOps practices to deploy a new pricing engine for mortgages in just three months—from concept to customer. In the past, this project would have taken Westpac 18 months.

Additionally, Westpac measures how well it’s using resources—for every $1 spent on a problem, how much is chewed up by politics and how much actually goes to solving the problem? Before adopting DevOps practices, Westpac was delivering a return of about 20 cents to 22 cents. Now, that number is just shy of 40 cents. That’s a near-2x increase in the value delivery. Westpac’s goal is 3x. In addition, Westpac has increased the amount of work its development teams accomplish by more than 40 percent over the previous year.

Like Westpac, Standard Bank in South Africa has boosted efficiency, thanks to a thriving DevOps culture. The cost of production at Standard, for example, has dropped from about $4,000 to $700—a 300 percent improvement.

The goal for companies in every industry and economic sector around the world today is clear: Accelerate your business so that you create and provide greater—and faster,—value for the customer. But, to drive a robust ROI on this journey, it’s crucial that you carefully measure the true velocity of your organization, whether you’re just beginning with DevOps or you’re well along in the transformation.

— Justin Arbuckle

October 26, 2016
An IT Practitioner’s Guide To DevOps
Two technology innovations of the past decade have fundamentally changed how IT works and given rise to the “DevOps” phenomenon. Before we dive into what DevOps is and how IT organizations should adopt this culture, let’s review what gave rise to it in the first place.

Pull from Agile Development

The last decade has seen many organizations abandon the traditional “waterfall” software development process in favor of “agile” development. Agile is all about small, specific user-centric stories, described by a product owner and delivered in short sprints of two to four weeks by small, focused development teams.

Agile development led to the emergence of more frequent code drops from software engineers, which needed to be deployed more quickly. No longer was it okay to take days or weeks to deploy a new software version; developers working in two-week sprint cycles want deployment to work within minutes with a high degree of predictability.

Enabled by the Cloud

Before the cloud came along, access to infrastructure required human interaction and manual operations. Processes typically could not be automated effectively. This meant that operations were slow and error-prone and created a huge point of friction in the software development life cycle.

Around the same time that developers were adopting agile development, the cloud became real, and infrastructure could be orchestrated effectively using APIs to provision everything from machines and networks to storage, routing policies and firewalls.

DevOps: Closed Loop Automation Between Development and IT

Agile development and cloud computing created the perfect storm in which DevOps was born: this new way of doing IT is all about ensuring closed loops, with 100 percent automation.

Figure 1: DevOps is about ensuring a closed-loop, fully automated workflow between software development and deployment.

In the simplest terms, this meant that each of the steps outlined in Figure 1 need to be fully automated, and integrated with preceding and following steps. Doing this ensures software can be developed faster, deployed quickly, and iterated upon more rapidly.

Benefits

Done right, DevOps leads to a virtuous cycle of greater productivity across development and IT, greater agility in bringing new experiences to market and greater impact on the business.

Adopting DevOps: You Can Phase It In

Organizations looking to adopt DevOps should do so in phases and define metrics for each phase. Here’s a typical phased approach that you can use as a baseline and customize to your organization’s unique needs.

Phase 1: Adopt Agile Development and Cloud Computing

Agile development and cloud computing are profound technology shifts and create the underlying culture in which DevOps can thrive. So, the first step is to ensure organizational buy-in for both of these trends:
1. Agile development can be adopted in stages. The most important is to agree that software will be built and shipped in short, two-to-four-week sprint cycles, and with a focus on user stories.
2. Cloud computing can be adopted no matter what your IT infrastructure looks like today. Use hybrid cloud platforms such as OpenStack to provide programmatic access to a VMware or Linux environment, or use a public cloud platform such as Amazon Web Services.
Phase 2: Commit to a CI/CD Workflow

Continuous integration (CI) systems such as Jenkins or TeamCity provide a way to pipeline and integrate the various steps in a DevOps workflow. The choice of system doesn’t matter as much as ensuring that all steps of the workflow will be integrated (eventually) into this workflow.

Phase 3: Repeatable, Automated Software Deployment

Many organizations tend to have some manual steps in the packaging and deployment of their software, especially for getting it into production. This often reflects underlying assumptions about manual oversight and intervention, and it breaks the closed-loop philosophy of DevOps.

To succeed with DevOps, you need to fully automate the software packaging and deployment process. In addition, ensure that production builds are deployed in the exact same manner as developer builds, using identical environments.

Cloud technologies such as orchestration (OpenStack HEAT or AWS CloudFormations) and configuration management tools such as Ansible, Puppet or Chef can be used to implement this phase.

Phase 4: Automate Software Testing (Continuous Delivery)

Once software deployment has been automated, the next phase is to automate the validation process for new software releases. If you’re like most organizations, testing involves several rounds of back and forth with QA teams and product managers, and it often takes weeks to resolve the last few bugs in each release. To reach the next level of DevOps automation, this process has to be completely lights-out and automated.

This requires a fully automated software test suite, across the back-end and front-end, so that product managers have confidence in the quality of the software being released. Once implemented, you finally have a closed loop between the critical elements of the DevOps life cycle. While release to production may still require someone to approve it or “push a button,” this is a very respectable state in the DevOps journey.

Phase 5: Continuous Deployment

To get to the most advanced DevOps phase, you need to automate the “go live” process for software releases. This in turn will allow you to take even smaller, more frequent code drops and push them into production.

The most advanced software teams in the world now operate this way, and the organizational implications can be transformative: Developers come into work feeling that they can implement a feature and have it go live immediately, and product managers can make commitments to the business with the confidence they can deliver, because they have this incredibly efficient software delivery machine backing them. Most importantly for you, IT is no longer seen as a cost center; it’s seen as the fundamental enabler for business productivity and time-to-value.

Closing Thoughts

Getting to Phase 5 represents the state-of-the-art in the industry today. However, another significant breakthrough promises to raise the bar even higher.

In each of the prior phases, the DevOps workflow is integrating software development with IT operations; but there is nothing influencing software architecture to take operations into account.

In other words, we’re still bolting IT operations automation onto software development.

This is about to change. We’re on the cusp of a new era in software: instead of bolting operations onto the software development process as an afterthought, a new framework promises to bring development and operations together by design. With declarative, infrastructure-agnostic constructs to describe how applications are composed, interact and managed, Kubernetes promises to increase the operability of modern software systems by an order of magnitude.

Everyone interested in the DevOps journey should be learning about Kubernetes and evaluating how it will improve your software development and operations workflows.

About the Author / Sirish Raghuram

Sirish Raghuram is co-founder & CEO of Platform9. As CEO, Sirish quarterbacks the team at Platform9 in realizing the company’s vision, which, on a given day, can range from strategy and road map to shipping code and running tests. Before co-founding Platform9, Sirish was an early engineer at VMware, where he held technical and management leadership roles and helped ship multiple vSphere products. Sirish’s work was reflected in several patents, VMworld talks and internal product innovations within VMware Engineering. Sirish has a B.E. in Computer Science from the University of Pune, India. Connect with him on LinkedIn.
October 25, 2016
A DevOps Checklist for Traditional Service Providers

As companies such as Amazon and Google rapidly innovate to deliver new services, traditional service providers recognize that change is necessary to keep up with shifting customer expectations. Given the pace at which technology continues to morph, creating better, faster and more innovative solutions is a necessity, and if this isn’t achieved, even longtime customers will eventually look elsewhere.

For younger businesses, taking a collaborative approach to IT and product development often comes naturally as it’s built into the company’s DNA. However, for more established service providers, adapting to this new way of doing business can prove challenging. Traditional service providers have spent years perfecting their technology and require multiple layers of review prior to rollout. While these checks and balances are in place to ensure a level of quality customers have relied on for years and expect to receive from traditional providers, it also can hinder their ability to react quickly to client demands. However, the adoption of the principles embedded in DevOps can allow traditional service providers to work more like a startup by accessing their technology and human skills in a new way, while also continuing to deliver products which meet the expectations of their established customers.

Now the question is, how can traditional services providers successfully implement this DevOps approach? While it can (and has been) done successfully, there are five critical steps that need to be taken into account:

1) Establish a clear goal. The most important step is the first: Management must have a clear understanding of what they are looking to accomplish with the DevOps transition. Service providers have a long legacy of well-defined, reliable services, so all changes must complement their current product offerings and live up to the quality their customer base has come to expect. If an overall goal is not defined, customers will be left confused and disappointed, and ultimately, there will be no impact on the bottom line.

2) Set a timeline. While creating an unrealistic timeline is never advised, it is critical to set an aggressive, pragmatic and adjustable one. As mentioned above, without aggressively pushing forward, an organization will find itself continuously playing catch up to its competitors. The timeline is meant to push an initiative forward even when there are setbacks or changes that need to be made along the way. The right timing is a balancing act: too fast and you have a poorly crafted product rollout; too slow and you are stuck behind the competition. The key metric to measure progress is the degree of evolution to the DevOps model within key stakeholder organizations. If solutions are delayed due to existing processes, then either the scope of the team needs to be adjusted to encompass those processes or the goal needs to be changed so they are no longer an issue.

3) Collect necessary resources. Typically, the biggest gap that needs to be filled to move forward is with human resources. For many established companies, having design and architecture skills—core capabilities needed to develop a new product—hasn’t been a priority. Creating the right team that consists of new talent that have these missing skills, along with the addition of current employees who have a more intimate understanding of the company and customer, is a critical aspect in making sure this more collaborative and innovative approach to solution development and delivery works. And it doesn’t stop once the team is created, it is important to continue to foster an environment that encourages the new team to work together and promote their process within the broader organization.

4) Create benchmarks. To ensure the established timeline is being met and that there is an opportunity to change course if necessary, it is important to set clear deliverables at various stages of the process. These benchmarks must look to what people are doing today, not just the end goal. Initial measures should focus on the pace at which progress is being made, and later, focus on the end products being delivered.

5) Get the entire company on board. Changing an embedded philosophy isn’t easy, especially one that has worked well for so long. No matter the organization there is almost always resistance on some level when fundamental changes are being made. It is up to leadership and the DevOps team(s) to demonstrate internally and externally that the changes being made are not just for the sake of change, but in the end will benefit the entire company, as well as the customer.

While taking a deeper look at an organization’s “people, process, technology and information” and moving to a new mindset doesn’t happen overnight or come without its hiccups, the status quo is no longer sufficient to compete. The reality is, a more collaborative and faster delivery model is necessary to be successful in tomorrow’s marketplace.

About the Author / Tim Pearson

Tim Pearson is the director of product line management at Ciena. In his role, Tim leverages DevOps practices to break barriers in network delivery to meet the demands of mobile, VR and video applications. Driven by challenge and fueled by deep industry knowledge, he has built a reputation for developing innovative business and product strategies to drive growth and open new markets. Tim holds a Bachelor degree in Electrical Engineering from Memorial University, a Masters degree in Management Science from the University of Waterloo and an MBA from the MIT Sloan School of Management. Connect with him on LinkedIn and Twitter.

October 19, 2016
Bucking Bank Bureaucrats: From Convergent Thinking to Divergent Doing

As customers, longstanding and new, the way we interact with banks and manage our personal spending, borrowing and saving is experiencing drastic change. New digital offerings in the form of apps and online banking show commitment to customer centricity, but overall, banks are slow to adapt to the pace of change. The vast majority of bankers do recognize the strategic need to respond to their industry’s digital disruption, and are busy assessing and developing transformative technologies such as mobile, blockchain and artificial intelligence, trying to find an emerging market need and a robust business model.

How Should Banks Adapt?

The operating environment of banks is unpredictable and, therefore, requires organizational agility. This agility requires a careful analysis of customer needs and (potential) capabilities. Bankers must ask, “How can my bank move from convergent thinking to divergent doing?”

From a strategic perspective, it’s essential to understand whether the person/organization wants to be a disruptor, innovator, ideator or bureaucrat. This differentiation defines the right strategic options available to them.

N.B.: A quick proxy answer to the above question is the design of the bank’s reception and organization of the people working there, to see if it is like every other bank, or is in some way distinctive.

Value Triangulation and Differentiation

Every banking CEO needs a clear strategy that addresses their organization’s internal and external challenges. There is one global commonality for a strategic foundation: Every bank needs to deliver change rapidly and frequently. The better a bank is at improving customer experience, the more likely it will survive the onslaught of disruptors and innovators. Additionally, most banks must deliver cost reduction and drive income. Customer desire, the third corner of the triangle, is not a familiar banking concept, but a successful value triangulation delivers customer desire in the most efficient way possible.

This isn’t about value chain mapping and cost re-engineering; it’s about creating great customer experiences with adaptive capabilities and capital-light business models. All three values must be understood and practiced throughout the entire organization, especially among boards, chief executives and key decision makers to make adaptive banking possible.

Customer Experience Comes First

A culture of centralization, product-centricity and control-via-process stifles change. To be adaptive and create their own Digital DNA, banks must form new work spaces and ways of working. When bank employees don’t use their own company’s products, they are out of touch with the customer experience. They also may excuse customer complaints based on their organizational knowledge. Employees need to be the bank’s most critical customers, and ought to feel fully connected to other customers and their experiences.

New, lean business models should flow through the organization and align to the desired customer experience. The business leaders who identify the “priority customer experience features” will focus the organization on accelerating delivery momentum. Rather than requesting large-scale, product-led programs, leaders need to focus on continually launching and redefining small-scale features.

Doing so moves away from the grand strategies or silver bullet approach so often prevalent during annual planning cycles and shifts toward agile, customer-centric strategies. Organizational change is fundamental for creating adaptive banks with Digital DNA.

Fail Fast, Succeed Even Faster

To accelerate growth, adaptive technology architectures and agile software delivery methods need to be available to deliver the relevant features. A “deliver, measure and learn” approach will amplify feedback across the organization to drive commercial decision-making. This will allow banks to fail fast, yet succeed even faster. The ongoing cycle—ideating, designing, building, shipping, measuring, learning and (of course) adjusting customer features—needs to become the norm.

By becoming adaptive, the bank’s capabilities (people, methods, technologies, values) become more attuned to customer needs, making it far easier to deliver personalized offerings. For example, the bank’s data can be used to create feature personalization, allowing customers to pick and choose the right features for their needs. A customer should always be able to overwrite the choice an algorithm provides.

Innovation, Transformation and Beyond

Currently, the most apparent transformation desire and delivery exists in IT departments as opposed to business departments. Agile ways of working are often still alien to most bankers in product, finance, compliance, marketing and front-line teams.

The banking playbook has already been rewritten for the digital age. It’s an exciting time to realize great change and how disruption/innovation can become commercial reality—this means bucking the system.

About the Author / Darren Oddie

Darren Oddie is strategy director at Wipro Digital, where he works with global financial services companies on digital transformation and innovation. He’s had hands-on experience of designing and building financial services brands from a blank sheet of paper to industry leading propositions in both corporates and startups. He’s delighted that banking and customer experience are now regularly mentioned in the same sentence. Connect with him on Twitter.

October 17, 2016