devopsy.kinsta.cloud

Author: contributor

Can DevOps Be Applied to Production Networks?
The key goal of DevOps is to bring development and operations together, bound by a common set of practices. For this to become reality, developers need to think more like the operations team—focusing on stability and reliability—while operators need to think more like developers—focusing on agility and speed. But how widely can this be applied?

We say regularly the goal of application development is to deploy new features into production continuously—daily, weekly or monthly, if possible. But what about the production infrastructure itself? Can we apply DevOps practices to making changes to the production infrastructure on which these applications run?

Recently, I met with customers who want to implement DevOps practices to how they manage their operational network, applying continuous deployment to the production network. For example, they would like to design, test and then deploy new network services such as load balancing and firewalls into production immediately to enable new applications and services—actions that currently could take weeks or months. Is this kind of continuous deployment to be expected, or is it pushing DevOps too far?

Let’s look at how key concepts of DevOps might be applied to adding network services to a production enterprise network:
1. Virtualize services: To enable rapid deployment of applications, it was first critical that they be enabled with virtualization and in the future, with containers to make them portable. We would need to apply this same approach to production network infrastructure if we expect to enable rapid deployment of new infrastructure. Unfortunately, there is no standard way to make physical network equipment portable. Of course, shifting heavily to software-defined networking (SDN) enables virtualization for free, but it doesn’t make even virtual network equipment “standard.” In the world of networking, there are a number of “standard” interfaces and modeling languages: SNMP, NetConf, Yang and TOSCA (though not originally designed for networking), to name a few. Using one of these standard interfaces, in combination with virtualization, potentially could enable the agility required for continuous deployment of network services.
2. Automate: DevOps success depends on automating the entire process of development, test and deployment so the process can be continuous. The same level of automation would need to be provided throughout the process of designing, testing and deploying new network services onto production networks. Network testing is very similar to other types of hardware and software testing so most test automation frameworks can be applied. However, network testing requires automation of the environment setup, and it also requires traffic generation to test with realistic network conditions.
3. Continuously deploy: While this is still a dream for many companies attempting to achieve DevOps nirvana, a key requirement for using DevOps for production network configuration is to be able to deploy new services continuously into a live production network. Most of the configuration management tools available today are good at configuring servers and software, but are more immature when it comes to configuring networks, which are far more complex and inter-dependent than a single server. A tool would be required that allows the configuration of the tested network to be deployed automatically in exactly the same way within a production network.
4. Monitor Operation: A good DevOps practice monitors applications in production and uses that information in planning new services. While it would seem that network monitoring is quite mature, an open question is whether that data is useful in planning future network services for improved operation.
Sandboxes for Network DevOps

Sandboxes are tools for defining and testing infrastructure configurations (physical, virtual and application). A good sandbox allows users to model any infrastructure, including any type of network infrastructure. This model also can serve as a way to implement “virtual services.” Models can be created in Yang or TOSCA, and under the covers these models can automate access to network infrastructure using CLIs, SNMP and/or NetConf where appropriate. This allows network services to be defined and tested using a high-level model, rather than the low-level interfaces.

Sandboxes also address automation by providing a context for automating not just individual network components but also entire configurations and connectivity that are required for a “network service.” In addition, automation can encompass traffic generators and test tools as part of the testing process.

Sandboxes also are designed to set up and tear down a network service automatically so they can be used for continuous deployment of new services. The same automation used to create the sandbox for a network service also can be used to deploy it into production.

Finally, monitoring used in production also can be performed in the Sandbox to compare performance between testing environments and production so future services can be improved.

While applying DevOps to production infrastructure and services is challenging and immature, in principle it can provide significant improvements in the speed of deployment of new network services—and the applications that depend on those services.

About the Author / Joan Wrabetz

Joan Wrabetz is the Chief Technology Officer for Quali. Most recently, she was the vice president and Chief Technology Officer for the Emerging Product Division of EMC. Ms. Wrabetz holds an MBA from the University of California, Berkeley, an MSEE from Stanford University, and BSEE from Yale University. She has taught as an adjunct faculty member at the University of St. Thomas, St. Mary’s University, and at the Carlson School of Business at the University of Minnesota. She holds patents in load balancing, distributed systems and machine learning classification and analytics.
July 15, 2016
What ‘Star Wars’ Can Teach Us About DevOps Automation
Perhaps no movie has ever received the same level of hype as “Star Wars: The Force Awakens.” Longtime fans yearned for the latest episode of the sprawling saga to bring back some of the magic of the original movies, while paving the way for the next generation of the “Star Wars” universe.

In this way, the current state of “Star Wars” provides a useful way for understanding the infrastructure evolution situation that many IT organizations now find themselves in.

‘Star Wars’ and DevOps Automation

Just as “Star Wars” has lots of previous characters and concepts—from Han Solo to the Force itself—to weave into its new properties, DevOps-minded teams have their own legacy infrastructure (admittedly, not as glamorous or nearly as profitable as the 1970s vintage “Star Wars” canon) to bring into a future defined by virtualization, automation and cloud orchestration. This modernization effort can go several ways depending on what approach is taken:
- Organizations may quickly run into difficulties if they treat their DevOps initiatives as simply a new type of silo, one that is focused intensely on virtual and cloud assets without ample attention paid to legacy and physical infrastructure. You could say that a similar tack was taken with the first Death Star: Despite all of the focus on its planet-destroying weaponry and enormous tractor beam (which hauled in the Millennium Falcon), the space station’s designers neglected its basic heating infrastructure, eventually settling on a small exhaust port that was vulnerable to torpedo attack (and, indeed, was destroyed in that exact way).
- On the other end of the spectrum, creating a culture of collaboration supported by highly capable technical tools can ensure that DevOps innovation takes hold and facilitates business agility. Think of this ideal as being similar to the relationship between R2-D2 and Luke Skywalker when the latter was piloting his X-Wing Fighter. There was not only a good understanding between the two of them, but also the powerful automation capabilities provided by R2’s advanced circuitry.
The new Star Wars movie provides an occasion to see how old and new infrastructure, such as that inside the Millennium Falcon, can work together.

Overall, we can see here both a “dark side” and “light side” approach to DevOps transformation. An old joke about the Force is that it is a lot like duct tape: it has a dark side and a light side and it holds everything together. Something similar could be said about the variety of infrastructure types that support critical business applications. They can cause a lot of problems (e.g., they probably can’t match the credit card-swipe convenience of Amazon Web Services), they can provide some key advantages (e.g., by being easy to control and secure on-premises) and ultimately they are essential cogs for supporting a wide range of applications, especially within bimodal IT setups.

Why the Millennium Falcon Would Have Benefited from a Better Self-Service Portal

Since we are on the subject on supporting and holding things together, we should talk a little about the Millennium Falcon, a ship that always seemed to be on the verge of breaking down in the original films. Its hyperdrive for making the leap to lightspeed was spotty at best, and it took its fair share of bruises. Keeping it in working order and out of the clutches of the Empire was a full-time job for Han and crew.

Part of the problem was the Falcon was this weird mix of old and new technologies. It could travel at warp speed, fire lasers and make the Kessel Run in less than 12 parsecs. But it also required human pilots and gunners; plus, it was accurately labeled a “bucket of bolts” by Princess Leia Organa during a critical escape sequence.

Maybe the Millennium Falcon would have run a lot more smoothly if it had a self-service portal that helped keep tabs on what was going on around its decks. Are the ship’s countless modifications playing nice with its older components? Is there a technical problem hidden in one of its smuggling holds? Is the holographic Dejarik table from Episode IV eating up too many resources? Having a centralized automation platform would have helped with these issues of accountability, visibility and IT governance.

One could say that the Millennium Falcon was already a rough approximation of environment-as-a-service (EaaS), since it passed muster in automating a lot of heterogeneous infrastructures implemented and modified by its various owners. But it could have gone further in bridging the gap, just as many IT organizations can take additional steps to ensure DevOps automation success by working with software such as CloudShell.

The takeaway: “Star Wars: The Force Awakens” provides an occasion to do more than sit back and enjoy the lightsaber fights —it’s also a moment to see how old and new infrastructure, such as that inside the Millennium Falcon, can work together. With an EaaS and self-service automation solution, IT organizations can bring their legacy, physical, virtual and cloud assets together into a more agile combination.

About the Author / Hans Ashlock

Hans Ashlock joined Quali in 2013 and heads Quali’s worldwide technical marketing. Hans has more than 15 years of software engineering, product management, marketing and sales experience in automation software solutions addressing, cloud, networking, telcom, data center, federal, and semiconductor industries. Before Quali, Hans was a founding member of a global engineering services company, developing products and services for enabling adoption of automation technologies. Though he only holds one patent, he’s glad he didn’t have to pay for it. An engineer at heart, Hans is passionate about the transformation to software-defined everything and its impact on business and technology. Connect with him on LinkedIn, Twitter, and SlideShare, or at www.qualisystems.com.
July 14, 2016
Follow the Simple Path of Nirvana to DevOps

Perhaps the title of this post is a misleading, as nothing is truly “simple. But there is a well-defined pathway to achieving DevOps goals for any size company. Just follow a step-up pattern by adopting the principles of agile, continuous integration and continuous delivery, and you will reach your goal of DevOps. This path is a natural progression of DevOps maturity that an organization can follow while adopting key principles and technology solutions along the way.

Agile Development

Adopting agile development methodologies often is the first step maturing organizations take, as they no doubt have heard the resounding benefits of leaving behind waterfall software development life cycles. There is a reason this is the first step. It helps your organization’s employees mature into a tightly knit working machine that fosters improvements in driving down technical debt while also laser-focused on building key features.

As agile teams continue their maturation, they’ve roped in key business stakeholders who are able to grasp the drivers of technology challenges and are able balance out the growth needs of the business with the sustainability requirements of the technology teams. Suddenly, teams are investing a minimum of 20 percent of their sprint time to dealing with technical debt and reporting out these victories to their product owner(s). These small wins become major victories as the unwieldy pieces of infrastructure and software of the business become relics and folklore of yesteryear.

But there is an underlying reason why agile development is the most crucial first step: It truly empowers the developers to improve the customer experience and feel good about it. With the scrum master’s shields up protecting the team from burnout and ensuring sprint planning happens, team members are more likely to be passionate about their work and the results of their efforts to the business and customer. This is where the empowerment of DevOps really begins.

Continuous Integration

As teams mature along the agile development continuum, the next step in their DevOps maturation is moving on to continuous integration. Continuous integration is the process by which a company automates its software build process: code check-in triggers a build job that then triggers a test validation job and so-on. This process enables teams to continuously integrate their code build with downstream processes, avoiding manual touchpoints.

In addition, this paradigm allows for faster feedback loops to ensure a higher quality product, and empowers diverse teams to work together utilizing automation as the “middle man.”

Continuous Delivery

The third step in DevOps maturation is continuous delivery. Often businesses confuse “delivery” with “deployment”—in fact, continuous delivery is simply the process to ensure that at any given point a company’s software is in a state where it can be deployed. This is useful for quick iterations, A/B testing and a more mature way of delighting customers fast and often.

A key component of continuous delivery is ensuring that software is ready for production. When a team is doing continuous delivery, they’re assured that they’re nimble and responsive and adhere to the highest quality standards.

DevOps

At this point, the company has reached the DevOps nirvana state. This is where software is not just delivered continuously, but all of the operational needs are constantly met. Monitoring, metrics, the ability to automatically roll back software releases, scalability and much more are the key building blocks to a powerful DevOps organization.

With this thoughtful step-by-step process of DevOps adoption, organizations can meet the needs of business in a mindful and cohesive way without compromise.

I’d love to hear any feedback or tips on DevOps at @jsin.

About the Author/Jason Shawn

Jason Shawn is a technology leader in DevOps and Cloud. His began his career as a software developer and has held multiple functional and managerial roles which has found him with a unique perspective on the software development lifecycle. He’s extremely passionate about Continuous Delivery, Agile software development, and empowering teams to adopt DevOps and Agile principles in their daily lives. He is also a member of the Cloudbees Customer Advisory Board and enjoys public speaking, travel, and time with his family.

July 13, 2016
What Does DevOps Mean to Test Teams?

It wasn’t long ago that the Dev and test teams would work late hours, focused and rushed to meet a deadline: rapid fixing, reprioritizing and deferring bugs to close out the bug list, move everything to the staging server, do one last run of the regression and pass it over to Ops/IT to move to production. What happened after? No one knew. For most Dev and test team members, Ops is a black box. More often than not, they are oblivious to what happens at Ops, the roles, responsibilities and timelines. One day—long after the drop-dead deadline for Dev and test teams—after delays, questions and changes, the product went live.

Does this scenario sound familiar? Well, DevOps is supposed to be a game changer.

DevOps means many things to many people. But for test teams, continuous testing (CT) is the primary task in DevOps. And, CT is not merely rerunning the same set of “full regression” tests over and over, as product gets delivered and deployed. This is actually a recipe for disaster and not the intent of continuous testing.

What is CT and How Do Traditional Test Teams Fit In?

Recently, test teams were challenged to ramp up their practices by moving to agile and scrum. DevOps will propel the teams to now ramp their practices up even more—more automation, more technical testing and more tools to leverage. If your test team already has mature agile practices, with continuous testing, they can mature further to eventually accelerate the product release.

Continuous testing grows from continuous integration (CI).

Starting with CI, test teams joined the process with automated smoke tests. A strict view on CI is about automating the build process. Yet for anybody doing CI today, that is just the start. The reality of CI using tools such as Jenkins, Bamboo or TeamCity is that after automating the build process, the unit tests are rerun, the automated smoke tests (at the user interface, API level or wherever) are run and many companies build on that to include automatic rerunning the full regression in the test environment, then automatically running some set of tests against some number of VMs. This growing in CI levels of “awesomeness” is becoming a common practice. You may ask, “Isn’t this continuous testing or approaching continuous testing?” CI is the foundation. It grows from there.

Continuous testing is a lean process of quality at every step. It includes quality user stories, quality environments, quality unit tests and quality performance tests. It is “testing at every step” and not “QA” at the end. It is also shifting testing tasks from Ops at the end, by testing earlier and testing all along the way. CT is adding more tasks as well.

Test Strategy

The success of CT is in its ability to get everyone involved and not just the test team.

Running the right tests at the right time on the right environments was never an easy task. In modern development, testing—unit tests, performance tests, security tests, functional tests and workflow tests—is everyone’s responsibility. It’s the foundation of continuous testing.

The test strategy for continuous testing is quite different than agile testing strategies. In agile, testing very often relies on a regression or a release sprint before deployment for bigger and longer tests, followed by Ops teams doing their own performance and security tests on a staging environment. In DevOps and the “shift left” approach, all these tests need to be run earlier and more often as part of the test team’s suites.

So far we have developer unit tests and automated smoke tests run with CI, an automated full regression that gets run often. These tests in addition to manual tests, which are the normal part of new user story development during sprints, continue to be a part of continuous testing.

Continuous testing gets bigger from here. Ops/IT builds environments and data for test teams through virtualization, VMs, containers, in the cloud, etc. Test teams will automate more and run different types of tests, much earlier, against the virtualized environments that are the same as in production. The idea is to take what used to be late development and final tests and shift them left, earlier in test cycles. Thus, eliminating a regression or release sprint and having the product ready for delivery continuously.

Longer workflow and integration tests now must be automated and run against the builds for continuous delivery, as well as load, performance and security tests run at various points during development. This presents many problems for testing and Ops to resolve.

This brings up the question, when do test teams do usability, soap opera, end-to-end path, workflow and task-based tests? These are tests focused on completing a full user task as opposed to testing an isolated individual function. Are those automated? Longer tests tend to break more and are more complicated to automate. Many teams automate their regression tests but exclude from the complex, difficult or long scenario tests. These require a higher level of automation skill. Too many automation tests today are small, isolated tests that validate individual functions. We all know that is not how users use a product. They use functions in various sequences, combinations and varieties of data. It is certainly not an easy task to automate and maintain all of those more complex tests.

Also, many test teams have not been responsible for running performance, load and security tests. Traditional test teams may, in the DevOps world, still not be the team writing them. They may manage or execute these new testing tasks earlier as part of their regular automated suites. A good strategy will define when and how often these tests will be run.

Remember, the goal of CT is immediate feedback. A common question that arises is that why can’t every test be put into one giant automated suite and have it run all the time. That is not required. We don’t need to run the same tests so often for immediate feedback. It is better to have lean and mean suites of tests that are run strategically. Run unit tests, feedback, go to the next step, run smoke tests, feedback, go to the next tests and so on.

Automation

I still see teams struggle with agile automation. They can’t automate enough in the same sprint, the automation is fragile, there is no methodology, which can lead to the suite staying too small, or conversely, getting bloated and slow when we need lean and mean. These problems need to be fixed before attempting CT.

Equally important is that some teams still don’t have adequate resources to manage and maintain an automation framework, code new tests, thoughtfully retire old tests and effectively work on new user stories in new sprints. Where there are many teams that will easily slide in continuous testing, there will be more teams who cannot keep up with the new demands and compressed delivery requirements.

Leveraging technology

Many teams still suffer from data and environment issues. Luckily, virtualization, containers and the cloud, can take care of these by leveraging popular technologies. But most test teams will need the help of Ops to get this accomplished.

SOA/Microservices

There is a swiftly growing trend of building applications out of services. The trend is not new but has been picking up steam. From APIs to service-oriented architecture to web services to microservices, testing is evolving into exercising the integration of products made up of services. The use of containers has sped up the use of services. Test teams must have the knowledge, skills and tools to adequately test and automate the testing of these services.

Outsource service testing if you don’t have the skill within your test team—don’t skip it. Waiting to test these through a user interface (UI) is always possible, but isolating a defect once it happens is much tougher through a UI than at the service level. Running a service test through the UI is not quick and easy. The most important reasons we automate so much in continuous testing is to get quick and easy feedback on the health of the system. Skipping testing API or any service at that service level is not OK anymore.

Summary

Testing is no longer the responsibility of a silo QA team. It is everyone’s job!

Scrum and big movement on XP practices such as TDD or simply doing a bunch more unit testing has greased the skids for the whole team to be more responsible. Everyone on the team has to test. If you can’t, then DevOps is not for you.

Test teams will need sophisticated automation engineers, and Ops/IT staff to support environments, data, VMs, containers and all of the virtualization needs of the test team.

Intelligent, lean and mean test automation is the test team’s primary goal. If your dev and test teams already have mature and efficient agile practices, low maintenance test automation and a good continuous integration process in place, then continuous testing and DevOps is the next step in achieving hyper-efficiency. Everyone in the enterprise benefits by bringing Ops/IT and its processes into the dev process much earlier, by shifting left, and taking advantage of the advances in virtualization and infrastructure automation.

About the Author/Michael Hackett

Michael is a co-founder of LogiGear Corporation, and has over two decades of experience in software engineering in banking, securities, healthcare and consumer electronics.

Michael is a Certified Scrum Master and has co-authored two books on software testing. Testing Applications on the Web: Test Planning for Mobile and Internet-Based Systems (Wiley, 2nd ed. 2003), available in English, Chinese and Japanese, and Global Software Test Automation (HappyAbout Publishing, 2006). He is a founding member of the Board of Advisors at the University of California Berkeley Extension and has taught for the Certificate in Software Quality Engineering and Management at the University of California Santa Cruz Extension. As a member of IEEE, his training courses have brought Silicon Valley testing expertise to over 16 countries.

Michael holds a Bachelor of Science in Engineering from Carnegie Mellon University.

July 4, 2016
Building the Search Engine Users Expect
The search experience most of us are familiar with is Google’s. The company has raised the expectations of search, conditioning us to its level of user experience. However, most websites and mobile applications still don’t offer this kind of advanced search.

Here are three things to consider to keep the search experience on your company website or mobile app competitive:

Speed

Amazon reports that every 100 millisecond in latency causes the company a 1 percent loss in sales. Google says that every 500 milliseconds in latency causes a 20 percent drop in traffic. In today’s “attention economy,” people won’t wait for a slow search when they can easily jump to a competitor’s site.

How can you ensure optimal speed from your search engine?

A robust infrastructure plays a key role in maximizing search speed. Search is quite resource-intensive, so start with solid hardware that maximizes performance when having big loads on CPU, solid state drive and memory. Beyond the hardware, the network also plays a huge role in search speed. Like a content delivery network, users need to be close to the search engine to have a truly interactive session; it is important to distribute the search in different regions if your users are distributed worldwide.

User Experience

Users also expect finding the content they need to be as intuitive as possible. Some search engines suggest queries to users based on their search input; however, this often falls short for several reasons:
1. If you suggest queries, you have little to no control over the quality of the results compared to the original intention. It is very frustrating as a user to follow suggestions that lead to poor results.
1. Suggesting queries, often accomplished by “did-you-mean” type proposals, introduces an unnecessary step in the user interaction, and we’ve already established that speed is an important factor in finding information. You should even consider allowing your search engine to handle typos on the fly without requesting further user validation.
1. If a user searches for something very specific, the probability of your frequent queries including their exact answer is low. In this case, the user is getting to a search without any feedback, which makes it much more difficult to actually find the intended content.
Many websites also encourage browsing activities based on an extensive site map. But expecting users to know and understand your site map to find the information or product they need is unrealistic.

Instead, users most benefit from a search experience that displays requested information as they type, updating results in real time. For example, on an ecommerce website, you should display products, categories, brand or reviews instead of queries. You should also propose a rich display with a readable summary of information.

Relevance

Finally, relevance should be an important consideration in building a search engine, but relevance is a difficult topic to understand, let alone master. Many existing searches require extensive fine-tuning of their engine and may still not understand the impact each change has on the search results. It often happens when a marketing manager comes to see the tech team to ask for a fix on a specific query. The tech team tunes the search rankings to fix that specific query, unknowingly breaking dozens of other queries. This approach is an endless game that will never globally improve the engine.

You should focus first on the different ways to enter a query (for example, “iphone,” “i-phone” and “i phone”), and be sure to retrieve the relevant information for all of them. You can then enrich results with business data such as the number of sales or number of times a product was put in a cart.

About the Author/Julien Lemoine

Julien is a search veteran who’s been working in the search landscape for over 10 years, with work experience in Thales and Exalead. He participated in the design of three different search engines prior to co-founding Algolia with Nicolas and is the author of the different algorithms responsible for Algolia’s super fast performance.

Julien is also passionate about imparting knowledge and spent some time teaching at his Alma Mater, EPITA. He holds an engineering degree from the same institution.
June 21, 2016
MTTR: The Magic Metric
How do you improve with one metric quality, cost, performance, employee and, most importantly, customer satisfaction? I think MTTR is one to consider, but with a DevOps twist (agile+lean+ITSM). The thing I love about DevOps is that it allows for people to think differently. DevOps is the movement driving organizations to find new ways of collaborating, communicating, cooperating and improving the way they use technology to create and support services.

While many agree that MTT stands for “Mean Time To,” there is no common agreement on the R. I like using 5 Rs, as they cover the life cycle of idea to realization (in the hands of the customers in a way they like).

So what are the 5 Rs? Well, let’s work backward (shift-left thinking):
- Review: What happened? How can we stop this in the future? How can we shorten the cycle?
- Return: Do customers agree that all is well? How satisfied are they with not only our performance but the fact the event occurred?
- Restore: The time and effort to get servers, network, applications, data, passwords back as they should be.
- Repair: The time and effort to resolve the issue.
- Respond: The time to notice, decide if we can fix or get others involved, and their reactions to begin getting things sorted.
MTTR is a cycle of events. Breaking them down and understanding the value of each step enables you to reduce time, effort, waste and cost. How often do things break and why? Are they self-inflicted because we test poorly? How can you apply automation or self-healing? How good is your monitoring and alerting and response mechanisms? How great is your documentation? Do you rely on a certain person to fix things or can anyone? Knowing the answers to these questions can make issues become less frequent—or, at least, reduce the effort to address them when they do occur, as things will break.

Knowing these answers also helps clarify priority of issues. Working back from the customer drives the flow of work to highlight faults. Why treat all issues as P1 all month? For example, payroll is only a P1 when you are getting paid, but the rest of the month it can be a P2. Working with customers will help ensure you are fixing the important things, not something someone wrote in a service level agreement that is rarely updated.

MTTR is also a fabulous learning measure. DevOps is all about collaboration and that begins when people talk to each other, repair things or introduce a better test or monitor or alert to capture issues and customers are minimally impacted. Let your service desk own MTTR and see how issues get resolved better, faster and more safely.

MTTR helps drive movement to virtual or cloud. MTTR can also help you improve your A/B use of infrastructure or services. Have an issue? We’ll flip a switch.

But the overall goal of any metric is to drive actions, decisions and improvement. MTTR is such a measure. Be brave, throw away your SLAs, get people involved for issues and use MTTR to make things better across the life cycle. Then celebrate the success of your journey!

Let us know. Do you use MTTR? Can you measure each step and perform a retrospective?

About the Author / Daniel Breston

Daniel Breston is the Chief of DevOps Transformation for Ranger4. Daniel has 30+years leading IT Service, Operations, Applications, Data Center and service partners in management of technology enabling business processes and functions. He has also spent 15 years of consulting and coaching leadership teams across the EU and UK in developing their people, processes and use of technology to deliver technology services better, faster, safer. Daniel has qualifications in IT Service Management (ITIL, COBIT,etc.), Lean and DevOPS.
June 17, 2016
Doing Nothing is Hard Work!

If you really want to know how good your software delivery process is, try doing nothing. Yes, you read that correctly, do nothing: Go through your normal release process without any requirements, no new code, no testing and no installation at the end. Sounds easy, right?

So let’s get started. This will be different for each organization of course, but typically a new release starts by creating an entry in some administrative system. Then there is usually a review on the scope, budget and overall fit within the company’s strategy. Since there are no new requirements, you should automatically pass any reviews. But if the approval requires a meeting or conference call, you still have to schedule it. Say “Hi” and walk out of the room or hang up the phone. Make sure you get any required e-mail approvals.

Next, go through all of your other mandatory reviews: security, data privacy, regulatory compliance, user interface design, infrastructure readiness and so on. You may need your test team to sign off on quality as well. And don’t forget to reserve an install window and take into account any change freeze periods.

If you’re lucky, you were able to skip or get auto-approval for most of these checkpoints because you had no new requirements, software or infrastructure. If you’re unlucky, you had to deal with review committees that only meet on the second Tuesday of each month. At any rate, add the elapsed days and see how you did. Five days or fewer? Give your company a gold star! Four to six weeks? Not too bad. Did it take longer than three months? Embarrassing of course, but take solace in the fact that you’re not alone.

When you have actual requirements to implement, a lot of this overhead is masked. You can do your approvals while there is real coding and testing going on. But this exercise will show you your limitations, and why optimizing your software delivery process is at least as important as automation.

Doing nothing won’t hit all your bottlenecks. In this exercise, we did not look at duplicating a development environment, getting capital for already-purchased production servers or repeating previously passed tests. I’ll talk about those tasks in another post.

How long do you think it would take your organization to do nothing? Leave a comment below.

About the author / Jan S (Bas) Pluim

Bas Pluim is a Cloud/DevOps solution architect in IBM’s innovation practice. He works with clients on infrastructure solutions, process re-engineering, automation tools and transforming the overall software delivery process. Bas is also a member of the IT Specialist certification board, a gaming enthusiast and an advocate of social media tools

Connect with Bas on LinkedIn / Twitter

June 16, 2016
3 Rules for Successful SAFe Adoption

Startups and enterprises alike are redefining—some say disrupting—markets, industries and customer experiences by delivering innovative ideas at a rapid pace and responding to feedback. Often they fail, but they fail quickly and adjust. You know the ones: Spotify, Etsy, Netflix, Airbnb. How do they do it? What can we learn from them that will work in our own organizations? What is different? In a nutshell, all of these successful organizations innovate completely, not only in terms of what they do, but also how they do it. And the common thread is the application of lean and agile practices to improve their culture and deliver innovation to customers faster than the rest of us.

Today’s business imperative is to be the disruptor and more and more organizations are turning to enterprise scaled agile as a set of practices to help lead transformations to achieve that goal. The Scaled Agile Framework (SAFe) is the industry-leading enterprise scaled agile framework with a proven track record in thousands of organizations for several reasons, not the least of which is the incredible breadth of definition and collateral so that you just don’t have to make this stuff up yourself. Nevertheless, there is not an one-size-fits-all framework and SAFe is no exception. But there are three key concepts prescribed by SAFe that, if you do nothing else, will help you be successful in your evolution toward becoming that disruptor.

Go SAFe or Go Home

1. Apply Lean Principles at an Enterprise level
You’ve likely heard the term “MVP,” or minimum viable product. This is about investing “just enough” to deliver a capability—a solution, product or application—and get feedback. It is about avoiding the pitfalls of over-engineering. This is perfect embodiment of lean thinking. By focusing on investing only as much as necessary to get feedback quickly (and doing this continuously) you not only reduce or eliminate wasted resources but also speed time to feedback. Lean is also about the notion that “capacity rules”—in other words, only invest as you have capacity to follow through. Otherwise, what’s the point?

2. Apply Economic Thinking
Traditional methodologies typically focus on priority to rank at all levels, from business initiatives and projects down to team-level work that delivers changes into products and applications. In the agile world, story points (or complexity) also might be used. So what’s the problem with this? First, it is only a single dimension. What about business or customer value? What about cost of delivery or reducing risks or enabling market opportunities? Is timing a factor? All of these aspects should be considered when ranking because, at the end of the day, organizations are looking for the “biggest bang for the buck.” By applying economic thinking, you are maximizing the business’s ability to deliver the most value at the lowest cost in a timely manner.

3. Focus on Delivery of Value
I’ve often heard that revenue generation is the primary goal when organizations look to speed delivery cycle times. In fact, the goal should be delighting the customer; revenue is a bonus, a side effect of happy customers who use your stuff and pay for it! A focus on improving feature delivery is too narrow. What you really want to do is improve time to value. That means articulating what value means, associating the notion of value with features that are delivered, assessing that value and then tracking so you have insight into your organization’s ability to deliver value and get better at it.

SAFe embraces all of these concepts through artifacts and activities anchored in lean and agile practices. Try it, you might be surprised at how impactful this can be in improving your ability to innovate and be the disruptor instead of being disrupted.

Read how a typical enterprise dipped its toe in SAFe water to drive delivery of a critical project, and be sure to visit the SAFe landing page to learn more.

About the Author/ Amy Silberbauer

Amy is a Solution Architect charged with the definition and delivery of the Enterprise Scaled Agile and IBM DevOps Plan solutions and strategy for the IBM Watson IoT line of business. She is a recognized subject matter expert on scaled agile and software development lifecycle solutions, including enterprise modernization, SOA and collaborative DevOps. She has been with IBM for 29 years and has two decades of software development experience as an engineer, architect and manager. She is a certified SAFe® Program Consultant experienced in leading and consulting on multiple internal and external SAFe (Scaled Agile Framework®) transformations.

Connect with Amy on LinkedIn.

June 15, 2016
Do You Continuously Deliver Your CD Pipeline?
If your job is to build and maintain the continuous delivery (CD) pipeline for your organization, do you feel you can’t keep up with your development counterparts in innovation? Everyone is raving about the new cool features in your company’s product. Who deserves the credit? People who wrote the code, right? It’s not that simple. Churning out new features requires a sound process and solid infrastructure. A well-designed CD pipeline is crucial to the developers’ success. It mitigates the inherent risks in a fast-paced, frequent iteration regime now prevalent in agile software shops. On the flip side, as software gets more and more complex and the appetite for a faster time to market grows, opportunities for innovation are in abundance for DevOps teams.

Multi-speed IT

So you’ve got two tactical teams with significantly different risk-reward equations, working at a different pace due to distinct objectives but marching toward the same strategic goal. You’re finding yourself in a multispeed IT environment. Development is focused on innovation and quick frequent iteration cycles. DevOps’ primary focus is to ensure the stability of the CD pipeline and keep a precise track record of what takes place inside the pipeline every step of the way. How do you make sure to stay ahead of the technological curve while minimizing risk?

Based on my experience on both sides of the DevOps isle, I recommend the following four steps to successfully build and maintain a continuous delivery pipeline:

1. Automate, Automate, Automate!

My personal motivations for rolling up my sleeves and scripting a task are:
- It became an annoying, mindless routine;
- It’s no longer a one-time thing and isn’t going away;
- My time can be spent on more important things; or
- I make mistakes and then waste time fixing them.
As with any type of code, organize the scripts into a sound logical structure easy to navigate and version-control it. You can always revise the code or completely rewrite it as the needs evolve.

2. Treat Automation Code as a First-Class Citizen

That’s right! As a practitioner, I learned the hard way that automation starts innocently small and simple. Then the requirements for more automated functionality begin to pile up rather quickly and things quickly become really messy. So write high-quality code from the get-go and reassess the design and implementation every step of the way. Remember that the visibility of your code is very high.

Apply the same standards to your code as your peer development teams do to theirs. Write tests—unit, integration, functional, end-to-end—as appropriate. Use the same CD pipeline to automatically build, package, deploy and test your automation code. This will give your pipeline another round of validation, your code will be of a much better quality, and your team’s credibility and reputation will soar.

3. Containerize the CD Pipeline Components

In the past, because the CD pipeline components such as build servers were manually provisioned, they often became the single points of failure, which made it hard to plan and roll out the modifications to homegrown, open source and commercial tools. It wasn’t unusual for some components to fall behind the latest release by a full year or more. In the meantime, a year’s worth of updates and patches could address a litany of issues, but the need for a maintenance window, usually on a weekend, and the uncertainty of the outcome often would outweigh the sought-after benefits. Containerizing the CD pipeline component can be the solution. Running a build server in a container, for example, will make it a lot easier to create an exact copy of the production version in a test environment and iron out the upgrade process more thoroughly ahead of the upgrade.

4. Test Commercial Tools

Commercial and open source projects test their software, of course. However, it could make a whole lot of sense for your organization to create your own integration, end-to-end and performance tests for the software you don’t write—especially if the software has a plug-in architecture like Jenkins or UrbanCode Deploy and you use community or homegrown plugins. Commercial tools typically don’t provide any guarantees that those plugins will work after the upgrade. Running your own automated tests that exercise the functionality of the tool in your environment and on your specific data gives the opportunity to validate updates as they become available and identify any regressions proactively.

The objectives of DevOps are not only to identify and eliminate the existing bottleneck in the organization’s continuous delivery process, but also to look out for any future potential bottlenecks and proactively address them before they become real issues.

So your team is DevOps. Your product is a CD pipeline. How do you continuously improve the CD pipeline for your product?

If want to learn more about how to effectively build and maintain your continuous delivery pipeline, please check out the newly released 2nd edition of the “Application Deployment and Release for Dummies” eBook.

About the author / Alex Seriy

Alex is an accomplished technologist and DevOps evangelist with over 25 years of experience under the belt. He is an automation enthusiast and usability guru, passionate about uncovering new use cases and innovating applications for platforms, frameworks and tools. At IBM, Alex is a subject matter expert in Continuous Integration, Delivery and Testing. He currently focuses on integration testing in the API Economy through service virtualization. Prior to joining IBM, Alex served as the Senior DevOps Architect at TIAA-CREF for 4 years. Before that, he was a consultant specializing in SCM, Release Engineering, QA and DevOps for over 10 years.

Connect with Alex on LinkedIn / Twitter
June 14, 2016
What Happens to Test in a DevOps World?
I am always surprised by the urban legends around DevOps. One of my favorites is, If you implement a DevOps methodology, you can stop doing test. People famously quote that Netflix only tests in production as evidence of this. This is perhaps more like of game of Telephone, where much of the story gets lost in translation. It is true that Netflix created a tool called “Chaos Monkey” which generates failures in production as a way to test the fault tolerance of its software in large-scale production use. However, this is not Netflix’s only test; it is an additional type of testing at scale that allows the company to test in new and different ways. Netflix still tests in software development.

Okay, now that we are past the notion that DevOps is about eliminating testing, let’s look at what really happens with test in the DevOps world. In fact, the goal is for testing to happen everywhere and all the time. Let’s look at the two directions test can shift: left or right.

Shift Left

DevOps methodology encourages automation of activities across the entire development process. This can be realized by automating the link between development and test. Often implemented with an automated build tool such as Jenkins, a set of automated tests are triggered whenever a developer completes a build of new code. Of course, this means that both development and testing processes must be automated. Once you reach this goal, then testing can happen all the time. Every time developers want to check in new code, they can trigger a small set of “smoke tests.” If those succeed, they can check in code and when the build happens, they can trigger a larger set of regression tests. The result is a left shift of testing into development.

Benefits of left-shifting test include:
- Move testers into scrum teams to work on stories together with developers.
- Design tests in parallel with development.
- Give developers and testers the flexibility to help each other to reduce bottlenecks.
- Early detection of bugs, shortening the critical metric for time between creating a bug and fixing it.
- Increased transparency of code dependency issues because they are raised during the nightly (or more often) regression testing.
- Virtual elimination of the test bottleneck that occurs after development is completed.
Shift Right

The urban legend about Netflix may be wrong, but that doesn’t mean that there isn’t merit to right-shifting test into production. Automated testing in production is effective for tests that would be impossible to perform outside of production, including scale testing and resiliency testing. Production is also a great place to introduce A/B testing by offering different interfaces to some customers and measuring the differences in business results.

Benefits of testing in production include:
- Better understanding of the real production environment and workloads.
- Early feedback from customers on new features and user experience.
- Better resiliency.
- Overall better user experience for customers.
No Shift

Wait, if we shift left and shift right, is there any testing left to be performed in test? There are still a range of tests that need to be performed by the test team. With full automation, these tests can be performed at a variety of times and don’t need to be a bottleneck to release. These tests include:
- Performance testing
- End-to-end use-case testing
- Security testing (don’t think about doing this in production!)
- Manual testing – often for user interface or mobile testing that is most difficult to automate
DevOps methodology is not promoting the end of testing, but in fact a widening of testing to encompass all of the steps in the DevOps cycle. You know what they say: Test early and often! And next time you are sitting around the campfire discussing urban legends, let’s leave Netflix out.

About the Author/Joan Wrabetz

Joan Wrabetz is the CTO for QualiSystems. Most recently, she was the VP and CTO for the Emerging Product Division of EMC. Joan received her MBA at the University of California, Berkley, and MSEE at Stanford, and a BSEE at Yale. She has held teaching positions at the University of St. Thomas, St. Mary’s University and the University of St. Thomas, St. Mary’s University. Joan holds patents in load balancing, distributed systems, and machine learning classification and analytics. Her experience inclues: founder and CEO of Aumni Data, CEO of Tricord Systems, Vice President and General Manager for SAN operations at StorageTek, Founder and CEO of Aggregate Computing, Management and senior technical positions at Control Data Corporation and SRI International, Partner with BlueStream Ventures.
June 13, 2016