devopsy.kinsta.cloud

Tag: continuous delivery

7 Causes of Continuous Delirium

When we think of CD, we typically think of continuous deployment as a result of continuous delivery, which is a goal of DevOps.

In this article, I coin the phrase, “Continuous Delirium,” to mean unending confusion or bewilderment in development environments. There are probably several examples that could fit the bill here, but for now I offer seven real-world examples. I bet you’ve seen or heard about at least one of these.

Now that I have your attention, here are seven causes of Continuous Delirium, each with cures rooted in DevOps:

No. 1: Insufficient Configuration and Build Management

Insufficient configuration and build management in software builds leads to the same or similar software impurities rearing their ugly heads again and again. You build, test, discover the defect, fix it—or so you think—build again, retest and boom: The same coding error or software malfunction recurs.

“You can cure this malady with a rational approach to the build and configuration management process, checking software in and out with established procedures for each software module using a tracking sheet that identifies the revision of the sub-modules for the entire software product,” notes Jon M. Quigley, founder of Value Transformation, a product development consultancy and training firm based in Lexington, North Carolina.

No. 2: What’s Going On? Who’s Doing What?

When there is no reasonable understanding of the work, who is doing what work and who is responsible for what—and there is no consistent measure of what is good work or bad work—you have a breeding ground for anarchy.

“You cure this ill by establishing what constitutes the appropriate quality of work as well as by periodically reviewing the state of the objective with the receiving or depending entity,” says Quigley. Make sure everyone is on the same page with the same definition of what constitutes quality work and make sure the objectives have not changed. Also, ensure these goals are also clearly understood by all.

No. 3: Confusing Clean Daily Compiles with Testing

When you do not test the software as early as possible and throughout the development process, you can’t really say it has no bugs. Error-free compiling is not error-free software. Software compilation does not have the same investigative effect as using the software in the customer environment.

“Test throughout development using complimentary testing approaches such as testing to requirements, combinations of stimulus testing and exploratory testing, for examples,” says Quigley.

No. 4: Planning Too Far Out to Control

The further you try to plan things out into the future, the less control you have as you follow the project through to conclusion. “When we treat a plan as though we know everything associated with the project for the next 12 to 18 months, we are fooling ourselves,” says Quigley. By breaking projects into bite-sized increments, you can better assure successful completion of sub-projects and of the project as a whole.

No. 5: A Lack of Customer Involvement

Customer involvement is critical to customer satisfaction. “By getting the product or a prototype into the customers’ hands for exploration, you can retrieve feedback for developers,” says Quigley. This may expose the project to risk in the form of the potential for a great deal of change at the customers’ request, but the other side of the coin is the risk that nothing changes and the customer doesn’t use the product at all.

“Rather than try to deliver one incarnation of the product at some future date, break down the delivery into separate iterations that meet prioritized customer expectations one by one as you refine the product and add features as you go along,” says Quigley.

No. 6: Isolating Change Management from Customer Involvement

Customer involvement means there will be change. Change requires change management. There is a given—and correct—assumption that customer involvement will trigger change management at some point. You cannot isolate the one from the other.

Further, just as you can better address customer requests for changes by breaking the project into smaller subsections, so also can you manage budget through change and change management. Project subsections enable you to see the need for change a little at a time so you can create change in a more targeted and cost-effective manner.

No. 7: Forget About Collaboration—at Your Own Peril

It is necessary for DevOps team members to collaborate at all levels, except those where the work on software is so minor that it wouldn’t benefit from shared thought. To maintain development as an expedient process that benefits from all stakeholders, open communications and collaboration must reach and pull from all team members. Otherwise, the project could suffer from a lack of critical input.

April 29, 2016
Legacy IT Challenges Continuous Delivery
Legacy information technology infrastructure has a strong effect on the potential success of DevOps. In environments where different teams have always used the same servers, change can be difficult due to inter-team coordination and limited server transparency and application inventories, says David Blank-Edelman, a 30-year systems administration/DevOps/SRE veteran and currently Technical Evangelist at Apcera.

“Unraveling the Gordian knot to bring DevOps principles and practices into that environment will be quite a challenge,” he says.

In such a legacy environment, the enterprise walks a topological tightrope stretched between one or more development machine clusters shared by multiple teams with no real isolation between teams and projects, Blank-Edelman notes. “In most cases, there is a second parallel set of machines for production and, if the business is very fortunate, for test/QA as well. There is seldom anything, however, to prevent a production workload from talking to a database server in development by mistake, which can lead to all sorts of fun.”

How to Move to CD

One solution is a policy system approach, in which the organization provides for a policy system to migrate a legacy information technology environment to a platform that uses continuous delivery (CD), Blank-Edelman says.

To create the policy system, he notes, the organization will need to address questions such as:
1. Are the workloads allocated/using only the resources that they should?
2. What is inside the workload? What is its provenance? Is it acceptable and in good order from security, coding and business requirements perspectives?
3. Where is the workload allowed to run? In which cloud? In which environment? In which country? Near what other containers?
4. Can the workload communicate with only the things it should? If I move it to another location, is this still true?
“The process of developing a policy system and then enacting it can lead an organization on a journey that brings their existing infrastructure to a much better place. If that platform can also automatically enforce the decisions that organization made during this question and answer discovery process, you can be certain that the answers to these questions stay answered,” says Blank-Edelman.

Other approaches to moving to continuous delivery include tool-first approaches in which the organization adopts one to several CI/CD/workflow tools and then makes an effort to implement those, possibly by shoehorning existing processes and runbooks into that tool or toolset.

A better tactic is the people-first or culture-first approach that focuses on how the development and operations personnel will work together to create a product vs. a project, says Blank-Edelman. While tools still have a role to play in the people-first approach, they are not the priority; rather, organizational change comes first. Once the people have made the commitment and the change, everything else will follow.

Challenges During the CD Move

It’s not the existing information technology infrastructure that presents moderate challenges to CD that is the issue; it’s the infrastructure that poses either very many challenges or very few that is the hurdle. “If you start with a more challenging situation, this will incentivize people to work very hard to reduce the associated pain points in their life,” Blank-Edelman says.

On the other hand, he notes, when an organization begins a move to CD with an infrastructure that is already in pretty good shape, you won’t often hear its people say that they should stop working on the product and work on CD all the time instead. People will still claim they have plenty to do already.

Continuous Deployment Can’t Exist Without CD

Continuous delivery starts with a code base and takes logical steps through that code base to increase confidence that it is ready for production, using a process that is as smooth, automated and loosely coupled as possible, he says. With confidence in that pipeline comes confidence in moving the result of that process into production automatically, which is continuous deployment, he adds.
April 20, 2016
Test Better, Test Faster, Test Smarter
How do you balance the need to “go fast” with the need to test everything and deliver high-quality software?

With applications the driving force in today’s economy, the quality and release cadence of your software are critical to your business and your bottom line. You want to get software updated in the hands of your end users as quickly as possible. However, even the fastest release cycle wouldn’t make a difference if your product ends up being buggy or malfunctioning, resulting in a bad user experience or service interruption.

I want to take a look at one of the most important (if not the most important) aspects of the software delivery process: testing! As software teams become more mature in their implementation of DevOps and continuous delivery to streamline their software delivery, testing takes a central role as a prerequisite to these practices.

You need to have testing on your mind from the very beginning, and plan your testing as part of the initial stages of designing your software—mapping your test procedures, infrastructure configuration and acceptance criteria. As you design and develop your app, you also need to think about how you’re going to test it. The more testing is on your mind, the more you’ll have a solid understanding of what you need to test, what’s the definition of “done,” and the faster, more streamlined, your overall process will be.

Let us review four questions every development or QA team should consider when planning their testing, as they try to balance quality with speed—managing and accelerating their testing, while ensuring a reasonable time-to-market.

How Much Testing is Enough?

The age-old question: How much testing do I really need to do? Unfortunately, there is no concrete answer. But with a bit of initial legwork, you can get a good feel for how much testing is necessary.

The amount of testing that is necessary is influenced largely by the industry you’re in. If you are testing mission-critical aircraft software, let’s hope it’s very thoroughly tested! On the other hand, if you’re building a game app, the amount of necessary testing may be lower. The truth is when you run tests, the success rate is usually not 100 percent. Therefore, the key is to define what, for your specific piece of software, can reasonably be considered “done.” Instead of arbitrarily deciding at some point that your release is ready to ship, make the decision before the fact: What are we going to accept? What level of pass/fail are we willing to roll out with? What are we comfortable with based on what we have done in the past?

The amount of testing required can sometimes be a fluid thing. A good way to measure your comfort level with your testing procedures and the amount of testing you’ve done, is to ask yourself, “If it was now 11 o’clock at night, and I needed to ship this code to my end users, am I sufficiently comfortable rolling this out, or am I not sure how it’s going to behave in the wild?” If you’re leaning toward the latter, than it’s probably an indication that you should have done some more testing. That’s a gut feeling and depending on the industry, the point at which you’ll fire off a release and not have any worries can vary greatly. And, obviously, for some regulated industries, you’ll need to prove compliance and ensure you’ve recorded that all tests were successful before you can even start thinking about releasing anything.

What Kind of Testing Should You Invest in?

Decide what kinds of things you really care about and what you need to verify and test for. What are the requirements that we expect from the application, and from the different stakeholders in this project?

Every piece of software needs to be tested differently. Because of this, you have to plan what the testing will be. You can’t just say, “I’m going to unit test the heck out of my code, each and every time, no matter what.” Sometimes, unit testing isn’t the most important test you need to be running.

The investment has to be in automation. No matter what type of tests you decide are most functional and relevant to your project—whether you’re doing a ton of unit testing or a bunch of functional stress tests—it needs to be automated.

Particularly, as you open up your services to the world, you need to put into place the relevant performance and reliability tests so that if suddenly your load increases by 200 percent, you have a plan for how to handle that. Are you going to autoscale? Can you autoscale? And finally, do you even need to auto-scale?

All of those questions need to be answered as part of your development cycle. Automating your tests will let you not only create tests for these scenarios, but it will let you run those tests over and over. By automating your tests, you can establish a baseline and see where you deviate.

It can also be helpful to think of testing in a different way. Don’t look at testing as a tool to validate technical requirements, but as a way to make your customers happy. You want to make sure your tests are directly contributing to your customer’s overall experience. So start with the highest-level tests that look at things such as overall business functionality—and make sure the software works at the business level.

How Can You Make Testing Faster?

Automate, automate, automate! That might seem obvious, but try to think outside the box. Find things you thought couldn’t be automated and wrangle them into shape. Remember that every minute that one of your developers is sitting there running a test that could be automated, it is costing you time (and money) that he or she could have spent on more productive things. In the long run, not automating can cost much more than making the commitment to doing it right.

Other simple things you can look at to increase the speed of your testing is to make sure your development environments are mirror images of your production environments, and that you have fidelity of configuration throughout the lower and higher environment.

It is also helpful to make sure you’re defining user requirements that are thin vertical slices, instead of trying to test huge cumbersome pieces of functionality. The ideal situation is if you can run only tests against components that have changed. Componentization of both your application and your testing procedures is critical because you do not want to have to rebuild and re-test everything all the time. Remember, also, that if you make incremental changes to your code and frequently release them to your users (Agile, anyone?), you will get just enough feedback to improve quickly and reliably. Be cognizant of not biting off more than you can chew.

Segregating the tests to several phases is a good way to accelerate your CI and test cycle—but make sure that both your application, build and test batteries are designed for that. With this approach, you are essentially running multiple phases of CI, starting with the fastest cycles that require the smallest build(s), with minimal dependencies, and move on to the outer/cascading cycles of tests. Following a submit, run a minimal build, so that you do not need to wait—sometimes many hours—to run an extensive set of all your tests with every single commit to see if it passes. Start with unit tests to minimize the build and provide fast feedback, then in various junctions incorporate additional tests, such as integration tests, installers and usability and performance tests.

Another way to streamline your testing process is to take a look at the entire scope of your build and test cycle and go after the most inefficient stage. After automating your manual processes, look at the bottlenecks and what takes the most time and figure out how you can optimize that specific process. Often the answer is parallelization, but other times it is hardware. Iterate on your process and figure out where you can gain the most ground!

Lastly, parallelization and distribution of the build and tests provide a huge time saver, so consider running tests across a cluster of CPUs to cut down the run time.

Where Do You Start if You Want to Make Sure All Your Builds Are Green?

Ask yourself, What are you trying to achieve? Is it a process where you can release code more quickly? Or where you can address bugs more readily? Then make sure you’re intimately aware of what you have in place. You need to know every step, task, process and tiny bit of your testing procedures and supporting infrastructure. Draw your entire end-to-end workflow on a whiteboard and find the bottlenecks that are slowing it down.

Once you’ve defined your objectives and are aware of your current state, you can put the relevant tools in place—from development environment to your testing frameworks all the way to post-dev tools such as your analytics tools—design the right testing pipelines and hire the relevant people with the relevant skills. Don’t insist on making all the mistakes yourself (tongue firmly in cheek), but find folks who have already made the mistakes and learned from them—and hire them!

Also, give your team a great system that gives feedback fast. Do frequent code reviews so there is accountability and a constant learning process.

Consider the following best practices for improving your testing processes:
1. Be sure to plan what your testing approach will be ahead of time if possible. When you’re designing your software, designing how you’re going to test it is important.
2. The amount of testing you need to do is dependent on what industry you’re in and your specific use case.
3. Consider what level of test pass/fail you’re comfortable with, based on what you’ve done in the past.
4. Consider pre-commit or pre-flight builds to ensure your tests are more likely to pass once checked to Mainline
5. Define your requirements for each test cycle
6. Segregate your tests to “fast” and “slow” and decide which battery of tests you need to run every time, and which processes you only need to run at certain junctions.
7. If you hire well and bring new skills to your team, your process will get better and faster.
8. Consider what problems your customers are facing and perform an analysis. Use whatever tools are best to test your software like a customer would to achieve the highest end user satisfaction.
9. Automate manual steps that you take for granted to accelerate cycle times and improve productivity. Speed comes second.
10. Iterate on the testing process, and not just on the code itself. You should always be looking for new ways to test.
11. Test your test processes. Hold regular meetings with your team to reassess if your testing strategy is working for you.
12. Enact code reviews so that there is accountability with the rest of the team.
As you map, design and optimize your tests, remember that your testing policies don’t need to be set in stone, and that you can—and should—adjust and tweak as you go.

Happy testing!

About the Author/Anders Wallgren

Anders Wallgren is chief technology officer at Electric Cloud. Anders has more than 25 years’ experience designing and building commercial software. Prior to joining Electric Cloud, he held executive positions at Aceva, Archistra, and Impresse and management positions at Macromedia (MACR), Common Ground Software, and Verity (VRTY), where he played critical technical leadership roles in delivering awardwinning technologies such as Macromedia’s Director 7. Anders holds a B.SC from MIT. He invites you to download the community edition of ElectricFlow to build your pipelines and deploy any application, to any environment for free.
April 18, 2016
Optimizing an Effective CI/CD Pipeline
Continuous integration and continuous delivery (CI/CD) are often cited as pillars of successful DevOps. To establish and optimize the CI/CD development model and reap the benefits, companies need to build an effective pipeline to automate their build, integration and testing processes. This requires architecting CI/CD with five key ingredients which, when combined effectively, operate as a successful cornerstone of DevOps and provide myriad bottom-line benefits.

Building the Pipeline

Let’s first take a look at the five essential ingredients needed for a healthy CI/CD architecture, following up with a discussion of their benefits to the business:
- Micro-agility in the form of testing and committing code frequently: The development life cycle begins with a request to development, which then takes the request and designs a solution, codes it, tests it and commits it a code repository. From here, the code is tested by QA and then deployed to production. While each step is important, we begin here, as local testing occurs most frequently and bugs found at this stage of the process help avoid the large expense of bugs found in production.To achieve micro-agility, developers must have a clear understanding of where they are expected to commit their code after changes are made and they should be expected to test and update their code frequently—many times a day. This means development testing should be easy to run locally and is a subset of QA. The result is that code pushed by development is by definition QA-ready.
- Automated QA: In my work I see many organizations whose QA team is one of the largest bottlenecks in the process. It’s not for a lack of skill, but too few staff who spend a good portion of their time waiting on the QA test environments they need to do their work. To avoid this situation, the QA process should be fully automated, from QA ready code coming in the door to one-click promotion on the other side. In between the two, QA should have a constant feedback loop with development where communication about—and tracking of—bugs is clear and easy to follow, allowing development to easily repair issues and recommit the code for further testing. Moreover, automated QA in a CI/CD environment should focus on microcycles consisting of small test batches rather than weeklong macro tests. Automation and communication are the overriding imperatives here, exhibited by reports to developers that are communicated immediately and code commits that are fail-rejected automatically.
- Production-like environments: To increase efficiency and effectiveness, development and testing environments should mirror production as closely as possible. They should follow the PRQ rule, as I like to call it—production-like, repeatable and quick. Specifically, it should be fast and easy to set up new environments, to the point where developers can, with the push of a button, deploy new environments for their development and testing needs. Doing so allows developers to fail fast, breaking and re-creating environments without fear as they know they can easily recreate the same environment every time. I often recommend using tools such as Jenkins to help with this process; if developers and IT operations use the same tool, it makes it that much easier to streamline the CI/CD process and achieve the promise of DevOps.
- One-click promotion: As I alluded to earlier, a well-architected CI/CD pipeline provides one-click promotion to production. This ensures changes are seamlessly moved to the overall production environment with the least amount of friction possible between development and operations. To that end, this reduction in friction should occur across all the steps of the process, with efficient code delivery pipelines that are designed to balance security and agility from the outset. Cloud infrastructure coupled with container technology and automation can easily optimize the process across teams and throughout the DevOps journey.
- Fully automated deployments: The last factor of an efficient CI/CD pipeline is full automation of deployment. Any configuration changes made by development and accepted by QA should automatically be applied. Continuous integration in this way helps avoid system downtime and human errors are decreased greatly. Deployments also should be continuously monitored so that IT is the first to learn of potential problems. This helps nip issues in the bud, allowing for faster rollback and return of code to development to fix anything that may not be functioning properly. Catching issues before customers do helps ensure customer satisfaction and strength of reputation.
Benefits of the Pipeline

In addition to these benefits, when combined effectively, these ingredients minimize the time and cost of local test, QA and production processes; reduce the time it takes to identify problems and their causes; and accelerate delivery of production-ready code. We can see this throughout the CI/CD pipeline:
- Developers have greater control over the process and can respond more quickly to market demands by quickly bringing up new environments, testing against them and easily starting over, if need be. Moreover, developers can quickly adjust to new environments—an approach that has proven to decrease failures and resolution time when they do occur.
- Operations benefits as reduced friction in the CI/CD pipeline created by automation decreases repetitive, manual work and with it the opportunity for “fat finger” mistakes and the greater opportunity to introduce inefficiency into the process.
- And, less manual, repetitive work provides everyone in the process with more time for strategic work, which provides direct bottom-line value to the organization.
CI/CD is a pillar of successful DevOps and according to research by CA, organizations that apply DevOps discover an 18 percent reduction in time to market of new applications and a 21 percent increase in delivering new software that they would not otherwise be able to complete. Teams that master an efficient CI/CD pipeline deliver faster, higher-quality applications to market bringing with it a 19 percent increase in revenue. Mixing these ingredients together results in an efficient, streamlined CI/CD pipeline that serves as a fundamental pillar of DevOps success that, when mixed together, delivers direct business benefits.

About the Author/Aater Suleman

Aater Suleman, CEO & co-founder at Flux7, is an industry veteran in performance optimization on servers and distributed systems. He earned his PhD at the University of Texas at Austin, where he also currently teaches computer systems design and architecture. As a faculty member at UT-Austin, Mr. Suleman focuses on optimizing DevOps architectures involving the AWS cloud. Along with the Flux7 consultant team, he works at every step of the dev flow, from creating local developer environments using Vagrant and Docker, to continuous integration and continuous delivery using Jenkins or RunDeck, to configuration management using Chef/Puppet/Ansible, to creating efficient systems for deploying new code from Git using Capistrano/Fabric/bash. His background is as an architect, and he has experience with both hardware and software. This combination brings a unique perspective and broad view to development and operations.
April 15, 2016
DevOpsQA NJ Meetup: Jenkins with Gerrit and BDD Automation

DevOpsQA NJ recently held its April Meetup on Automation with Jenkins and BDD, with Kishore Bhatia from CloudBees and Tom Bartolucci from Billtrust as speakers. The venue, food and refreshments were provided by Audible at a state-of-the-art, vast office space in historical city of Newark.

After fun networking over pizza, wine and beer, we kicked off with Tom Bartolucci’s presentation on BDD Automation framework using Codeception. Tom has been in web software development for over 13 years. He has built applications in Java, .NET, PHP, Python and Javascript for clients in publishing and mobile video. Currently, he is a VP of Development at Billtrust – a SaaS-based payment cycle management company. His team of 13 web developers works on delivering six of the Billtrust SaaS products and uses Codeception, a PHP-based BDD framework to test.

According to Tom (and we all agree), the idea is not to find bugs but prevent them. His team at Billtrust chose Codeception because it allows tests to be written in PHP, which is what its development teams use; it also supports Selenium WebDriver, PhantomJS and BrowserStack; and has hooks for various tools and platforms and build-in reports. As a result, it had a low barrier of entry and got Codeception’s team off the ground quickly. Doing cross-browser testing allowed them to find IE bugs, which they previously missed due to testing primarily in Firefox. Codeception allows testers to create BDD-style readable tests, similar to Cucumber spec but written as code. The easy syntax enabled the QA team to write code that they could read and understand. And developers are able to add custom modules because the framework is developed in PHP, which they are familiar with. Codeception has a variety of uses including to write acceptance tests, unit tests and API tests.

Tom also described how Billtrust integrated Codeception with Jenkins to execute tests and publish reports. He then proceeded with a live demo of the framework. At the end of the session there were a lot of good questions on Vagrant integration, parallelization, failed test rerun and differences between Codeception and Cucumber.

Next, Kishore Bhatia from Cloudbees gave a talk on Jenkins pipelines and Gerrit Code reviews. Kishore Bhatia works for CloudBees, building custom frameworks with open source software and helping customers solve engineering problems around continuous delivery and DevOps at scale. Kishore is a Jenkins expert with development experience in FinTech enterprise software and Cloud platforms. Couple of months back, Kishore and Issac gave a presentation on Jenkins Workflow, demonstrating how workflow removes bottleneck of connecting multiple pipelines together.

This time, the focus was on a very valid use case: a large codebase, many teams contributing to the code. Teams should not collaborate on emails. They should get notifications and post-commit validations done by the tool that they use. In this case it’s Gerrit—Google open source project, offers its own implementation of Git. You just want the job to magically run behind the scenes doing the code reviews for you. The primary challenges in this scenario are that you want to fail-fast—report CI results as soon as any steps fail and make sure there are no orphaned builds.

Kishore then demonstrated the steps involved in implementing this use case in Jenkins including adding pull request validation to the build step, allowing developers to get build status directly in the VCS tool only coming back to Jenkins if something fails. Currently there is a limitation with posting each status to GitHub. Key things that need to be set up—Gerrit infrastructure, Jenkins installation, workflow that Jenkins will refer to. While it might look complex at first it becomes easier once you start using the workflow, with such things as Groovy script helper. Jenkinsfile will define the flow of what should happen in dev, QA, staging and production.
Kishore then gave a great demo of the flow and answered questions from the audience which covered parallel commits and test data management.

At the end of the sessions, we did a raffle drawing where three lucky winners got prizes sponsored by Audible, including an Amazon Echo. Looking forward to our next session.

If you would like to learn more about Jenkins Workflow, you can reach Kishore at kbhatia@cloudbees.com. To learn more about exciting opportunities at Audible visit http://about.audible.com/ or reach out to sonatore@audible.com.

April 15, 2016
Continuous Testing Live Network DevOps
My earlier blog, “From Laboratories to DevOps: Factories for Continuous Testing,” discussed the importance of and suggested practices for a fully automated, orchestrated software DevOps laboratory infrastructure, akin to a modern factory staffed by robots churning out products at a dizzy pace. The blog applies to development of software for networks, and the practices increasingly are important as the network becomes more software-based thanks to software-defined networking (SDN), network functions virtualization infrastructure (NFVI) and cloud-based networking. The edges of mobile networks, even the radio access networks, are becoming more software-dependent due to the need to position computing resources closer to mobile customers to resolve latency, bandwidth and spectrum concerns.

However, the need for highly automated software processes does not stop within the development lab.

At a recent DevOps Networking event in Santa Clara, California, Google Network Operations Manager Anees Shaikh indicated that Google’s live network has many thousands of nodes and management elements, supported by millions of lines of software-based configuration files that are updated tens of thousands of times per month. The sheer volume and speed of software updates for large networks is not feasible using manual methods. Automation tools for live network software updates are vital.

Typical live network DevOps environments use DevOps continuous tool chains such as Git for software version management, YAML for defining abstract configuration definitions, Jinja2 to translate the configurations into concrete templates that can be consumed by real devices and Ansible for automating the distribution and activation of config files to the network nodes. But what about testing? How can a network operator be confident that the many updates will perform as expected once deployed across the live network? This requires much more than simple syntax and semantics checking of the config file entries.

Recommended Practices for Continuous Testing of Live Network DevOps Environments

For continuous testing of live network DevOps environments to work, here are some best practices to consider:
1. Lab-to-live transition: Prior to deployment of any updates to the live network software updates should be tested in a lab environment using production-equivalent equipment and topologies.
2. What to test? The actual tests to be executed depend on the risk of changes that are being deployed. If a small, low-risk configuration change is all that is being deployed, then a simple configuration check may be sufficient. On the other hand, if a change has the potential to disrupt the network performance significantly, then a much more comprehensive performance test is needed.
3. Test orchestration tools: The test environment prior to deployment will vary depending on the tests that are selected. Test environment orchestration tools are needed to set up network test topologies that match the tests rapidly—and release them just as rapidly so the test resources can be used for the next tests.
4. Elastic on-demand topologies: The ability to scale test resources horizontally and vertically, on-demand, is essential to be able to match the speed of testing required for short duration deployment cycles.
5. Automate live network tests during deployment: To verify software updates do not interrupt or degrade network performance, functional and performance tests between pairs of nodes across the network mesh are required as the updates are deployed in stages across the live network nodes, clusters, centers and regions.
6. Automate live network testing after deployment: After verifying that the deployments are successful, network tests need to be scheduled periodically to ensure latent problems with the new software do not impact the network.
This is just a partial list of suggestions for continuous testing essentials for live network DevOps.
April 11, 2016
Submit your proposal for DevOps Enterprise Summit San Francisco!

The theme for this year’s #DOES16 San Francisco conference is Leading Change.

We want to hear stories from the leaders of large, complex organizations implementing DevOps principles and practices.

The submission deadline is midnight PDT, May 2, 2016.

All submissions will be notified by June 17, 2016.

Get details >>>

This year’s programming will be based on a series of survey questions about the biggest issues IT professionals continue to work with. Several universal topics emerged, and we will be selecting talks that address them:

People and Culture: We’re interested in the strategies and methods you employed to successfully lead change and to enable DevOps outcomes in large organizations. How did your approach to leadership and organizational design differ from those used in more traditional IT organizations? How is the workforce different, both for employees as well as contractors and outsourcers? And what have been your results?

Process and Technology: We are interested in the practices you have adopted to integrate change into your workforce and build on changing relationships with your vendors. What are modern architectural and technical practices that every technology leader needs to know about?

Compliance and Security: We’re interested in concrete ways you have helped DevOps bridge the information security and compliance gap, showing auditors and regulators that effective controls exist to prevent, detect, and correct problems.
Submit your proposal >>>

For the full news release, click here.

EARLY BIRD TICKETS ON SALE NOW!

If you missed the Blind Bird tickets, don’t fret.

We are offering a discounted Early Bird rate for $1,450 (ends May 16). That’s a savings of more than 25% off the full-price ticket.

We encourage you to act quickly as the past two years have sold out.

Get tickets >>>

April 11, 2016
DevOps and Continuous Delivery: Not the Same

DevOps and continuous delivery are the same thing, right? Far from it. We peer through the fog to understand the difference between the two related but distinct methodologies.

We live in an era of apps. More and more of our personal, social and work life is governed by apps—communicating with one another through social apps, booking cinema tickets through your mobile device or scheduling a business trip through the airline app. This digitization of services has arrived. Organizations that understand this sea change have a unique opportunity to jump ahead of their competitors, acquire customers more quickly and grow revenues.

The battleground for this service development is DevOps. It’s not a product, but rather a methodology for helping organizations build teams and build software. DevOps is a culture, movement or practice that emphasizes the collaboration and communication of both software developers and other IT professionals while automating the process of software delivery and infrastructure changes. It creates a culture and environment where building, testing and releasing software can happen rapidly, frequently and more reliably.

Organizations are marching with their feet. According to a recent study, market demand for DevOps skills is growing, and DevOps engineers are among the highest paid IT practitioners today. And it looks set to stay that way. Research in the U.K. shows that 30 percent of 300 organizations surveyed in the run up to the 2015 DevOps Summit had either started to implement a DevOps strategy or had already merged Dev and Ops, with another 13 percent planning to go down this route.

Two Related Children: DevOps and Continuous Delivery

So now we arrive at continuous delivery. If DevOps is about accelerating the delivery of new, high-quality services, surely continuous delivery is the same thing, isn’t it? In other words, ensuring a steady stream of new services? You might look at it as a manufacturing production line: DevOps is the machine that builds the service, while continuous delivery is the conveyer belt that rolls the services off the production line—one big unified service development cycle.

That’s not the case. Although the distinction between DevOps and continuous delivery can be a little grey and cause confusion, there is in reality clear water between the two. Continuous delivery has become an essential ingredient for teams doing iterative and incremental software delivery. It is an approach whereby teams ensure that every change to the system can be released, and that any version can be released at the push of a button. Think of it this way: Continuous delivery aims to make releases dull and reliable, so organizations can deliver frequently, at less risk, and get feedback faster from the end users until deployment becomes an integral part of the business process and competitiveness of the enterprise.

Gartner summed up the distinction between DevOps and continuous delivery very neatly with a report that stated, “DevOps is not a market, but a tool-centric philosophy that supports a Continuous Delivery value chain.” The analyst argues that bimodal and digital business strategies stimulate demand for improved speed and effectiveness of software delivery.

Continuous delivery and DevOps do share common traits, though. Both methodologies are aimed at agile and lean thinking: each delivers small and quick changes; each relies on tight business an IT collaboration; and each share the common goal of faster time to market for new services.

However, the goal of DevOps is merge dev and ops roles together—and the processes they manage—to achieve business goals. It’s all about a common, shared culture and enhanced collaboration. About clearly defined business processes.

Automation has an essential role to play in both DevOps and continuous delivery. Automating DevOps helps your organization manage and secure continuous delivery to the business and IT users. The result is a faster, low-risk migration to a new world of composite applications, faster business innovation and better support for rapidly changing business process needs.

April 8, 2016
Rookie Mistake: Not Managing Mobile Customer Feedback
Managing mobile customer feedback effectively could be the single-most important success factor for your company and customers. Call it addiction or habit, but every passing day we encounter new data confirming the growth of mobile app usage and total hours spent interacting with them. In the “2015 U.S. Mobile App Report,” the CoreMetrics team has shared some useful data metrics regarding digital media consumption growth and app usage. A couple of notable themes or trends are:
1. Americans across all age groups spend increasing time on Mobile Apps: By mid-2015, Americans on average had spent approximately 68 hours per month on their mobile devices. That number rises even further to more than 90 hours per month among millenials.
2. A whopping 50 percent of time is spent within a single app and about 90 percent on the top five apps: Not only is the average time spent on mobile app usage increasing, it is concentrated in a few limited apps. A staggering 50 percent of their time is spent working within a single app, and that number tapers off rapidly as popularity, utility and quality decreases.
It is evident from this data that you want customers to include your app on their home screen. That means not only having a compelling utility, but also ensuring that the app is of the highest quality. In this article I will illustrate how you can accomplish that by incorporating mobile customer feedback loops in every stage of the development cycle and involving and engaging customers through your path to production. Along the way, we will look at various open source and commercial tools that enhance the ability to collect feedback at various stages of the continuous delivery pipeline.

Typical Continuous Delivery Pipeline

Most agile or lean software organizations have an automated continuous integration and delivery pipeline that helps manage the flow of release artifacts to production. While there might be nuanced variations across teams and enterprises, the typical path to production for apps or services is represented through the steps shown:

It will be useful to review this pipeline as we explore options to incorporate feedback throughout the path to production.

Continuous Delivery and Mobile Customer Feedback

Not too long ago, most negative user feedback consisted solely of poor ratings, comments and rants on social media platforms. Without having context, metrics and data, most development teams were left guessing on how best to reproduce and determine extent of the issue. This usually resulted in issues and features getting prioritized in an ad hoc manner.

Today, numerous open source and commercial tools are available that help solve the feedback problem. Most of them can be categorized in the following broad areas:
- Instrumentation
  Despite recent reports that Facebook made its Android App crash to test user loyalty, most app crashes are not of the deliberate nature. There are a plethora of crash testing and reporting solutions, the most popular being Crashlytics (part of Twitter Fabric), Apteligent (formerly Crittercism), TestFlight (Apple) and HockeyApp (Microsoft). They each provide a client-side SDK to capture and report crash reports and a server-side service to to manage and analyze those reports.
- Phased Releases – Alpha, Beta, Staged Rollout
  Android application developers can take advantage of the alpha, beta and staged rollout capabilities available as part of Google Play. For alpha and beta tests, production APK is not required. Developers have a number of options to target specific groups for their alpha and beta tests. They also can utilize feedback channels available within the app to obtain feedback. Staged rollout allows a developer to progress from alpha and beta testing to production in a phased manner, constantly collecting feedback and metrics, comparing against baseline (if available) and progressing to 100 percent in production based on satisfactory results. Developers on the iOS platform can avail of beta testing capabilities provided by TestFlight and can invite up to 2,000 beta testers via email to test the App pre-production.
- Direct In-App Feedback
  Gone are the days when email, messenger or app store comments provided the only semblance of feedback. Customers increasingly expect to communicate via the app, and provide feedback without leaving it. Meanwhile, mobile app developers have better to tools to scale that feedback and manage it effectively to address bugs, prioritize features and get context—screen shots, system information, etc.—for specific feedback. HelpStack is an open source framework that makes in-app customer feedback relatively easy, so they can report issues, attach screen shots, embed device information and more. Customer feedback can be reported via email, or help desk integrations are available.
  Apptentive, Crittercism, HelpShift, UserVoice and Parature are some of the notable commercial solutions that enable in-app feedback.
- Metrics & Performance
  When business analytics do not provide provide adequate answers, diving deeper into technical data might provide insights into what is going wrong. Two key mobile customer feedback metrics worth tracking is app uptime and performance. App uptime helps identify customers facing crashes while performance issues could be impacting engagement rates. Some of the tools mentioned in this article do provide an APM solution that enables monitoring of these key metrics.
- Experimentation & Analytics
  A/B testing or multivariate testing is a data-driven experimentation approach to promoting app feature changes. While some of the above tools enabled you as a developer to better listen to your app community, A/B split testing allows you to validate your hypothesis and invest further in features and capabilities that demonstrate lift.
  Apptimize, Optimizely and MixPanel are few of the available commercial products that offer multivariate testing capabilities that can be incorporated in some situations without code modification or app updates. They also provide abilities to compare test campaigns against baseline and promote features that demonstrate positive feedback.
Incorporating Mobile Customer Feedback In Continuous Delivery

Leveraging data-driven tools throughout the continuous delivery pipeline increases the odds of a quality and high-utility mobile app. Even in situations where it is not possible to expose apps prematurely, it is possible to “dog food” your app—a term referencing a team or company that actively utilizes its own product with the intent of testing it. For example, frequent builds tied to developer commits enable alpha build testers and other team members to download and test the app. Meanwhile, beta builds can be downloaded by other members of the company. As release candidates emerge the beta community can be expanded to include members of the app community.

So take steps to incorporate mobile customer feedback loops at every step of the development cycle and garner a majority of the app user’s engagement time. Would love to hear how you and your teams are leveraging tools and techniques to build better apps and continuously listen to customer feedback.
April 8, 2016
Trusting Continuous Delivery
DevOps, continuous delivery and the pipeline for faster software development are fundamentally different from traditional software development. It’s such a different approach and a different culture that it can take some time before Dev and Ops teams embrace and trust it—a necessary step for DevOps to make progress in any organization. By mixing containers and their value proposition into DevOps, this new software creation and deployment paradigm can readily foster that trust.

What Continuous Delivery Brings

DevOps and continuous delivery offer application development using containers, providing the required level of trust to get over that hump and move things forward. According to David N. Blank-Edelman, a DevOps veteran and currently technical evangelist at Apcera, the following container features lead to trust because they answer Dev and Ops concerns:
- Containers are resource-friendly replacements for virtual machines, built to encase the coder’s development environment
- Containers are the complete package format replacement, possessing everything you need to run an app
- Containers create an abstraction that enables development and its results to sit in a host in any of a number of infrastructures including on-premise and inside any of several public cloud providers
- Containers are the fastest means to launch short-lived, purpose-built testing sandboxes as part of a continuous integration process
- Containers are the end result of a build pipeline (artifact-to-workload)
- Containers form the boundary for loosely coupled components as part of a larger app or service. Such is the case, for example, when each part of a microservice-based app lives in its own container
According to Blank-Edelman, the aforementioned features imbue trust because they satisfy the following Dev and Ops queries:
- Is the container allocated to use only the resources it should?
- What resides in that container / workload?
- Where did that container come from? Is it OK from a security, coding and business requirements perspective?
- Where is the container permitted to live and run such as in what cloud, environment, country, and in proximity to other containers technically?
- Can the container communicate with only the things it should such as other containers and any hypervisor or other management resource? “If I move the container to another location, do the answers to these same questions still apply?” asks Blank-Edelman.
Though these questions express concerns in direct and simple language, experienced DevOps engineers who have felt the frustration of building development processes and attendant infrastructure understands these are difficult questions to satisfy. Without modern containerization efforts to sustain DevOps, development and production goals grow further out of reach.

The Importance of Policies

As containers undergird DevOps and continuous delivery, policies support containers. Developers input changes and innovations in code to the continuous delivery pipeline. Developers then respond to the needs of the growing code in a nurturing fashion, fixing errors and enabling new features. This pipeline works when automation, together with each progressive step in the development process, lays a foundation for the next step in a predictable and reliable manner.

“Here’s where it gets tricky: First, not only do you have to be able to answer the trust-related questions for each step, but it is entirely possible (and entirely likely in an enterprise environment) that different people/teams will share control of and input to each step on that path,” says Blank-Edelman. With so much collaborative effort in play juggling all these pieces, policies are necessary to configure these steps, address networking and other elements, and control the component parts of the process ahead of time and everywhere along the way.

Policies that make particularly good container watchdogs include those that set steps in the DevOps/continuous delivery pipeline apart, hemming them in on all sides. “Just like having your papers in order makes for an easier border crossing when you travel abroad, so, too, does having your policies reflect the trust required to move from one step to another,” says Blank-Edelman.
March 28, 2016