devopsy.kinsta.cloud

Tag: quality

What Developers Can Learn from Apple’s iOS Woes

Apple’s decision to delay new features in iOS to address performance and quality issues is a setback for the company, but it also holds valuable lessons for software developers and dev teams about the environment we work in today.

A recent report said Apple plans to pull back on several new iOS features until next year, including a refresh of its interface and improvements to core apps such as mail. It’s an uncharacteristic move for Apple, and follows a wave of criticism over a number of security and quality issues in recent months.

Apple’s missteps come against a backdrop in which companies are under more pressure than ever to get new features to market quickly. It reveals just how intense that pressure has become—and the problems that can result if companies get careless or try to move too fast. Dev teams must learn to navigate a delicate balance between speed and quality.

Here are four takeaways for dev teams from Apple’s decision to delay features in iOS.

The Pressure to Ship Fast Has Never Been Greater

The tech industry is going through a startling period of innovation, with smart assistants, augmented reality, biometrics, wireless charging and host of other technologies all entering the mainstream at the same time. The fact that a company like Apple felt sufficient pressure to push features out before they were ready is a sign of just how intense this pressure is. If Apple was famous for anything, it was quality; the company never released anything until it was rigorously tested and perfected. This has been changing for some time under Tim Cook, but Apple has never felt quite the pressure that it has lately. While not all dev teams are pushing multiple bleeding-edge technologies simultaneously the way Apple is, it’s a reminder of the pace of innovation that we all need to design our pipelines for today.

Prioritizing Speed Over Quality Will Hurt You

While dev teams need to move fast, they can’t afford to sacrifice quality for speed. Every company must make careful trade-offs between the need to move quickly for competitive reasons and the need to get it right. Moving too slowly can allow a competitor in the door, but moving too fast can diminish quality, drive users away and create the kind of backlash Apple is now seeing. Developers need to learn to make this difficult calculus balancing quality and speed. And quality isn’t just about bugs; it’s everything. Ease of use, design, build—these are all elements of quality, and they must all be factored into the decision of how fast you move.

User Experience is Everything

The industry continues to evolve toward better usability, with more energy being invested in creating easy-to-use, well-designed user experiences. In a world where your customers can easily switch their smartphone and the applications running on it, good design in technology is imperative for keeping users delighted and engaged. It’s gone from being a nice-to-have to an essential feature.

Developers Should ‘Shift Left’ for Higher-Quality Code

Traditionally, QA has gotten involved toward the end of the development life cycle. But when you move quickly, code can suffer and you end up shipping features that could do with a bit more work, as Apple apparently has. By “shifting left”—or testing earlier in the process—any issues are easier to fix. Why? Because slower feedback cycles create distance from the problem. Developers become forgetful and shift to other tasks, which means they lose context. Additionally, code can change, which complicates the process of fixing the problem. Testing earlier surfaces errors more quickly, tightening the feedback loop between QA and development.

We don’t have details about what happened inside Apple, but it appears the company is having to slow its release cycle to fix quality issues that weren’t identified the first time around. This is a danger for all companies that are trying to keep pace with the latest technology trends. Developers need to think about all aspects of software quality, ensure their pipeline can accommodate the pace at which they work, and avoid shipping bugs into production that will come back to bite them later on.

— Fred Stevens-Smith

February 13, 2018
Keep Developers Happy and Velocity High

In today’s economy, the companies that prevail are those that can accelerate software delivery and increase developer productivity. Yet, as applications become more complex and codebases grow, companies are forced to implement more comprehensive quality safeguards, which slow the rate at which they can deliver new software. Delivering more incremental builds in shorter intervals can address this conundrum: Customer needs are serviced quicker, overhead for engineers is reduced, and risk is mitigated because changes released into production are smaller.

That’s why high-velocity engineering teams are applying not only continuous delivery processes but also lessons in experimentation from established leaders including Amazon, Netflix and Facebook. These companies have made experimentation a foundation for their release processes, trying out major feature releases and redesigns within smaller groups before making them broadly available. By using new techniques such as feature flagging, rollouts and traffic splitting, experimentation is no longer just the future for marketing teams; it’s quickly becoming an essential practice for high-performing development teams as well.

Today, discussions around velocity focus on key practices that developers can use to maximize the effectiveness of continuous delivery, software development and experimentation programs. Here are some key ideas to consider.

Reducing Your Dependency on a QA Team Actually Makes You Faster (Shifting Left)

Many organizations eschew the conventional wisdom and deprioritize the heavy lifting of writing comprehensive test suites, opting instead to throw them over the fence to a Quality Assurance team. This approach is almost always less effective and causes organizations to lose momentum instead of conserving it.

Higher-performing teams actually “shift left” and empower developers to own many parts of the QA process. A solid foundation of tests creates less overhead later on and empowers engineers to release quality code into production faster. Companies can address customer and product needs efficiently, quickly and with more confidence than if they wait for full QA cycles to finish.

But what if you don’t have the ability (or luxury) to rewrite tens of thousands of tests? The good news is that even without expansive test coverage, you can still take advantage of this approach using code coverage tools. These work by evaluating how much of your codebase gets executed during test runs. If the overall coverage decreases, it means a feature was potentially rushed and is untested and should not be merged into the codebase until the ratio is balanced out. This code-to-test measurement can even “block” code merges unless it meets or exceed the previous threshold.

Prioritize the Build and Test Frameworks Themselves

Teams that that prioritize the development of build-and-test frameworks—as opposed to individual tests—are rewarded with faster, more reliable builds. But it’s a struggle to build out reliable processes that can handle tens of thousands of tests in a performant way for each developer.

Particularly with more complicated test suites, there’s a tendency for these tests to become brittle or prone to occasional failure because of all the moving pieces. Optimizely built a “test gatekeeper” that automatically disables tests that fail in this way, without the need for human intervention, and assigns them to an owner to be fixed. Remember: unreliable tests are sometimes worse than no test at all.

Achieve Faster Release Cycles Through Experimentation

Companies that want additional, real-time insights and fine-tuning for their application stack are shifting to feature flagging and feature rollouts to expand their capabilities.

Feature flags are simple control-flow boolean values that allow developers to use a new, experimental process on their application by gating it behind a virtual on/off switch, and have the option to disable it if a problem is found, all without another code deployment cycle. An example could be a “down for maintenance” notice, or something more advanced like a new product homepage that doesn’t require a code rollout to update the content.

Feature rollouts take this concept a step closer to real experimentation by placing users into two or more experimentation groups, allowing every new feature to be beta-tested before a full general release. Each new feature becomes an opportunity to run an experiment and collect immediate feedback on your development hypothesis.

Finally, there’s traffic splitting, a concept similar to feature rollouts that transparently routes a small number of users to a new experimental version to identify any bugs before they reach a broader audience. The primary difference between feature rollouts and traffic splitting is the “all-in” nature of traffic splitting an entire application vs. a single feature.

Worried that the automated and manual testing of your large product release didn’t catch everything? Send a small percentage of your traffic to the new application as a “canary build” and see what happens. Optimizely experiments with traffic splitting between different production environments (stable versus release candidate, for example) to a small number of users. Facebook, and others have utilized this process for years.

Shift Left, and Enter a High-Velocity Future

Thorough testing helps companies catch many issues before they reach customers, but it’s extraordinarily difficult to anticipate every possible usage or scenario for your product. Investing in these release strategies will help companies mitigate the risks of unintended bugs creeping into code or, worse, unwittingly turning their customers into the QA team.

Our advice? Don’t let feature development take priority over the necessary steps to keep your organization from working lean. Empower your developers to participate in the process early by shifting left, and watch developer productivity and velocity soar.

About the Author / Brian Lucas

Brian Lucas is Senior Software Engineer at Optimizely, responsible for the Build, Test and Release Engineering. Previously, he was CTO at Credible and Hoopla, and lead architect at Sling (Slingbox). He holds four patents. Connect with him on LinkedIn and Twitter.

October 26, 2017
Quality Culture, DevOps Forge Better Connections Between IT, Lines of Business
The automation of software development does not obviate the need for a collaboration between IT and the lines of business, or the development of a culture of quality on the part of the development team. Achieving that quality culture requires more than a change in mindset as developers seek to align quality with the speed and automation goals of DevOps.

There are five items required for a quality-driven DevOps environment:
1. Understanding the proper role of “minimum viable product”
2. Collaboration between IT and lines of business
3. Having metrics in place
4. Across-the-board buy-in
“Quality culture is characterized by employees who are motivated to consistently strive for the highest level of quality,” said Al Riles, principal consultant at Tunnell Consulting, a firm providing solutions to the complex problems faced by life sciences and biopharma companies. “That’s one key component of it, as well as the ongoing process of continuous improvement.”

Minimum Viable Product and Continuous Quality

Aligning quality with the speed and automation goals of DevOps is critical to success, and part of this is gaining a deeper understanding of the role of “minimum viable product” and understanding when that approach contributes to the development process.

“The concept of ‘minimum viable product’ does not, as its name would suggest, contribute to a substandard product,” said C.R. Venkatesh, CEO of Dot Com Infoway, a CMMI Level 3 IT company. “Quite the contrary, the concept is the basis of continuous and iterative improvement, and encourages the use of user feedback, which often results in insights that might be missed by the in-house team.”

The concept of quality culture and MVP are not mutually exclusive in a DevOps environment, so long as continuous improvement is part of the equation.

Collaboration Between IT and Lines of Business

The collaboration that is a key part of DevOps is also a key part of quality culture. “You have a culture in place where everyone embraces that fabric that includes the right type of behaviors and the right values, including communication, reward and recognition, as well as clarity on roles and responsibilities,” said Riles. “That working fabric gets well-defined, because everyone is functioning around the same key elements that constitute that strong corporate culture.”

“Collaboration has always been important in a DevOps environment,” said Venkatesh. “But we need to define precisely where that collaboration exists. In an internal environment, that collaboration is between IT and lines of business. Increasingly though, IT and development is an outsourced function, and in such cases, that collaboration must be preserved. What this means is that it exists not just within the confines of the outsourcing provider, but also between the provider and the client. The developer must seek out alliances not only with the client’s sourcing manager, but also with other internal stakeholders in the project.”

If You Don’t Measure, it Doesn’t Exist

Quality initiatives require an active role in changing the mindset of all stakeholders, but mindset alone won’t get the job done. Metrics must be in place and tied to those quality initiatives. “Even in the smallest of development projects, defined quality metrics must be set ahead of time, with clearly defined outcomes,” said Venkatesh. “By asking what we hope to accomplish with a development project, clearly defining the goals and adhering to a set of predefined quality standards, developers—whether in-house or third-party—are able to accurately predict what the outcome will be.”

Riles’ process involves a quality management maturity grid, which identifies multiple dashboards. “It allows the organization to become familiar with what a quality organization consists of, and then you can use this maturity grid to measure where the organization is once you embrace this initiative. Then, you can implement the initiative, and come back and re-evaluate yourself to see the progress you’ve made,” said Riles.

Everybody Has a Seat at the Table

Incentivizing all stakeholders, throughout every level of the company, is paramount and, according to Riles, “It has to do with communicating what the vision is, and having a clear quality message that’s coming from leadership. Then, there has to be a shared quality mindset among all staff.” Riles notes that the quality mindset is shared by everyone in terms of understanding the organization’s objectives, policies and procedures and their roles in hoping to achieve them.

This is reinforced by leadership at all levels, who are actively engaged in supporting the development of a quality culture, and they engage and motivate others to do the same. For it to be sustainable, we realize that success is based on the sum of all parts of the organization. They must be continuously monitored and fine-tuned as the organization grows, and this includes communication, reward and recognition, and engagement.

About the Author / Dan Blacharski

Dan Blacharski is a thought leader, advisor, industry observer and PR counsel to several Internet startups, and author of the book “Born in the Cloud Marketing: Transformative Strategies for the Next Generation of Cloud-Based Businesses.” He has been widely published on diverse subjects relating to cloud computing and technology, and he is editor of BoozeHub.com. He lives in South Bend, Indiana with his wife Charoenkwan and their Boston Terrier, Ling Ba.
September 15, 2017
In DevOps, Quality is a Team Effort

Technology is changing every industry; we can all agree on this foundational premise. If you disagree, I believe the rest of this post is going to do nothing for you, so feel free to stop reading now. If you do agree, then you know that apps, the internet of things (IoT) and APIs are table stakes now. Businesses need strategies for each, and the forward-thinking executives are already adapting. The ones that aren’t risk alienating a channel in their overall digital strategy. It’s not all about strategies for building new types of functionality and partnerships; organizations also need to validate they will work specifically with their customers’ use cases in mind.

When we study software development, the history books teach us that waterfall strategies followed a linear path. In the past, this pace may have worked, but the future seems bleak with the evolution of technology and its disruption of traditional, manual business processes advancing at an exponential rate. Over time, the stages in the software development life cycle (SDLC) may remain the same, but organizations need to focus specifically on becoming more efficient at completing more work in less time. With this “do more with less” mindset, it’s no wonder many organizations see a backlog of work piling up. This point is especially true for quality assurance and quality engineering teams.

First, let’s unpack the software development stages from a traditional sense. Historically, organizations would get an idea and that idea could come from the businesses stakeholders (sales, marketing, customers, etc). The business would prioritize/stack rank all of their ideas by which carried the most merit and how much work those ideas would require for implementation. This is known as the “planning” phase. From planning, the idea would progress to be built. “Building” is the part of the life cycle where your artists/engineers live. You know, the creative people, the developers. They build the technology according to specifications that come from the planning phase. Most importantly, they are incentivized based on the project being delivered on time.

After the technology is built and packaged, it is passed on to the team that has been traditionally responsible for quality—quality assurance (QA). QA has their time with the product to validate the product works as intended, can scale to the appropriate demand levels, is secure and meets all the use cases without error before they pass it over to the team that will deploy it to the hands of their customers. Finally, the business is measuring feedback and customer sentiment, and then rolling that feedback back to the plan stage for improvement.

As time has passed, we have also seen another major trend develop. Organizations have developed agile/iterative SDLC strategies that are focused around bringing smaller chunks of functionality to their users/customers faster. Just like “lean manufacturing,” this concept has been focused predominantly around helping the developers get their projects completed faster. The primary challenge with waterfall development strategies is that by the time an organization brings the plan to market, the needs and priorities of the customer will have already changed. The speed of idea generation outpaces the execution of said ideas. That being written, while development has kept pace, quality has not, unfortunately.

Now I pose a simple, albeit, extremely important question: Who owns responsibility for ensuring quality of the finished product? If your answer is QA (like stated above), and I can see why that would be the instinctual response—the process is set up for failure. Here’s the fundamental reason why: if quality is so important to the process, then why is it the third phase in the SDLC? This implies planning and building without quality in mind is an acceptable process.

Quality should be owned by everybody in the process. QA/QE teams are your enablers of embedding quality throughout the SDLC. The distinction is more about instituting an ideology or culture based on quality.

With a culture focused on driving quality, when the planning team works on creating a specification for the idea, they should work with development and QA/QE as they are designing the flows, behaviors of the application. This allows for QA/QE teams to understand what to validate much earlier in the process. Teams are always looking for ways to factor more automation into the process. QA teams are also looking for ways to make their test environments more closely resemble the production environment where their customers interact with the business.

Unfortunately for QA/QE, they feel pressure from the business to execute within a constrained time frame with better results. This typically means inadequate windows of opportunity to start building automated tests or an incomplete environment that doesn’t quite resemble what the end user will face.

The last frontier to validate is performance which answers the important question: Will the app scale for the expected user population? This is often the last step before flipping the switch and shipping the product to your customers. But what happens if the application doesn’t scale? Organizations need to drive performance validation earlier. Better yet, they should engineer the application for the desired performance, and that means—yes, you guessed it—taking care of this earlier in the life cycle.

What if it were possible to drive more improvement?:
1. … To start building functional test plans and test automation at the time when planning happens?
2. … To ensure that development and test environments more accurately have access to the constrained downstream resources that cause unexpected “wait time”?
3. … To ensure that the automated testing that will be executed in the testing phase, has the right data to power the tests?
4. Finally, … to make sure that performance isn’t just an afterthought but a strategy to scale to the changing landscape of a digital business?

It’s more than possible. It’s obtainable. This process is known as continuous testing. It’s about the validation that needs to occur earlier—not just within a window of allotted time within an overall life cycle. It’s also important to understand that it’s a maturity curve, and it will take work to evolve your individual process.

Can your organization embrace continuous testing strategies? If not, what’s stopping you? If so, where can we begin to improve? Driving quality isn’t just about your company’s KPIs—it’s about your customers’ perception of your brand and what you do for them. You never get a second chance to make a first impression. Let’s build the right impression today!

— BobbyMelwani

August 15, 2017
The Role of QA in the DevOps World

The need for faster time to market and superior quality is getting intense in the software development process. Software trends for 2017 suggest that DevOps will grow as a trend and become more widely accepted across enterprises. Organizations are reaping the benefits of DevOps approach and are incorporating the rules within every aspect of software testing. A case in point: DevSecOps—security testing integrated with DevOps—is an emerging concept and is being considered by enterprises to execute application security testing.

The underlying objectives of adopting DevOps are to make the testing and development process more collaborative, ensure software quality and to help launch the product faster to the market. This makes quality assurance (QA) a key component in the DevOps methodology. New-age applications need real-time changes and updates to meet the growing demands of the consumer. This requires continuous testing, continuous development and continuous delivery/deployment. DevOps enables teams to constantly upgrade and deliver.

In 2016 a number of enterprises adopted DevOps. According to industry experts, 2017 will see more defining moments for DevOps. The idea of accepting DevOps is to consider a cultural shift and not just a technical shift. As digital transformation expert Jeff Sussna says about DevOps, “The key to following the continuous delivery path is to continually question your own assumptions about what’s possible.” Currently, DevOps is on the route of being questioned and defined at every level. It will result in bringing more value to the methodology.

Every enterprise and its style of working differs. The way they would adopt DevOps will differ, too. But digital transformation and the need to be available for customers all the time with new updates is transforming the way enterprises adopt new practices and methodologies. Every enterprise has to evolve in a sustainable manner, so quality is at the core of every practice, including DevOps.

Why is QA Integral to DevOps?

In the DevOps scenario, QA integrates into the testing and development process and takes the collaborative approach. Quality is ensured throughout the testing and delivery cycle and both the testing and development teams are responsible for it. In the traditional pattern, quality creeps in toward the end, but with DevOps it comes in at every level.

In terms of code development, the QA team pushes the new development code into production and DevOps facilitates to ensure a seamless deployment of infrastructure and tools. Thus, it helps in releasing a code on a consistent basis and the responsibility to ensure quality is not just limited to killing the bug; it is built in the overall application release.

In the current scenario, hourly releases are required to meet the demands of the consumers and deal with competition. Web/mobile applications such as an ecommerce or financial services application need to be updated frequently. It is a key business requirement. DevOps helps to make this possible, but with QA involved, the quality of the release is ensured. Moreover, with DevOps, the reality check can happen in real time rather than taking the code offline and testing it.

On the whole, the popularity of DevOps over the years has intertwined quality into every engineer’s responsibility, which is a positive sign. This helps in addressing the needs of the market and ensure that customers do not suffer. QA further enables and directs teams to check the application for performance and quality before it is delivered.

The role of QA in the current software development process is changing, where the customer comes into focus and the needs of the customer are brought into perspective while delivering the application. Releasing new and functional builds at any point through the application’s life cycle can become a disaster if quality is not ensured constantly. Automation of tests plays a critical role to guarantee quality and bring speed.

Building a QA strategy with the necessary best practices is essential for organizations that are embracing the DevOps movement. This will help in delivering effective software development and operations to achieve user experience. DevOps practices are adopted to improve the regularity of the releases and reduce the defects. It is a methodology that boosts rapid, short, iterative development and deployment of applications.

So, the idea of incorporating QA is to ensure continuous improvement in the delivery process. DevOps helps digital enterprises to meet the needs of the competition and enable overall business growth. As a result, enterprises are able to streamline and pace up the delivery of applications with changes and innovation.

Best Practices for Incorporating QA in DevOps

The key aspect to consider while merging QA with DevOps is that the QA and testing teams must team up with the technical teams. In this way QA becomes an enabler for ensuring quality in the software development cycle. In addition, it is important to define tangible aspects of the quality that are expected from the project/application. This helps in achieving results within stipulated time frames.

Putting down the quality metrics is absolutely critical in DevOps, which helps teams to identify bugs early in the development cycle. Defining the metrics also enables teams to integrate them within the organizational culture at various levels. In this way, quality is ensured at every level and development teams also get enabled.

The idea of DevOps is to make the developer think in line with testers and constantly check the code for potential errors. On the other hand, a tester who holds the responsibility to test the application/software with various testing tools and methodologies also needs to fix the code. In this way, enterprises with an inadequate QA or testing strategy cannot ensure continuous testing and continuous delivery.

In Conclusion

The application scenario is changing and it is getting very complex. DevOps can speed up the development, but what can you expect without a robust QA strategy? QA is absolutely integral to the DevOps process. Continuous development and delivery is impossible without a comprehensive QA/testing strategy.

This blog has been drafted in collaboration with Cigniti Technologies, an independent software testing company. Click here to read more blogs on DevOps.

About the Author / Komal Lopez

Komal Lopez works with Cigniti Technologies and is instrumental in helping enterprises take better decisions related to Quality Assurance products, tools and services by leveraging research and content. She specializes in writing about technology trends, testing trends and has been in the Marketing and Communications industry for over a decade. Connect with her at komal.lopez@cigniti.com

July 28, 2017
APIs: Automation is Great. So is Usability

Many developers are not so great at documentation. Ask anyone who’s been thrown into a large code base. Changes aren’t documented well, tribal knowledge thrives in the code comments, and anyone who says, “I’m confused, what does this chunk of spaghetti code mean?” is often treated with annoyance or even disdain.

That is something that DevOps doesn’t change. And there are, as with so many things in IT right now, complicating factors. First is APIs. In almost every shop, developers are responsible for the public-facing APIs. The same developers who don’t comment well are now documenting public-facing APIs. The other complicating factor is DevOps documentation mechanisms. Automated systems generate API documentation from developer comments. Again, the same developers that often don’t document their code well enough to efficiently on-board new developers.

Those of us who have used more than a couple of APIs know the problem intimately. Dozens, or hundreds, or thousands of us have to search for what the allowable values for a parameter or JSON field are in an API call are because far too often that is not well-documented.

The whole system grew naturally. APIs are the realm of developers; documentation was painful to get created because it required tech writers that were advanced enough to understand the purpose and function of the API; and development of documentation delayed API deployment—all are issues that lent themselves to automated generation and the developer in the code was best-situated to document it. In the API economy, and in light of both agile and DevOps, it just made sense to use generated documentation for this part of the project.

But we’ve got enough experience now to know that “good enough” isn’t. Far too often users are left to figure out the details, and quite often, those users don’t bother. That’s not a benefit in the API economy; it’s a risk.

Get Help With Your APIs

And it’s pretty easily resolvable. Get someone not knee deep in the code to try and use your API as documented. Get their recommendations on usability, and fix the issues they find. I’ve used APIs that have a dozen JSON base parameters, none of which are documented as to what’s allowed or not. I’ve used APIs where the JSON field was text, but the values were from a specialized list that was not made public and users had to hunt for allowed values. We’ve all been through it, and it is a massive waste of time.

If you have a customer base, use Alpha/Beta users to help with this process. Early access is generally people who know the project/product, so feedback will be more focused, but as outsiders they do not share in tribal knowledge. Without tribal knowledge, they will have solid input to improve API usability.

And in the end, an API is a consumable that you want customers to use. Telling them how to do so is important. And the current trendy way of doing so is not good enough. So make certain your process is good enough, or accept user loss as a consequence.

For those with established products/projects, do not wag your head and assume you have it covered. Some very cool, long-lived APIs have entire sets of JSON input values that are not well-documented. They were important enough to make them parameters (or return values for that matter), yet you didn’t document them well? Please, ask new users if they agree with your assessment of your API.

Don’t have time/man hours/whatever to address documentation shortcomings in your API? Then don’t publish an API. The only thing worse than saying, “You can’t integrate via API with our system,” is saying, “You can integrate with our system via API,” then not documenting it well enough to do so.

So keep the automation. But follow the new mantra:

Now that you have generated documentation, check that you have usable documentation.

And keep kicking it. APIs are the hard part, telling people how to use them is comparatively easy, so take the time to make your development efforts worthwhile.

— Don Macvittie

June 16, 2017
Release Automation: Delivering the Big Quality Payoff

Have you ever gone down the path of testing the latest release of an application only to find out that you had the wrong version?

Maybe you accidentally grabbed the wrong build or you didn’t have the correct configuration file for the environment you were testing against—or your database wasn’t updated correctly, or the script you were using was still pointing to a prior version of the software.

Whatever the case, we can agree that keeping track of all the various artifacts, database schema changes, configuration parameters and other components needed to accurately test and deploy an application is both time-consuming and prone to errors.

Today, many companies are looking to DevOps, continuous delivery and agile development methodologies to shorten their application delivery cycles and reduce the number of errors and the costs associated with application deployment processes. Nearly all enterprise companies are software companies, and their mere existence depends on delivering differentiating products and services via software to their customers.

It’s imperative to be able to react to customer demands and competitive pressures quickly, if not immediately, to stay ahead of the competition. Gone are the days of yearly release cycles.

Today we see companies deploying new capabilities into production environments on a monthly, weekly or even daily basis. At some companies, hundreds, if not thousands, of deployments are occurring across dev, test and production environments daily. The old methodologies we once used simply can’t support this type of volume and cadence.

Employing Release Automation

So, how are these companies transitioning to this new way of thinking? In a word—automation; specifically, by automating early and often.

Everything about your release process is a candidate for automation. This includes environment provisioning and configuration automation, build automation, test automation and release and deployment automation.

A customer of ours recently implemented CA Release Automation, and within eight weeks had automated 90 percent of its deployments. Deployments that once took more than five hours and a team of people now take a matter of minutes.

“This not only saves us a huge amount of time, but it also mitigates the risk of human error being introduced by manual processes,” said Tim Dallmann, platform engineer at Jewelers Mutual.

Another customer, a major airline, implemented CA Release Automation and immediately reduced their production outages from an average of 237 over a three-month period, down to 15, seven, six and five over the following four months.

And Molina Medicaid Solutions reduced their cost of human error by 90 percent, all the while responding quicker to emergency deployment requests and improving deployment quality.

Automation is critical, but I believe these results prove that implementing a framework to manage and deploy release processes in addition to automation works even better.

Increased Flexibility and Visibility

Continuous delivery helps break down barriers often seen between development and operations teams. It provides, even forces, a more collaborative approach and automation is a key enabler of this.

“One of the best things about automation is that it gives us the flexibility to skill-up end users—deployments in our test environments are no longer solely the responsibility of the operations or IT teams,” explained Dallmann.

With an application release automation (ARA) solution, artifact, build, configuration and release processes are no longer tracked with spreadsheets or embedded in complicated scripts.

Increased visibility into these processes ultimately enables all stakeholders to take a more active role in the release process, and access to real-time information can significantly reduce the amount of time and resources needed to resolve and recover from problems when they do occur.

Automation leads to:

Integrating with the Ecosystem

An enterprise ARA solution can help you create a single control point to streamline all your application delivery processes for better predictability and reduced risk. A critical component of this is the ability to integrate and orchestrate tools and technologies required along the continuous delivery pipeline, including source control, automated testing tools, provisioning and configuration management solutions and change management and performance management solutions.

Integration with an application lifecycle management solution can provide even further alignment, governance and traceability between development projects and what is ultimately being tested and promoted to production.

“By integrating the two solutions, we will have a complete audit trail of deployments to help us understand how, where and by whom stories have changed,” said Phil Nickolai, director of application services at Jewelers Mutual.

“What we really needed was a solution to streamline our whole way of working,” he said. “We had numerous people working on projects, competing for resources and causing complexity. Now we’re back in control and getting smarter every day.”

The Time Is Now

Let’s get rid of the problem that exists between the keyboard and information kept in spreadsheets, scripts or emails. Not only will you stop wasting time testing the wrong version of an application, but you can also ensure you have the right test data, the appropriate test cases and the correct configuration parameters for your environment.

Stress-inducing release weekends and nights become a thing of the past. Developers can spend more time on developing innovative capabilities that generate revenue and keep customers coming back for more.

— Tim Mueting

March 7, 2017
Automation Killed QA – It’s Time to Deal With It

Quality assurance (QA) had a good run.

From the time it was considered an afterthought by organizations and software vendors alike to the era of massive outsourcing contracts, followed by significant vendor investments in related tooling, and, finally, its recognition as a critical success factor among high-performing organizations, quality has evolved to become even more important than functionality to end users.

Customer experience, today, is defined by speed and quality above all else. Namely, can I access the information and products that I want in the blink of an eye and get expected feedback based on input? As a result, customer experience has also shifted: It’s all about application experience, not the friendliness of a checkout cashier or sales/service rep.

With this massive shift toward customer experience platforms (think omni-channel) and increasingly lofty expectations, the operative question becomes, How are we enabling velocity in our testing practices to ensure the highest quality while keeping up?

We’ve tried everything from outsourcing to insourcing, investing in tooling, investing in people and even building testing architecture teams to oversee the automation, quality and consistency of testing results. Roughly seven years ago, my top priority for testing teams was to automate 30 percent of our tests. Today, if you are not 100 percent automated, or close to it, you are severely lagging in output.

But what’s next? What if I’ve automated all of my tests and my manual testers have been replaced with automation engineers, but I’m still having quality issues and being perceived as a bottleneck?

The answer is simple. Automate everything.

Makes sense, right? Isn’t everyone telling you that? But what are we defining as “everything” from a testing perspective? Therein lies the problem; we are thinking about testing as its own factory within the larger software development life cycle.

While we believe we are changing this perception by going agile and moving most QA functions into scrum teams, the reality is we still look at testing as a standalone function. Centralized or decentralized, this is the case, and we need to think differently.

Automating for quality begins with defining requirements. How am I ensuring that I’m covering all cases? How do I ensure I’m translating the business requirements properly? These questions, the time it takes to answer them and the risk of human miscalculation all impact quality and velocity.

We must start our automation for quality at the beginning, during establishment of business requirements.

Let me introduce you to next-generation QA. Intelligent “continuous testing” platforms are replacing humans for testing, automating tests, creating data and building test cases. These platforms are able to translate model-based business requirements into complete automation—test cases, test data and test script automation.

Orchestration across your software development life cycle creates your environments, lays down your test data, deploys your software, runs your test scripts and reports results. No more human error, no costly resources and guaranteed test coverage that ensures quality.

Integrating your existing automation around continuous integration, portfolio management reporting, and execution reporting to this framework now gives you the ability to report and automate your ideas all the way through to outcomes.

Early adopters are already doing this. QA as a functional organization is gone. If you aren’t already automating to this end, you are severely behind.

Questions? Comments? Perspective? Have at it in the comments below.

— Ron DeCanio

January 27, 2017
Don’t Leave Software Testers Out of DevOps

The term DevOps implies that software delivery is all about integrating the development and IT operations teams. But the software testing team is an important, if oft-overlooked, part of the picture. Here’s why.

Could you deliver software continuously without involving the QA team in the continuous delivery pipeline? You could try, but it would probably not go well. Software testers play a vital role in continuous delivery, for the following reasons:

They are the glue connecting developers to IT Ops. If there is no one to test the code that developers write, IT Ops cannot deliver a stable product.

They help find problems before they become costly. If you skimp on integration, unit and performance tests, you won’t find problems in your software until after it reaches users. By that point, fixing those problems will be much more time-intensive than it would if they had been caught early on. Plus, your organization’s reputation will have taken a hit, because problems that could have been nipped in the bud prior to delivery ended up affecting users.

They know things that developers and IT Ops don’t. If DevOps is all about continuous visibility across the delivery chain, then it only works well when the delivery team’s collective expertise covers all parts of the delivery process. Since developers and IT Ops people are not experts in quality assurance or testing, you need to involve testing professionals in the continuous delivery chain so that they can help everyone else understand the parts of the process that QA professionals know in detail.

All of the above is to say that if the QA team is not plugged into every stage of the delivery process, your software either won’t be tested efficiently, or it won’t be tested at all. Not testing efficiently means that the QA team is not in sync with the rest of the organization. In this case, the QA team would delay software delivery because it would have to wait on developers and admins to complete its work, and vice versa.

Not testing at all, of course, is even worse. No one should deliver unvetted software to users.

Don’t Forget about QA

The takeaway for organizations seeking to adopt DevOps practices is this: Even though the DevOps conversation on the surface may seem to be about only developers and IT Ops professionals, QA teams play an equally crucial role in continuous delivery. They need to be as involved in the continuous delivery pipeline as everyone else if an organization wishes to reap the full benefits of DevOps.

— Chris Tozzi

November 8, 2016
DevOps: Commoditizing Infrastructure

Hardware vendors face a quandary. How do you distinguish yourself as superior from your competitors if you are unable to continue to grow a distinct architecture as your competitive advantage? But this dilemma does not begin and end with only hardware vendors; operating system vendors and virtual machine vendors are on the same horizon. In an effort to “uncomplicate” hardware idiosyncrasies, our industry evolves efforts from open standards to least-common-denominator functionality for a “common” language or set of features, and over time technologies such as containers emerge. If I can abstract my application from the “uniqueness” of hardware vendors, from the complexity of configuration settings, and make only assumptions about basics such as CPU, disk, memory, etc., then I am free to focus on the application and allow my “hardware mitigation technology” to perform its function.

The Relationship Between App and Storage

But to accomplish this feat, I must agree to a basic set of features or functions I intend to call, such as making “x” amount of machine memory available to my application. Whether disk or RAM, I am looking to define what my application needs and then expect the container (or hardware mitigation/abstraction manager of my choice) to manage the actual allocation. Then the quandary begins. Asking for 1TB of disk space as an example is generic enough. But does my request include the ability to specify I am looking only for disks that spin at more than 7,500 revolutions per minute, or will I be saddled with 5,000 rpm disks? There is a clear implication in speed for my app.

Ultimately, as I sit and tune my code because my app has become “slow,” it could be something as innocuous as having its actual execution occurring in a cloud provider who uses the slower disk revolutions as opposed to the faster ones. Worse, it simply could be that the same cloud provider has a batch of the slower ones in one of its data centers where I wind up executing (only on a temporary basis), and I am never fully able to figure out why my app got “slow” for a while. That kind of random bad ju-ju requires the execution of a chicken to fix (I prefer Popeye’s, myself).

Different storage vendors might have additional peculiarities that make one or the other superior in speed, reliability or performance over their respective competitors. But the container’s main functionality is to provide “disk” to my app. Specifications of specific disk vendors or providers are the problem of the cloud provider, not the app developer, in theory. But hardware refresh can be pricey, and to make a claim that “all our disks” have rpm specifications above 10,000 will be meaningful only for about 10 minutes when the 12,000 rpm models come out. And of course, there are many more architectural designs in the engineering that are intended to boost speed and reliability. If these are not to be ignored, a decision must be made as to “who” has the responsibility to manage their usage (usually the cloud provider), and “who” will complain when things get slow (usually you the customer).

And what happens when “breakthroughs” occur in a specific technology? Solid-state hard drives, for example, are not exactly new now, but they once were. When you move your app from operating on a high-speed hard drive that spins to a chip-based hard drive the operates at the speed of the RAM it is based on, the difference in speed is an order of magnitude better. The inevitable question emerges, Will my container have the ability to specify RAM-based disks instead of traditional spinning disks, as part of the disk commodity it is providing to my app?

Since the difference in speed is so great, you would think it would come up with a way. You would equally expect that providing offline storage such as tape (from the age of the dinosaur) would have its own designation in my container structure (not be equated with offline spinning drives due to the speed differential). When you go to make a backup or restore, the difference between tape and hard drive becomes as apparent as the difference between solid-state and spinning drives on app execution. So when a platform evolves, my container mitigation must evolve with it. But to offer the uniqueness of vendor superiority requires my programmers to learn who to request (or what level of hardware specs they must know), which is the whole point of wanting a commodity in the first place: I don’t have to know.

Impact of Commodity on Other Components

This same dilemma can be exhibited in CPU vendors. Intel touts its superiority, AMD would disagree, and there may always be an upstart who wants to rival the current leader in any space. Lest we think Intel-based computing is the undisputed champion, quantum computing may upset that assumption (unless, of course, Intel corners that market as well). The point here is simply to ask: how many features, how many specifications, how much do we the programming team need to know about our platform provider to understand why and where our application runs better and worse? In the days before container-style ideology, hardware vendors touted their differences and their uniqueness as the reason they were better. Now what?

Then comes our virtual machine providers, and eventually our operating systems. If I want to truly abstract from my hardware layer, I must not call an OS-unique command that only Linux or only Windows can do. I must stick to functionality that both OS instances can reasonably understand and accommodate. While this least common denominator is great for extending where my app can run (on multiple types of hardware, in multiple types of OS installations), it has the same effect on uniqueness and potentially on competitive advantage. If the intent is to come up with a one-world OS, I am generally in favor, but our journey there is sure to have certain casualties in innovation that a single OS is suited for and its competitors are not.

The inevitable question becomes, How far down, or how far up, do we want to form commodity provisioning of computing capabilities? One could argue that the simplicity of the iPhone platform is a user-centric expression of commodity provisioning at the top of the stack. Drag-and-drop programming languages are yet another expression of bucketing computing capabilities into more simplistic expressions (that more people can understand and, therefore, utilize). Commonality and simplicity do increase audience. In many segments of our industry, this is the nirvana of sales to the public.

Impacts to the Bottom Line

DevOps focuses on enabling innovation. At its core DevOps is focused on software innovation. Let’s face it—changes to hardware platforms are generally less frequent. If those hardware platforms are married to a particular app (not serving a large number of apps on the same box, same virtual instance, etc.) then at least provisioning can be tuned one to one with the app. But with the desire to turn infrastructure into a generic commodity that application programmers don’t have to think about comes the fact that they stop thinking about them. Over time they will understand less and less about why things become slow, or work better, as the knowledge of uniqueness goes away.

Depending on your market focus, ubiquity is either your friend (as it touches consumers) or your enemy (if you are the hardware vendor attempting to maintain relevance over your competitors in an increasingly commoditized market). The questions I raise in this article are designed to focus on how to keep our trend toward hardware mitigation from stifling innovation in the very hardware platforms we depend upon. In any case, I welcome your thoughts on the topic.

To continue the conversation, feel free to contact me.

August 10, 2016