devopsy.kinsta.cloud

Tag: development

SRE vs. DevOps — a False Distinction?

Just a few days before he died at the beginning of the 1990s, a wise man taught us that “the show must go on.” Freddie Mercury’s parting words have long provided the guiding light for many, if not all, ops teams. In their eyes, the production environment should be exposed to minimum risk, even at the expense of new features and problem resolution.

About 10 years ago, Google decided to change its approach to production management. It took the company only a few years to realize that while R&D focused on creating new features and pushing them to production, the Operations group was trying to keep production as stable as possible—the two teams were pulling in opposite directions. This tension arose due to the groups’ different backgrounds, skill sets, incentives and metrics by which they were measured.

Trying to bridge this gap between the two groups, one of Google’s ops leaders, Ben Treynor, thought of an innovative solution. Instead of having an Ops team built solely from system administrators, software engineers—with an R&D background and mentality—could enrich the way the team worked with the development group, change its goals and help with automating solutions.

Site Reliability Engineering, or SRE

And so, the Site Reliability Engineering position was created. According to Google, SRE engineers are responsible for the stability of the production environment, but at the same time are committed to new features and operational improvement. Google decided its SRE teams should be composed of 50 percent software engineers and 50 percent system administrators. The engineers were driven to use software as a way of solving problems and perfecting what had historically been solved by hand. They integrated easily with the development team, and encouraged code quality improvements and automation testing.

Understanding the value of SRE, several organizations of various sizes decided to embrace its principles. Some, such as Dropbox, Netflix and Github, are well-known for being at the forefront of technology leadership.

Wait, Isn’t That DevOps?

DevOps is a more recent movement, designed to help organizations’ IT department move in agile and performant ways. It builds a healthy working relationship between the Operations staff and Dev team, allowing each to see how their work influences and affects the other. By combining knowledge and effort, DevOps should produce a more robust, reliable, agile product.

Both SRE and DevOps are methodologies addressing organizations’ needs for production operation management. But the differences between the two doctrines are quite significant: While DevOps raise problems and dispatch them to Dev to solve, the SRE approach is to find problems and solve some of them themselves. While DevOps teams would usually choose the more conservative approach, leaving the production environment untouched unless absolutely necessary, SREs are more confident in their ability to maintain a stable production environment and push for rapid changes and software updates. Not unlike the DevOps team, SREs also thrive on a stable production environment, but one of the SRE team’s goals is to improve performance and operational efficiency.

Google tried a few approaches to implementing the SRE process before finding the one that suited them best. One of these approaches attempted to tie the number of permitted releases to the product’s stability. The principle underlying this process is that new releases are green-lighted based on current product performance.

For each service, the SRE team sets a service-level agreement (SLA) that defines how reliable the system needs to be to end users. If the team agrees on a 99.9 percent SLA, that gives them an error budget of 0.1 percent. An error budget is exactly what the name suggests: the maximum allowable threshold for errors and outages. Here’s the interesting thing: the development team can “spend” this error budget in any way they like. If the product is currently running flawlessly, with few or no errors, they can launch whatever they want, whenever they want. Conversely, if they have met or exceeded the error budget, and are operating at or below the defined SLA, all new releases are frozen until they reduce the number of errors to a level that allows the launch to proceed. This process ensures that both the SREs and developers have a strong incentive to minimize the number of errors in production.

Another interesting approach recommended by Treynor, which is more related to SRE professionalism and efficiency, is allowing SREs to move between projects. Moreover, he suggests allowing SRE engineers to move to development, and even the other way around. As the work done by both teams is similar, these transitions help the Ops team gain better and deeper knowledge of the product and code, and bring the Dev teams into the production space to help them understand its challenges. This strongly promotes a team atmosphere, rather than one in which an individual feels that “I’m on the SRE team for this product.” As part of this approach, Treynor had Dev teams handle 5 percent of the operations workload. This, according to many organizations, adds to the SRE team’s motivation and effectiveness.

Riddles in the Dark

Well, SRE seems perfect. As already stated, several large-scale organizations have chosen to move some of their production operations from old Ops to SRE. However, there are still a few questions that need to be asked:

Is the position of SysAdmin (production/ops) no longer relevant for SysAdmins?

Historically, almost all system administrators have come into their roles through tech support and similar work, or even just running Linux on their desktops and then transitioning into server work. It should be pretty clear that the same path is not available into SRE. To retain their positions, SysAdmins should now be more code-oriented, have better technological knowledge and be receptive to new methods of conducting the work they already do.

Can an SRE team prevent production incidents?

One of the strengths of a professional site reliability engineer is the ability to handle the growth of production load and traffic in-house. Monitoring and analyzing processes and logs with platforms such as the ELK Stack is part of the day-to-day workload, and the team should be able to identify problems as they occur, and even foresee risks to software stability. The power of this position—based on the skill set it requires—lies in developing solutions for these problems and risks. As Ciara, a software engineer in Google’s cloud storage SRE team, described it in a “Life in Google” post, “We solve cooler problems in cooler ways.”

Are we doing DevOps or are we doing SRE?

This question is usually asked by people trying to position themselves in the Operations world. According to many companies that implemented SRE in a slightly different way than Google, you don’t have to decide. At Reddit, ops engineers work on reducing toil, improving deployment and scaling processes, but they are referred to as “DevOps.”

At Logz.io, we bridge the gap between developers and production through the use of automated monitoring and performance stress-testing. And we personally refer to it as “DevOps” rather than “SRE.”

So, is it Just a Question of Context?

In her blog, Charity Majors talks about SRE and DevOps being two different operational approaches that any organization can choose to work by, but insists on emphasizing that there is no “correct” approach. Although Google and Dropbox have decided on SRE, this does not mean the rest of the world should do so as well. What fits Google’s needs and organizational philosophy does not necessarily work for other orgs—at any scale. Moreover, Charity believes that DevOps, having grown and evolved within a broader variety of software organizations, is the more flexible, collaborative and adaptable approach, and will work better for most software organizations at all stages of development.

On the other hand, SoundCloud senior engineer, Matthias Rampke issued a series of tweets on how SRE and DevOps were basically the same, with only one difference—high management support. Though in contradiction to her blog, Charity lent weight to this opinion during an AskMeAnything event that revolved around DevOps and SRE. She shared a story about a friend who was hiring for a startup, which, as an experiment, posted the exact same job description twice, the only difference being that one listing was titled “DevOps engineer,” and the other “SRE.” At the time of relating the story, DevOps was winning by 10 percent or 20 percent. However, all this means is that the job title is irrelevant, and naming an SRE team does not magically bestow qualities upon it. Rather than focusing on the title, organizations must focus on the work being done.

To summarize, we can quote Matt Simmons, a technologist with many insights into this topic, who says, “Not every infrastructure needs an SRE, but every infrastructure could use an administrator who acted more like one.”

— Asaf Yigal

May 18, 2017
DevOps is a Lifeline to Connected Mainframe ROI
A new IDC white paper sponsored by IBM and CA Technologies reveals the adoption of DevOps is a hallmark of those organizations that are actively modernizing and integrating legacy systems such as a mainframe to support digital transformation.

Early adopters are blazing a path for faster innovation and greater ROI.

DevOps Delivers Mainframe Modernization and Integration

I’m encouraged to see DevOps recognized as an important element in realizing a “connected mainframe” vision. Respondents demonstrated a continued commitment to modernizing their mainframe environment.

The IDC study confirmed that leveraging Java on the mainframe was one of the most popular approaches. In fact, I’m constantly surprised at how few managers realize that these days, Java and Git can be used by developers in such a way that allows them to completely separate themselves from the mainframe “green screen” environment when coding or maintaining applications.

DevOps automation has also helped organizations overcome many of the cultural and operational challenges to a fully end-to-end enterprise by aligning application development tools and orchestration chains across multiple platforms and teams.

Adopt a Business-First Approach

Customers I’ve spoken with are experimenting with DevOps for mainframe tools and practices, with many piloting new approaches across the life cycle. They’re immediately becoming more agile, realizing shortened development cycles, improved collaboration between distributed and mainframe teams and faster MTTR that maximizes availability. In all, these early adopters have benefited from increased speed to market, improved business agility and better platform economics.

The IDC study concurs, identifying more effective delivery of applications and services as a powerful cost-saver, with savings of $58,900 per application achieved through increased developer productivity and day-to-day support efficiencies. By helping mainframe organizations to realize improvements at every stage of the application life cycle—from build through test, deploy and operate—DevOps delivers compelling benefits; for example, from our recent study Accelerating Velocity and Customer Value with Agile and DevOps:
- Advanced DevOps users reduce the time taken to develop, test and release new applications by 42 percent.
- Advanced agile users reduce their time to decide and act on new opportunities by 40 percent.
- Advanced agile users accelerate the time to develop, test and release new apps by 36 percent.¹
Excited? I encourage you to catch up on the full IDC White Paper here.

— Aline Gerew
May 3, 2017
CD 101 Podcast: The What, How and Future of Continuous Delivery
Now that we’ve arrived at Episode 4 of our CD 101 Podcast, let’s address some basic concepts—like the what and how of it all and the future.

Wait … perhaps we could—or should—have touched on these key topics earlier. Or not. At least not the future part? Anyway …

While this (mostly) backwards-looking orientation may seem a bit odd, I’d argue that we’re actually navigating the conversation on precisely the correct path. Let’s rewind for a moment to look at where we’ve come from.

Episode 1: We highlighted many of the fundamentals of this thing called CD, yet we did so from more of a conceptual and best-practices standpoint. While the ideas and processes involved do resonate in the real world, the topics we cover in this episode are bit closer to the water-cooler conversations we’re hearing about CD as it grows in practical adoption.

In Episode 2: We covered the whole how do you “realize agility” piece of the conversation. This is important and a precursor to some of these other basic questions. The entire process of digital transformation begins with getting agile and you can’t truly adopt continuous delivery without doing so; therefore, this was another good place to feed today’s discussion.

In Episode 3: We looked at the prospect of re-inventing the “Modern Software Factory” with agile development and continuous delivery. OK, so, perhaps we were getting a bit ahead of the basics here, but I’d maintain that this is an important piece to consider upfront that builds on these other building blocks. As with the topics covered in Episode 2, it is critical to understand the big picture challenges and benefits before you begin to get your hands dirty.

And now for some nitty gritty.

In Episode 4, we hear from Shamim Ahmed, CTO of Continuous Delivery Solutions at CA Technologies, as he answers the nine burning questions he most often hears in speaking to enterprise practitioners as they look to employ DevOps and Continuous Delivery practices in the real world.

So, while these previous conversations were material, this is a crucial set of topics to consider as you invoke your own strategy, including:
1. What is the difference between DevOps and continuous delivery?
2. What are the key impediments/obstacles to achieving speed/quality in CD?
3. How do we overcome these obstacles?
4. How do we get started on the CD journey?
5. What are the key top-level metrics you want to track in a CD program?
6. What org changes do we need to make to implement CD?
7. What the key capabilities we need to implement?
8. How do we integrate all the different tools we have in our environment?
9. How will the future of continuous delivery evolve?
OK, that last part is a bit more postulation on our part—though we like to think on the practical side, based on the answers to all these other important real-world adoption dynamics.

Tune in and hopefully learn a good deal about where your approach stacks up, and needs to evolve.

As always, you can find said Episode 4 of the Continuous Delivery 101 Podcast by:
- Listening to it on your computer on YouTube
- Downloading it from iTunes store
- Downloading it via the Podcast App by searching for “Continuous Delivery 101 podcast”
Enjoy!

— Scott Edwards
April 21, 2017
From Artisanal to Factory: The IT Pendulum is Swinging Back

Splunk Vice President of IT Markets and Chief Technology Advocate Andi Mann sees DevOps moving from an artisanal approach to an IT factory.

This represents a reversal: Historically, developers operated in a factory-like environment, passing chunks of work down the line, but that caused problems because “different managers owned different sections of the production line.”

Agile and DevOps were a response to that issue, he said, resulting in “the artisan developer … the full-stack developer.”‘

The trend is now flowing in the opposite direction, with Splunk customers wanting to see “the whole flow.” Furthermore, enterprise DevOps is too big for an artisanal approach.

“I worry about that [reversal],” he said, but technology can help artisanal quality within an IT factory. Collaboration tools such as permanent video links between different locations enable specialists to work together and understand each others’ work.

“That’s a different kind of factory,” said Mann, in which development, security, QA, operations and other staff do not need physical proximity to work closely together. This is particularly important for large enterprises, he observed, as you cannot pack 1,500 developers into a small space.

One example he cites to show this approach can work is a telco with its developers spread across multiple locations in Sydney and Singapore using DevOps practices to achieve continuous delivery. Another is one of the largest UK retailers getting weekly releases from developers in London and Mumbai, with QA work done at another location in India.

“Executives I talk to are very heavily committed to improving cycle time,” said Mann. But they are also concerned with quality (e.g., problems should be detected and fixed before they reach the user, and the percentage of changes sent for rework should be minimized) and business impact. Executives want to link DevOps to their own KPIs: “Are they moving the needle on my business requirements” such as shareholder value and reputation?

“You get the changes you drive by your measures,” he said, so it is up to managers to make sure their measures make sense in the broader context. It may be time for a balanced scorecard to be applied to DevOps, with metrics in all three areas, he suggested.

These metrics could include code coverage, error rates, customer complaints or trouble tickets raised, plus business impact measures such as click-through rates, cart abandonment, subscriptions cancelled, open rates (for mobile apps) and customer sentiment.

Mann noted executive demand for the adoption of DevOps, even where “they don’t even know what they’re talking about.” He feels it is really about a desire for visibility into the development pipeline, and “that’s really challenging.”

Part of the problem seems to come from the tension between traditional IT (especially “ITIL-owned shops”) and new-generation executives with titles such as head of digital.

Slavish adherence to ITIL practices won’t allow the required speed, may not achieve the expected level of compliance and business goals probably won’t be met, he suggested.

“ITIL came out of a fear of failure,” but failure is no longer catastrophic. “If you’re not failing, you’re not trying, you’re not innovating.” What’s important is to fail fast, fail small, and fail cheap—and DevOps allows IT to fail in low-impact ways. Even when an organization is making significant changes to software, the effects of failure can be contained through measures such as canary releases (where the new versions are initially made available to a small subset of users).

Furthermore, customers are getting used to the idea that technology isn’t and doesn’t need to be perfect, largely because of their experiences with speech recognition services such as Siri, he said.

— Stephen Withers

April 7, 2017
DevOps in Legacy Systems: A Mission Impossible?

The word “legacy” evokes mixed reactions from the IT and business crowds alike. For those who depend on legacy systems such as the zSeries and iSeries platforms, it is a way to deliver stable and reliable applications into production. Business stakeholders who may not know the intricacies of the technology often take a view of, “Why fix it when it ain’t broken?”

On the other hand, for those who haven’t experienced the joys and pains of legacy systems , legacy world will continue to defy their logic and assumptions because their view of the world is shaped by the open system world which has completely different characteristics: being more malleable to change and easy to maintain etc. Hence there is this great divide between the legacy and new worlds. Add Agile and DevOps to the equation with the lofty goals of increased release frequency and reduced lead times, we end up with a divide that is too far to bridge.

The likes of Gartner have come up with constructs such as two-speed delivery to help deal with this legacy challenge. In my view, there is a problem with the idea of dual-speed because it is based on a wrong analogy to start with — that of allowing vehicles to move at different speeds depending on whether you are on the motorway or a city road. That analogy works well for traffic management scenarios but not for complex IT architectures we see in the enterprise. The dependencies across the systems in the enterprise landscape are just too intricate and too tightly coupled to be defined using such a binary logic. In my experience, I find that many Agile and DevOps transformations hit the roadblock and never achieve the desired levels of productivity and effectiveness because of an oversimplified view of these complex dependencies across legacy and open systems.

The three biggest challenges that prevent a legacy DevOps transformation are mindset problems, architectural constraints and tooling limitations.

The Mindset Problem

“There are no legacy systems, just legacy thinking.” That’s a great quote I heard recently from an engineer in a large organization trying to adopt DevOps at enterprise scale. The initial resistance from teams to move into DevOps ways is due to reluctance to change, emerging from an inertia of doing things a certain way for years if not decades. The mindset problem is as much a problem for those outside the legacy world as those inside. The “outsiders”—typically leadership, management team and consultants—tend to oversimplify the problem and, too often, rely on examples from the open systems world. This just alienates the incumbent legacy teams and makes them even more skeptical of DevOps driven change. The outsiders need to have a genuine appreciation of the challenges with legacy architecture and need to work in collaboration with the legacy teams if they really want to transform their organization.

Architectural Constraints

Legacy systems have certain characteristics such as being static, tightly coupled and monolithic blocks of procedural code. These systems were not built for the modern Agile workflows that focus on incremental and iterative deliveries. A significant majority of these systems are written to operate in batch mode.Testing the systems requires cycling through multiple business days, which simply limits the level of agility you want to introduce in the organization. I am currently working with a legacy engineering team in the financial services space where you have to cycle through a minimum of 20 business days to simulate even 70 percent of the test scenarios. However, there are a number of solution options to overcome the architectural challenges; they range from killing applications to migrating to modern platforms or re-engineering legacy code. Not surprisingly, the right approach depends very much on your context.

Tooling

Tooling in the legacy world to support a DevOps ways of delivery has always been a challenge. But it is fast changing now, thanks to a number of startups and even old-school IT product companies investing in this space. If you consider zSeries as an example, there are better IDEs (integrated development environments) such as RDz that can replace traditional code editors typically based on “green screens.” These IDEs provide great features such as better navigation, debugging, split views and searches on the legacy code. One of the CIOs I worked with made it clear to his team that he wanted to get rid of “green” screens from the IT floor. There are tools such as SmartBear Collaborator that support a great peer review experience and help bring one of the original XP practice of pair programming to the legacy world. There are tools such as endeavor from CA that promise automated build and deployments in the z-series world. Similarly, there are tools that support test automation of mainframe applications and tools for code coverage and code quality checks on legacy programming languages like COBOL. XATester looks like a promising tool for unit test automation from a Danish startup called Xact.

While this write-up is not intended to be a compendium of all DevOps and engineering tools, the central message is that there are a number of DevOps tools emerging on the legacy landscape which can help your legacy delivery teams get on to the DevOps and Agile journey.

What is more important is for organizations to realize that many of these legacy systems will remain part of their IT landscape for the next foreseeable future and it is time to invest in these core systems. They may not look as appealing as the digital platforms, but they are fundamental for businesses and you cannot ignore them on your DevOps journey. DevOps in the legacy world is not going to be smooth, but it is certainly not a mission impossible!

— Aditya Vadaganadam

March 17, 2017
Getting the Most Out of QA in Complex Environments
We live in a world in which software is increasingly dominant. Not only do we rely ever more on software, but its complexity is becoming even greater. And with increased complexity, bugs are more likely to be intermittent, making failures difficult to reproduce. Sometimes this non-determinism results in failures never getting fixed. Any unfixed bug is potentially a security breach or catastrophic customer outage waiting to happen. With such risks on the line, ensuring your software is of the highest quality is essential to running a successful operation.

To cope with this complexity while still providing some assurance of delivered software quality, the DevOps industry has embraced a variety of automated testing regimes. A development team of a given size is able to run many more tests than it could before and must triage, and attempt to resolve, a correspondingly larger number of test failures. Diagnosing the cause of each test failure can still take a lot of time, particularly if the problem is intermittent or otherwise not easily reproducible. In fact, developers can spend on average as much as 50 percent of their programming time debugging, not coding. Think of all the innovation that could occur if that time were reallocated.

Traditional QA

When it comes to Quality Assurance (QA), developers and testers are all aware of the sequence of deliverables, beginning with development delivering an untested product to QA, which releases the tested and approved product to the end user. However, when a bug is found, QA and Development must communicate with one another to solve the issue quickly. The effectiveness of the delivery channel between QA and Development hinges on the quality of the information about the failure.

Some common challenges raised by inefficient QA processes include:
- Complex control flow – Difficult to make inferences about how a failure unfolded
- Non-deterministic failures – Hard to reliably reproduce a failure to investigate the root cause
- International R&D – Collaboration is made more difficult by geographic separation
Conventional test systems inform that there is a bug but offer limited information about the cause. Workflows such as continuous integration (CI) often offer clues, such as informing which commit to the codebase caused the failure. However, there are several cases where CI cannot directly help identify problematic code changes. For example, the triggering code change may be entirely correct, yet cause a pre-existing bug to manifest. The developer cannot simply stare at the modification in the expectation that the source of the bug will eventually appear. Conversely, the failure may not easily reproduce for the developers even if they run the same test again. In short, CI is often very good at making the easier bugs very easy to fix, but typically is of little help with the more difficult bugs, particularly intermittent ones.

The Next Evolution of QA

To bridge the gap between identifying a failure in the QA process and resolution by development, DevOps teams have adopted a variety of technologies to complement their automated testing setups. New to this scene is technology capable of recording all or part of a program’s execution for subsequent replay and analysis, capturing test failures as they happen. The recording contains all the information a developer needs to understand the exact set of conditions that led to the test failure. It can be analyzed on different machines to the one on which the failure occurred, facilitating collaboration between colleagues who can work on separate machines to resolve the same issue—a particularly useful benefit for large software companies with geographically distributed teams. By gaining the ability to debug an exact replica of the failure, developers immediately reduce the time and effort required to diagnose and fix failures, therefore speeding the development cycle.

Being able to generate recordings of failed tests, especially those intermittent issues which might not otherwise get fixed, coupled with the productivity gains of increased collaboration between development and QA allows companies to maintain a competitive advantage in a volatile marketplace.

About the Author / Greg Law

Greg is the co-founder and CEO of Undo. Greg has 20 years’ experience in the software industry and has held development and management roles at companies including the pioneering British computer firm Acorn, as well as fast-growing startups, NexWave and Solarflare. Greg left Solarflare in 2012 to lead Undo as CEO and has overseen the company as it transitioned from the shed in his back garden to a scalable award-winning business. Greg holds a PhD from City University, London, that was nominated for the 2001 British Computer Society Distinguished Dissertation Award. He lives in Cambridge, UK with his wife and two children.
March 16, 2017
The Tester: Saving Your Soul in a DevOps World

Software testers have seen their careers go through some ups and downs in recent years. In the days of Waterfall, the tester’s job was clearly defined. Like other functions of software development and delivery, testers lived in a neat and tidy universe, testing features and design, finding bugs, sending back feedback to developers for correction and eventually, passing off code to the production people to work their magic. Testers had discrete roles with strong value propositions in the software organization.

Then, Agile came along and new teams were created, lumping together developers, QA and Ops people for the sake of better collaboration and efficiencies. This began to create some confusion regarding who was responsible for what, when.

DevOps brought order to the process, adding automation to every step, speeding cycles and reducing bottlenecks in a highly standardized, streamlined approach. In the continuous delivery pipeline, each step of the process creates its own contract for software quality—and that’s good.

Development teams still value high quality, which customers demand. Yet in some DevOps circles, there’s been talk that testers aren’t needed much anymore. They bring bottlenecks to the process, forcing developers to work outside of the DevOps toolsets for code management and testing to respond to a defect ticket posted by the tester in an issue tracking tool.

Eliminating that layer of process helps developers be as efficient as possible using continuous integration (CI) tools, proponents say. As well, some developers are more than comfortable handling testing, using test-driven development (TDD) methodologies.

On the other hand, there’s faulty thinking here. Testers, whether deploying manual or automated testing methods, provide a valuable second set of technical eyes. Testers have a solid take on the user experience perspective, not just the fact that the code executes correctly.

This human behavior and design insight is critical to the ultimate success of a software application. Finally, not all developers want to spend time testing nor have the skills to do so outside of unit testing.

Moving forward into this new age of DevOps, testers play a vital role. The distribution of the tester into each phase of development (rather than at the end) is necessary to maintain both high standards of delivery and to keep communications fluid between different parties.

Here’s how testers can contribute to DevOps teams:

Testers need to be included in the continuous integration process

Developers might be automating their unit test coverage, but they don’t replace all the test activities a tester can bring to the table. There are still other types of functional and non-functional test activities that must occur to validate the health of the product. Instead of transferring more testing responsibilities to the already overworked developer, testers can integrate their processes and systems with those of developers.

Although there are version control tools (GitHub) and feature tracking systems (JIRA) to help developers, testers are often disconnected from these central tools. To connect with developer tools, testers can start creating and versioning their test cases within those development platforms. By doing so, testers can radiate feedback to the development team on issues to fix quickly through source code check-ins and in a way that won’t disrupt the normal flow of work.

If testers can learn to use the DevOps tools developers love, much of the conflict goes away.

Testers should increase their adoption of test automation, no matter their skill level

Testers can learn newer test automation driven frameworks, especially those that help with transitioning manual test cases into automated scripts that can be reused, such as Cucumber or Gauge.

For example, Gauge allows testers to write test specifications in Markdown, which provides a simple transition to automate your favorite programming language. These newer tools help testers transition to test automation despite not having experience writing tests this way—a common issue for testers working in DevOps.

A growing number of test case management providers are starting to integrate their test design and execution around these automation frameworks to make the manual to automation transition smoother.

Although developers might have automated unit test coverage, they struggle with developing functional and user acceptance automated tests. Testers can provide value by suggesting to the development team which are the best areas for test automation based on common issues and their unique knowledge of the application.

They also can recommend areas for exploratory or manual test execution, depending on application complexity. This is especially helpful when teams starting down the path of DevOps do not have a regression test suite built that can be triggered through CI.

Testers should focus on quality over finding defects

Developers often complain that when testers find defects, it slows down their development processes. On a fast-moving Agile or DevOps team, filing bugs in a defect or issue tracking system is a total waste of time and creates unnecessary clutter. Developers want instant feedback on code that’s broken and the related feature, rather than seeing another defect ticket they must open and close upon fixing.

Therefore, testers should be giving feedback directly to the features that are being developed within a branch of code. They can do this through adopting tools that facilitate “ChatOps,” such as HipChat and Slack, which can integrate with version control systems. Testers can also provide direct feedback through a commenting system on the feature itself.

With the advent of Agile and DevOps, which emphasize speed and frequent releases, testing is more important than ever in software development. By making some slight modifications to how they work and the tools they use, experienced testers can add considerable value to these teams by providing another layer of quality checks and a holistic view of an application’s performance and usability.

About the Author / Ryan Yackel

Ryan Yackel is Director of Product Marketing at QASymphony. He is responsible for all marketing activities around product releases, including development of strategic messaging. Ryan also helps to train the Sales teams on key value propositions for new products and features, and he provides customer feedback to the product team for roadmap initiatives. He can be found online at Twitter and LinkedIn.

March 16, 2017
Release Automation: Delivering the Big Quality Payoff

Have you ever gone down the path of testing the latest release of an application only to find out that you had the wrong version?

Maybe you accidentally grabbed the wrong build or you didn’t have the correct configuration file for the environment you were testing against—or your database wasn’t updated correctly, or the script you were using was still pointing to a prior version of the software.

Whatever the case, we can agree that keeping track of all the various artifacts, database schema changes, configuration parameters and other components needed to accurately test and deploy an application is both time-consuming and prone to errors.

Today, many companies are looking to DevOps, continuous delivery and agile development methodologies to shorten their application delivery cycles and reduce the number of errors and the costs associated with application deployment processes. Nearly all enterprise companies are software companies, and their mere existence depends on delivering differentiating products and services via software to their customers.

It’s imperative to be able to react to customer demands and competitive pressures quickly, if not immediately, to stay ahead of the competition. Gone are the days of yearly release cycles.

Today we see companies deploying new capabilities into production environments on a monthly, weekly or even daily basis. At some companies, hundreds, if not thousands, of deployments are occurring across dev, test and production environments daily. The old methodologies we once used simply can’t support this type of volume and cadence.

Employing Release Automation

So, how are these companies transitioning to this new way of thinking? In a word—automation; specifically, by automating early and often.

Everything about your release process is a candidate for automation. This includes environment provisioning and configuration automation, build automation, test automation and release and deployment automation.

A customer of ours recently implemented CA Release Automation, and within eight weeks had automated 90 percent of its deployments. Deployments that once took more than five hours and a team of people now take a matter of minutes.

“This not only saves us a huge amount of time, but it also mitigates the risk of human error being introduced by manual processes,” said Tim Dallmann, platform engineer at Jewelers Mutual.

Another customer, a major airline, implemented CA Release Automation and immediately reduced their production outages from an average of 237 over a three-month period, down to 15, seven, six and five over the following four months.

And Molina Medicaid Solutions reduced their cost of human error by 90 percent, all the while responding quicker to emergency deployment requests and improving deployment quality.

Automation is critical, but I believe these results prove that implementing a framework to manage and deploy release processes in addition to automation works even better.

Increased Flexibility and Visibility

Continuous delivery helps break down barriers often seen between development and operations teams. It provides, even forces, a more collaborative approach and automation is a key enabler of this.

“One of the best things about automation is that it gives us the flexibility to skill-up end users—deployments in our test environments are no longer solely the responsibility of the operations or IT teams,” explained Dallmann.

With an application release automation (ARA) solution, artifact, build, configuration and release processes are no longer tracked with spreadsheets or embedded in complicated scripts.

Increased visibility into these processes ultimately enables all stakeholders to take a more active role in the release process, and access to real-time information can significantly reduce the amount of time and resources needed to resolve and recover from problems when they do occur.

Automation leads to:

Integrating with the Ecosystem

An enterprise ARA solution can help you create a single control point to streamline all your application delivery processes for better predictability and reduced risk. A critical component of this is the ability to integrate and orchestrate tools and technologies required along the continuous delivery pipeline, including source control, automated testing tools, provisioning and configuration management solutions and change management and performance management solutions.

Integration with an application lifecycle management solution can provide even further alignment, governance and traceability between development projects and what is ultimately being tested and promoted to production.

“By integrating the two solutions, we will have a complete audit trail of deployments to help us understand how, where and by whom stories have changed,” said Phil Nickolai, director of application services at Jewelers Mutual.

“What we really needed was a solution to streamline our whole way of working,” he said. “We had numerous people working on projects, competing for resources and causing complexity. Now we’re back in control and getting smarter every day.”

The Time Is Now

Let’s get rid of the problem that exists between the keyboard and information kept in spreadsheets, scripts or emails. Not only will you stop wasting time testing the wrong version of an application, but you can also ensure you have the right test data, the appropriate test cases and the correct configuration parameters for your environment.

Stress-inducing release weekends and nights become a thing of the past. Developers can spend more time on developing innovative capabilities that generate revenue and keep customers coming back for more.

— Tim Mueting

March 7, 2017
Why Financial Companies Love Continuous Delivery
The other day I was on the American Express website paying bills and browsing around the benefits and services afforded me when I ran across a “who we are” section. It read:

“We’re a global services company that provides customers with access to products, insights and experiences that enrich lives and build business success.”

Surprisingly, the term “financial” was missing from the description. This led me to wonder if other financial institutions are evolving how they think about themselves. Here are some other examples:
- Bank of America: “Here to help connect you to what matters most.”
- ING: “Empowering people to stay a step ahead in life and in business.”
- MasterCard: “As a technology company, we connect individuals, businesses and organizations around the world, creating greater opportunities for all.”
Financial Companies Love Their Customers

My unscientific assessment of various financial services companies suggests that they are centered on delighting their customers. They are transforming who they are by altering their customer engagement models and essentially becoming better software companies. MasterCard has even reclassified itself as a “technology” company. These companies are transcending the role of transaction engine to offer seamless, intuitive digital experiences for their customers.

And the proof is in the performance. According to Forrester’s Customer Experience (CX) Index, financial services firms, including retail banks, credit card companies, investment firms and auto and home insurers, are some of the top brands worldwide when it comes to CX.

You might think that large financial institutions would be digital laggards because of legacy infrastructure, regulatory compliance pressures and overall risk-avoidance. But you’d be wrong.

In another study, 81 percent of financial services firms said they are embracing digital to redefine their businesses, and 87 percent are on the DevOps bandwagon. Sixty-one percent even said they are disrupting their industry with advanced digital initiatives.

To stay competitive and improve their customer experience aptitude, financial services firms are undergoing significant digital transformation.

Getting the Pulse of Fintech Continuous Delivery Adoption

But how progressive are these firms in terms of their application delivery practices? Working with CA, Gatepoint Research recently surveyed more than 100 IT executives from financial services companies to take the pulse of where they are in continuous delivery adoption. Results can be found in this report and infographic.

First off, to see if these companies might be considered unicorns or stragglers, they were asked the total volume of monthly deployments in all environments. Forty percent fell at the low end with fewer than 50 deployments per month. Now, lower deployment numbers could mean that these are less agile organizations or that they have fewer applications in general. Somewhat correlative to lower deployment numbers, the survey found that 48 percent currently are not using an application release automation (ARA) solution. Without automation, it’s tough to scale release processes—particularly to keep up with agile development.

Moving up the spectrum, 41 percent are moderately prolific at 50-500 deployments per month, and then 15 percent are what I would consider rock stars with more than 500 (7 percent of those over 1,000). Those doing hundreds of deployments a month often have agile development teams and more modular applications. They likely have continuous delivery practices in place that push more frequent, iterative deployments left, helping higher-quality applications to move through the pipeline to production.

The primary application delivery goals for financial services firms for the next 12 months are to get better control of release processes, tools and environments; to reduce risk and failures; and to release applications more frequently. All of these needs typically stem from companies undergoing digital transformation—it feels more chaotic and disconnected, manual processes are breaking under the weight and there’s intense pressure to move faster.

Compliance was not a top goal, and the majority (83 percent) claim that their current release process is adequate to meet regulatory requirements. However, they did acknowledge that they could improve in terms of knowledge safekeeping and auditability of the release process.

The data certainly shows that financial services companies are adopting continuous delivery practices in some form and exploring solutions, including many open-source tools, to help them get the job done. They currently are focusing continuous delivery practices on tracking and reporting releases, planning and scheduling releases and automating test deployments. These practices ultimately help financial services companies to release high-quality applications anywhere at any time.

How Continuous Delivery Helps Fintech Win

Continuous delivery sits at the heart of many digital initiatives because of what it enables these companies to achieve for their customers:
- Delivering features and updates that customers need when they need them
- Delivering applications that meet performance and quality expectations of customers
- Delivering secure, compliant applications (for external and internal customers)
Continuous delivery enables continuous devotion to the customer experience.

Continuous Release: Fast Customer Response

With continuous delivery, you establish an application release workflow that automatically connects a developer’s update to parallel testing to staging and ultimately to production release. Because of the underlying end-to-end automation technology, these steps can happen quickly, seamlessly and with little to no manual intervention. Some might call it “push button release” or “zero-touch release.”

It’s a controlled, tracked, governed, collaborative process. Application stakeholders stay aligned as application changes traverse the pipeline. Your customers can have access to fixes, updates and new innovations in days instead of weeks or months.

Continuous Testing: Building in Quality

In contrast to the significant time penalties and shortfalls that can happen with manual testing, in the continuous delivery pipeline, testing is shifting left and happening in parallel. Hand-offs and feedback loops are automated and efficient.

Testing can happen earlier and without delay with the help of test environment simulation. Testing is on target through automatic test case creation, even from requirements. Testing is real-world with on-demand test data generation. And testing is rigorous with developers able to performance test apps at their end. The continuous testing that occurs as part of the process helps to build in quality and ensure top performance before reaching the customer.

Continuous Compliance: Audit-Ready Processes

Ensuring applications are compliant and secure is critical for your customers. It’s making sure customer information is safe and that application changes are tracked sufficiently for SOX or other reporting. It’s also about helping out those internal “customers” like auditors to prove compliancy and easily build reports.

An automated continuous delivery pipeline can track releases, apply role-based governance, orchestrate proper data masking, deploy security policy updates and provide an audit trail for application releases—who did what, when and why. All this data can be collected and reported easily, which saves time and effort for auditors. You essentially stay compliant and secure all the time.

Customer Satisfaction Through Automation

Financial services companies are laser focused on using technology to delight and retain customers. And continuous delivery is the application delivery practice that’s underpinning these efforts. The companies that are exceptional at this, with thousands of deployments per month, are using automation tools such as Application Release Automation (ARA). These tools are the backbone of continuous delivery, providing the automated orchestration, governance, tracking and reporting for your pipeline—development through production.

That’s why I’m not surprised that leading financial services companies make up a large portion of CA’s release automation customers, including Euroclear, GM Financial, ING, Jewelers Mutual and Western Union. Our customers put high value on delivering excellent customer experiences and are implementing modern software delivery methodologies, practices and associated tools to support their efforts. GM Financial recently said it is “the software company that does auto finance.”

If you are serious about customer satisfaction and staying ahead of nimble competitive threats in the market, you must get serious about end-to-end automation. It’s time to truly embrace the practice of continuous delivery to be able to consistently release applications that are progressive, always compliant and completely customer-satisfying.

— Patricia Johnson
March 2, 2017
Dispelling the Myths Around Continuous Testing

You may have seen the term “continuous testing” and thought it was just the DevOps flavor of the month. Some might even think it isn’t part of DevOps or isn’t for cloud or hybrid cloud applications. Well, think again.

As it turns out, there are a few more fallacies around the idea of continuous testing floating around the application development community. Ready to find out?

Chapter five in the new, free eBook “Continuous Testing for Dummies, IBM Limited Edition,” lists 10 myths. Here are a few to ponder:

Continuous testing is only about executing test scripts

Limiting the term “testing” to only the execution of test scripts is a misnomer. As my co-author, @Al_Wagner and I point out, testing and checking application functionality are not equal. There’s a long list of testing items to check off one’s list, the least of which is collaborating.

Continuous testing is merely a buzzword

Guess what? It’s a core part of that essential DevOps practice, continuous delivery. And, with customers demanding higher quality software faster, it isn’t going anywhere soon. In short, invest in continuous testing.

Only agile teams do continuous testing

False! Some or all parts of continuous testing are critical for any type of team as well as all players. This includes test automation, production-like test environments and realistic test data. A related myth—only testers contribute to testing—is also not true at all. The idea of continuous testing includes all team members. The book describes the roles everybody has to play. For example, did you know that designers can improve testing? They can work with testers to help them understand where the fragile parts of the application are.

Automating tests means we need fewer tests

No, the machines are not taking over. Automating tests does not mean cutting back on testing jobs. To the contrary, automation is reducing testers’ needs to act like cyborgs and spend more time doing what they do best: learn and experiment.

Continuous testing isn’t for large, complex systems

Au contraire! The best thing you can do for large or complex systems is leverage continuous testing practices and tools such as service virtualization to conduct end-to-end testing earlier and more frequently, rather than having each subsystem do all its own testing in isolation until just before deployment to production.

To read more myths about continuous testing, check out “Continuous Testing for Dummies, IBM Limited Edition,” here.

About the Author / Marianne Hollier

Marianne Hollier is an IBM and Open Group Master Certified IT Specialist in application development. She has strong, practical expertise in measurably improving the software development life cycle and driving the necessary cultural changes to make it work. She is instrumental in architecting, tailoring and deploying the IBM DevOps best practices as well as appropriate software development tools on many types of projects, from large to small, long to fast-track, agile to traditional. Marianne is especially passionate about all things testing—process, tools, culture and automation. Her experience is broad-based, spanning both custom projects as well as standard software packages that apply to many industries, including refining, pharmaceutical, telecommunications, health care, financial, automotive, government and retail. Connect with Marianne on LinkedIn or Twitter.

February 23, 2017