devopsy.kinsta.cloud

Tag: continuous testing

DevOps Matchmaking: APM + CI = A Perfect Marriage

As anyone who has ever been set up on blind dates by friends can attest, finding the right match in today’s world is a tricky feat.

There’s a lot of evidence suggesting that the frantic pace of life makes such matchmaking more difficult than ever. That said, when the right set of complimentary traits exist and there’s a willingness to find common ground, things can work out.

Spike Jonzian-notions of artificial intelligence aside, certain technological partnerships are increasingly being perceived as “a perfect marriage” within the context of DevOps automation.

Register today for the upcoming webcast: “Using APM to Turbocharge Continuous Integration”

These technological matrimonies closely support the underlying tenets of DevOps methodology—collaboration, experimentation and increased velocity. In fact, it’s easy to wonder why no one ever conjoined them before. But we were using traditional waterfall development and operations and so couldn’t conceive of such a thing. On the other hand, emerging availability of tooling supporting new integrated workflows represents a significant catalyst.

Finding the Perfect Fit

One such DevOps confederation getting a lot of attention is the matchmaking of continuous integration (CI) and application performance management (APM) technologies. Tying together these once-disparate systems makes a lot of sense, despite the fact that few tools would seemingly appeal to more (traditionally) polarized constituencies.

Continuous integration has been a staple of the development community for many years. It emerged as teams evolved the practice of long build times into more frequent check-ins to deliver code and look for potential errors more quickly. Today, with the rise of CI tools such as the Cloudbees Jenkins platform, some development teams are touting tens of thousands of builds per day.

On the other end of the spectrum, squarely on the turf of IT operations, APM has been evolving rapidly over the past few years as organizations have sought to improve use experience and applications reliability via dedicated monitoring. Enabling this ability to identify and address emerging issues has resulted in APM becoming the fastest-growing segment in the IT ops management market, according to Gartner.

DevOps matchmaking factors in when you bring these tools together to create an automated workflow that spans their respective use cases. A cross-functional team can leverage APM-level monitoring and KPIs within the domain of CI and testing to identify potential performance obstacles introduced in new code. Doing so then rather than waiting to respond later in the applications life cycle benefits everyone.

Reaping the Benefits

For example, say an e-commerce applications developer has implemented a new inventory search functionality—something considered a key differentiator by business management. The developer knows this feature could rely heavily on some back-end database system and wants to ensure it’s tested properly, even as the code is being introduced.

Integrating APM and CI allows the entire team to monitor the process and test various conditions related to the update, tracking multiple builds and looking out for related performance issues. Such integration also allows for centralized tracking of other key attributes such as build number, status and build date, to further ease management.

Most importantly, by tying these two critical aspects of the applications delivery life cycle together, the collaboration and feedback loops required to truly embrace DevOps methodology become ever more valuable as these core Dev and Ops functions become intrinsically linked.

If this type of DevOps advancement gets your heart pumping, you’re not alone. Later this month, we’ll host a webcast here on devopsy.kinsta.cloud featuring experts from CA Technologies and Cloudbees. They’ll be discussing the benefits of matchmaking with APM + CI integration. They’ll also detail real-world benefits organizations leveraging integration of our respective technologies (CA APM and Cloudbees Jenkins) enjoy.

We obviously feel that this is a DevOps tooling marriage that will mature, change and stand the test of time. We’d love to garner your questions and feedback on the topic.

Which is all good, because thankfully there’s no such thing as Tinder for DevOps … yet.

June 21, 2016
DevOps: Shift Left to Reduce Failure

DevOps is all about increasing the feedback loop from idea to customer and back again. Tremendous gains have been achieved in automation and tools to drastically reduce the amount of manual labor involved in progressing down the release pipeline. But mission-critical applications in a highly competitive market cannot afford to go down. Complex applications, environments and infrastructure increase risk, which can cause failure.

The failure rate of production deployments is still too large. Studies show that the average cost of a production outage is roughly $5,000 per minute, while the average cost of a critical application failure per hour is $500,000 to $1 million.

Why so much failure? All too often we’re still seeing a lack of consistency across the release pipeline. Many times the provisioning and deployment processes are very different at the opposite ends of the pipeline. But by shifting left the operations concerns, you can achieve much more success in production.

Understanding How to Shift Left

The term “shift left” refers to a practice in software development in which teams focus on quality, work on problem prevention instead of detection, and begin testing earlier than ever before. The goal is to increase quality, shorten long test cycles and reduce the possibility of unpleasant surprises at the end of the development cycle—or, worse, in production.

Shifting left requires two key DevOps practices: continuous testing and continuous deployment. Continuous testing involves automating tests and running those tests as early and often as possible, along with service virtualization to mimic unavailable systems. Continuous deployment automates the provisioning and deployment of new builds, enabling continuous testing to happen quickly and efficiently.

The first and most obvious way to shift left operations is to work side by side with development in creating the deployment and testing processes. Failures observed in production often are not seen earlier in the life cycle. Many times these failures can be attributed directly to differences in deployment procedures. Development may create its own deployment procedures that are very different from those used by operations for production. Sometimes production procedures are much more manual and may even use different tooling. Operations and development need to take ownership in building standard deployment procedures. The deployment process is then practiced potentially hundreds of times in test environments before reaching production. You then have much more confidence that the production deployment will be successful.

Another way to reduce the failure rate is to make all environments in the pipeline look as much like production as possible. This is almost impossible using traditional provisioning processes. Production environments usually are much heavier and detailed than are needed by development. But by using cloud and pattern capabilities, you can get much closer.

The ability to use patterns to define consistent environments eliminates the failures that occur simply by configuration inconsistencies. This again requires operations and development to work together in creating the provisioning process so developers create their test environments the same way they are created in production. Using IBM UrbanCode as an example, this can be done by connecting UrbanCode Deploy to an automated self-service provisioning system such as OpenStack HEAT or IBM Cloud Orchestrator connected to OpenStack.

These standard patterns can be used to create environments for all stages of the pipeline. And by using cloud capabilities you can deprovision these environments as fast as you can create them. This also allows the enterprise to have confidence in their environment patterns. They are tested on a daily basis and, therefore, operations can feel confident that testing is done on production-like environments.

No one wants to experience production failures—and it is in everyone’s best interest to do all they can to prevent them. At the root of DevOps is the removal of those silos between development and operations. By applying these shift left operations principles you are well on your way to reducing your failure rate.

The ingredients to make shift left are real and available: automated testing and deployment tools, service virtualization technology, agile practices and cloud environments. The main obstacle that remains is the mindset of delivery organizations. Learn more from the newly updated Application Deployment and Release for Dummies ebook to find out more.

About the author / Dr. Darrell R. Schrag

Dr. Darrell R. Schrag (@devopsdrs) is an experienced software lifecycle consultant specializing in the IBM cloud solution, helping customers establish their cloud strategy across the enterprise. He’s focused on implementing continuous delivery of software applications through the delivery pipeline incorporating cloud and provisioning solutions. He’s experienced in helping clients transition to “born on the cloud” applications utilizing microservices to take advantage of a flexible and nimble cloud platform.

June 2, 2016
Data Protection for Operational Reporting Platforms
Making real-time business decisions based on data from social, mobile and cloud platforms is a requirement in today’s world. That’s why an ecosystem of next-generation operational reporting platforms such as Spark, Apache Cassandra, Kafka, Docker, Mesos, Marathon and more have come to life. Enterprises are using these technologies to re-architect their data storage and processing frameworks to achieve lower costs, scalability and faster response times. However, as enterprises onboard business-critical applications on this next-generation technology stack, they need to prepare for failures or errors that may disrupt customer-facing applications.

One of the key benefits that operational reporting platforms provide is real-time visibility, which allows businesses to make strategic decisions based on actionable insights. However, the new applications generate a large amount of data and archaic operational reporting platforms (those based on traditional data warehouses, data marts and ETL logic) are unable to catch up and, as a result:
- Business decisions are based on stale data
- Fragmented data increases the risk of errors
- Multiple copies of data increase storage costs
These issues have driven many progressive, data-centric enterprises to re-architect their operational reporting platforms. In this new architecture, Kafka serves as a persistent message bus that ingests data from relational stores and provides failure resiliency and message buffering. Spark or Spark Streaming may be used for transformation, aggregation and other lightweight stream processing. These processing nodes may be deployed in Docker containers. Finally, Apache Cassandra serves as a distributed data storage layer that provides linear scalability and high-availability. Kafka also enables the ability to add multiple data consumers such as Hadoop/HBase through Flafka.

However, as enterprises adopt operational reporting platforms, they want to ensure that bad data feeds and human error do not result in data loss. Apache Cassandra natively replicates data but that does not provide the capability to go back in time to recover data efficiently and at scale.

As enterprises onboard their critical applications to achieve development agility, scalability and lower operating costs, they must also explore data protection solutions that are specifically dedicated for scale-out databases.
May 18, 2016
Continuous Testing Live Network DevOps
My earlier blog, “From Laboratories to DevOps: Factories for Continuous Testing,” discussed the importance of and suggested practices for a fully automated, orchestrated software DevOps laboratory infrastructure, akin to a modern factory staffed by robots churning out products at a dizzy pace. The blog applies to development of software for networks, and the practices increasingly are important as the network becomes more software-based thanks to software-defined networking (SDN), network functions virtualization infrastructure (NFVI) and cloud-based networking. The edges of mobile networks, even the radio access networks, are becoming more software-dependent due to the need to position computing resources closer to mobile customers to resolve latency, bandwidth and spectrum concerns.

However, the need for highly automated software processes does not stop within the development lab.

At a recent DevOps Networking event in Santa Clara, California, Google Network Operations Manager Anees Shaikh indicated that Google’s live network has many thousands of nodes and management elements, supported by millions of lines of software-based configuration files that are updated tens of thousands of times per month. The sheer volume and speed of software updates for large networks is not feasible using manual methods. Automation tools for live network software updates are vital.

Typical live network DevOps environments use DevOps continuous tool chains such as Git for software version management, YAML for defining abstract configuration definitions, Jinja2 to translate the configurations into concrete templates that can be consumed by real devices and Ansible for automating the distribution and activation of config files to the network nodes. But what about testing? How can a network operator be confident that the many updates will perform as expected once deployed across the live network? This requires much more than simple syntax and semantics checking of the config file entries.

Recommended Practices for Continuous Testing of Live Network DevOps Environments

For continuous testing of live network DevOps environments to work, here are some best practices to consider:
1. Lab-to-live transition: Prior to deployment of any updates to the live network software updates should be tested in a lab environment using production-equivalent equipment and topologies.
2. What to test? The actual tests to be executed depend on the risk of changes that are being deployed. If a small, low-risk configuration change is all that is being deployed, then a simple configuration check may be sufficient. On the other hand, if a change has the potential to disrupt the network performance significantly, then a much more comprehensive performance test is needed.
3. Test orchestration tools: The test environment prior to deployment will vary depending on the tests that are selected. Test environment orchestration tools are needed to set up network test topologies that match the tests rapidly—and release them just as rapidly so the test resources can be used for the next tests.
4. Elastic on-demand topologies: The ability to scale test resources horizontally and vertically, on-demand, is essential to be able to match the speed of testing required for short duration deployment cycles.
5. Automate live network tests during deployment: To verify software updates do not interrupt or degrade network performance, functional and performance tests between pairs of nodes across the network mesh are required as the updates are deployed in stages across the live network nodes, clusters, centers and regions.
6. Automate live network testing after deployment: After verifying that the deployments are successful, network tests need to be scheduled periodically to ensure latent problems with the new software do not impact the network.
This is just a partial list of suggestions for continuous testing essentials for live network DevOps.
April 11, 2016
Continuous Testing Improves Speed, Quality

We live in a world that expects immediacy. For a society that gets news updates from Twitter in 140-character increments as it happens while we microwave a complete meal in less than five minutes, anything that takes months—or even weeks or days—seems like an eternity.

When it comes to continuous performance testing, organizations and developers need to keep up with the accelerated pace of development common in a DevOps environment—but without sacrificing the quality of the testing. That’s the real challenge.

Twitter can deliver news virtually in real time, but the quality of the “journalism” is sometimes questionable. It’s possible to microwave a meal in a matter of minutes, but few would argue that the resulting food is actually as good as the same dish prepared the old-fashioned way. When it comes to continuous performance testing, you can’t afford to sacrifice the quality of the testing in the name of expediency.

BlazeMeter thinks it has a way to address that challenge. It recently announced new functionality that allows developers to run tests using Gatling, The Grinder, Locust, Selenium, and JMeter—both locally and in the cloud—in parallel using a single, unified control language.

BlazeMeter referenced statements from Diego Lo Giudice, vice president & principal analyst for Forrester, who noted during a recent webinar the demand for software development speed is pushing teams to look at delivery cycles in terms of weeks, days, hours and even seconds.

The ability to run the tests in parallel shortens the build/test cycles. Developers can add more tests without extending the time involved in the testing cycle. Because all of the tests are run simultaneously, the total time will be only as long as the longest single test in the bunch, no matter how many tests are included in the mix. Running the tests simultaneously allows organizations to achieve greater speed without sacrificing the quality of the testing results.

“Developers that want to excel in the era of continuous delivery need tools that give them the most speed and flexibility possible,” said Andrey Pokhilko, chief scientist at BlazeMeter, in a press release. “We’ve been working with the open source community to develop ideal performance testing tools for any scenario. The new functionality combines the best of our community work with the infrastructure we’ve built up at BlazeMeter, marking a new set of capabilities to bring performance testing to new heights.”

The new BlazeMeter capability also streamlines testing by giving developers a unified control language for managing it all. BlazeMeter developed a “performance-tests-as-code” format that lets developers define tests using common YAML or JSON file formats. Testing teams can create load and performance tests as small fragments using any text editor. The BlazeMeter format enables developers to manage test logic and performance thresholds for various open source tools without needing to be an expert in the separate tools themselves.

A unified solution that allows for simultaneously running multiple test loads in parallel seems like a valuable tool for continuous performance testing. If it works as advertised, it seems BlazeMeter has combined simplicity, speed and quality in one.

April 4, 2016
Does DevOps Improve or Undermine Quality?

In my earlier blog, “From QA to Continuous Testing,” I explained that “several industry developments converged to make continuous testing the winning method for software engineering quality assurance (QA).” The “combination of improved test methods, agile processes, continuous integration speed afforded by automation and orchestration tools and low-cost test dynamically configurable infrastructures afforded by virtualization has resulted in DevOps.”

However, a recent survey by Parasoft indicated that some important concerns such as non-functional requirements (NFRs) (examples are: reliability, testability, availability, resiliency) are being overlooked in many organizations that are operating fast-paced Agile and DevOps environments.

Some results gleaned from Parasoft’s survey run counter to popular expectations:
1. Agile teams are less likely to monitor NFRs than non-agile teams.
2. Ownership of quality, especially for agile teams, tends to be QA team members.
3. Business members think NFRs are being measured but technical folks do not.
4. Confusion and disagreements exist regarding who owns NFR objectives.

The report implies that quick Agile and DevOps processing cycles steal time from broader QA concerns.

The Question

So what is the truth? Do agile, DevOps and continuous testing improve QA, or do they undermine quality?

My own observation is that DevOps can improve or undermine QA. The outcome depends on whether an organization explicitly takes a holistic view of quality that assures all of the “ilities” are taken care of during agile, DevOps and continuous testing.

A complete DevOps environment requires a more sophisticated view of continuous testing than simply “shift left,” “fail early,” “fail often” and “test fast,” which are four popular tenets espoused by the DevOps community. A fifth tenet, “relevance,” must also be fully considered. With “relevance” as a tenet of DevOps QA, the entire DevOps and continuous testing pipeline has woven into it, from end to end, the concept that all of the quality aspects relevant to customers are emphasized in continuous testing and analysis of test results. When relevance is fully considered as an equal tenet to the others, then NFRs and “ilities” are not forgotten and the tools, goals and tests explicitly take into account all aspects relevant to customers’ perception of quality. What is more, when relevance is applied to continuous test selection and results analysis, the system not only produces better quality but it also is more effective and efficient.

At Spirent we think that a holistic approach to DevOps and continuous testing, including “relevance,” is critical for successful DevOps. When continuous testing is implemented with relevance as an equal tenet, quality is improved.

You can read more about our views at Spirent.com/solutions/DevOps

What are your experiences and suggestions?

April 4, 2016
Evolving to Continuous Testing
As agile development practices mature and DevOps principles infiltrate our corporate cultures, organizations are realizing the distinct opportunity to accelerate software delivery. However, when you speed up any process, immature practice areas such as testing and roadblocks become much more pronounced. It’s the difference between driving over a speed bump at 5 mph vs. 50 mph … at 50 mph, that speed bump is going to be quite jarring.

Accelerating any business process will expose systemic constraints that shackle the entire organization to its slowest moving component. In the case of the accelerated software development life cycle (SDLC), testing has become the most significant barrier to taking full advantage of more iterative approaches to software development. For organizations to leverage these transformative development strategies, they must shift from test automation to continuous testing.

Drawing a distinction between test automation and continuous testing may seem like an exercise in semantics, but the gap between automating functional tests and executing a continuous testing process is substantial. This gap will be bridged over time as the process of delivering software matures. Both internal and external influences will drive the evolution of continuous testing. Internally, agile, DevOps and lean process initiatives will be the main drivers that generate the demand for change. Externally, the expense and overhead of auditing government and industry-based compliance programs will be the primary impetus for change.

A Roadmap for Testing

Any true change initiative requires the alignment of people, process and technology—with technology being an enabler and not the silver bullet. Yet there are some basic technology themes we must explore as we migrate to a true quality assurance process. In general, we must shift from a sole focus on test automation to automating the process of measuring risk. To begin this journey, we must consider the following:

Driven by business objectives, organizations must shift to more automated methods of quality assurance and away from the tactical task of testing software from the bottom up.

From Causal Observations to Probabilistic Risk Assessment

With quality assurance (QA) traditionally executing manual or automated tests, the feedback from the testing effort is focused on the event of a test passing or failing—this is not enough. Tests are causal, meaning that tests are constructed to validate a very specific scope of functionality and are evaluated as isolated data points. Although these standalone data points are critical, we must also use them as inputs to an expanded equation for statistically identifying application hot spots.

The SDLC produces a significant amount of data that is rather simple to correlate. Monitoring process patterns can produce very actionable results. For example, a code review should be triggered if an application component experiences all of the following issues in a given continuous integration build:
- Regression failures greater than the average
- Static analysis defect density greater than the average
- Cyclomatic complexity greater than a prescribed threshold
From Defect Documentation to Simulated Replay

The ping-pong between testers and developers over the reproducibility of a reported defect has become legendary. It’s harder to return a defect to development than it is to send back an entrée from a world-renowned chef. Given the aggressive goal to accelerate software release cycles, most organizations will save a significant amount of time by just eliminating this back and forth.

By leveraging Service Virtualization for simulating a test environment and/or virtual machine record and playback technologies for observing how a program executed, testers should be able to ship development a very specific test and environment instance in a simple containerized package. This package should isolate a defect by encapsulating it with a test, as well as give developers the framework required to verify the fix.

From Structured Data to Structured and Unstructured

The current tools and infrastructure systems used to manage the SDLC have made significant improvements in the generation and integration of structured data (e.g., how CI engines import and present test results). This data is valuable and must be leveraged much more effectively (as we stated above in the “From Causal Observations to Probabilistic” section.

The wealth of unstructured quality data scattered across both internal and publicly-accessible applications often holds the secrets that make the difference between happy end users and unhappy prospects using a competitor’s product. For example, developers of a mobile application would want constant feedback on trends from end user comments on:
- iTunes app store
- Android app store
- Stackoverflow
- Twitter
- Facebook
- The company’s release announcements
- Competitors’ release announcements
This data is considered unstructured since the critical findings are not presented in a canonical format: parsing and secondary analysis are required to extract the valuable information. Although these inputs might be monitored by product marketers or managers, providing these data points directly to development and testing teams—in terms that practitioners can take action on—is imperative.

From Dashboards to Business Policies

In a Continuous Everything world, quality gates will enable a release candidate to be promoted through the delivery pipeline. Anything that requires human validation clogs the pipeline. Dashboards require human interpretation—delaying the process.

Dashboards are very convenient for aggregating data, providing historical perspectives on repetitive data, and visualizing information. However, they are too cumbersome for real-time decision making because they do not offer actionable intelligence.

Business policies help organizations evolve from dashboards to automated decision making. By defining and automatically monitoring policies that determine whether the release candidate is satisfying business expectations, quality gates will stop high-risk candidates from reaching the end user. This is key for mitigating the risks inherent in rapid and fully-automated delivery processes such as Continuous Delivery.

From Tool Dependent to SDLC Sensors

Let’s face it—it’s cheap to run tools. And with the availability of process intelligence engines, the more data observations we can collect across the SDLC, the more opportunities will emerge to discover defect prevention patterns.

Given the benefit of a large and diverse tool set, we need to shift focus from depending on a single “suite” of tools from a specific vendor (with a specific set of strengths and weaknesses) to having a broad array of SDLC sensors scattered across the software development life cycle. And to optimize both the accuracy and value of these sensors, it’s critical to stop allowing tools to be applied in the ad hoc manner that is still extremely common today. Rather, we need to ensure they are applied consistently and that their observations are funneled into a process intelligence engine, where they can be correlated with other observations across tools, across test runs and over time. This will not only increase the likelihood of identifying application hot spots, but will also decrease the risk of false negatives.

About the Author/Wayne Ariola

Wayne Ariola, Chief Strategy Officer, leads the development and execution of Parasoft’s long-term strategy. He leverages customer input and fosters partnerships with industry leaders to ensure that Parasoft solutions continuously evolve to support the ever-changing complexities of real-world business processes and systems. Ariola has contributed to the design of core Parasoft technologies and has been awarded several patents for his inventions. A recognized leader on topics such as service virtualization, SOA and API quality, quality policy governance, and application security, Ariola is a frequent contributor to publications including software test & performance, SOA World, and ebizQ—as well as a sought-after speaker at key industry events. Ariola brings more than 20 years strategic consulting experience within the technology and software development industries.
March 29, 2016
Flash Mob Inflection: Rugged DevOps Revolution

Truthfully, I was never a huge fan of the HBO series “The Sopranos.” It’s not that it wasn’t entertaining; I just didn’t agree with the “best ever” label that so many espoused during the show’s halcyon days.

This had something to do with living in Hoboken, N.J., at the time and feeling the show was a heavily glamorized caricature of the real-life Mafioso I still perceived to be in my midst. More specifically, it’s hard for anything mob-related to stand up next to “Goodfellas” and “The Godfather”—among my favorite films, ever.

Either way, I currently find myself identifying with the series’ defining sentiment (copped directly from The Godfather, but hammered home as The Sopranos main theme): “Just when I thought I was out … they pull me back in.”

It wasn’t that I assumed that I’d never delve back into the domain of IT security after transitioning into the DevOps market last year; leveraging DevOps methodologies to improve security, policy compliance and related workflows stands as a core element of the overall movement.

But, in recent months and weeks it has become increasingly obvious that the security facet of the larger DevOps revolution appears poised for explosion as the buzz around emerging best practices, business models and solutions providers grows from a low hum into a steady roar.

After spending a few days talking DevOps security with Derek Weeks of Sonatype at CA World ’15 (we were booth neighbors and dinner mates) and giving more consideration to his company’s angle on securing the systems development life cycle (SDLC) supply chain, the internal gears really got spinning.

Regardless of whether you buy into Sonatype’s security-heavy approach, the concept of aggressively and proactively addressing security across the DevOps spectrum makes infinite sense. The ultimate downfall of applications security has always been the challenge of securing all the underlying layers of code (and today, microservices), especially as applications are assembled using heavier doses of shared, open source componentry.

As further evidence, nearly every major solutions vendor focused on the DevOps space—whether oriented toward the vibrant application performance management (APM), configuration management and continuous testing segments, among many others—seems to have launched some initiative aimed at integrating security into their products.

The rise of security capabilities adjacent to—and supporting inclusion into—the DevOps world, such as container giant Docker’s Content Trust initiative, along with the advancement of related startups including Twistlock, Scalock and StackRoxs, offers more proof of this growing momentum. And that’s only the container security piece of the puzzle.

Of course, there are also a number of emerging startups aimed at addressing the DevOps security opportunity, backed by well-known venture capitalists. A shortlist might include providers such as Immunio, Prevoty and UpGuard (formerly known as ScriptRock, which recently rebranded itself under the guise of a “DevOps security” provider), and we’re sure to see many more. And don’t forget the growing emphasis on DevOps security among established security industry stalwarts such as CyberArk, Snort and Tripwire, among others.

For a comprehensive list of the various types of DevOps security best practices and tools that we should expect to see gain wider adoption, this October 2015 blog by noted security analyst Adrian Lane of Securosis serves as a basic framework. Each of the areas of focus that he cites clearly offers significant opportunity for both new and existing methodologies.

Finally, one need look no further than this year’s RSA Security Conference, to be held in San Francisco at this end of this month, as tacit proof of the continued growth of what people now varyingly refer to as DevSecOps or Rugged DevOps (I prefer the latter as it just sounds cooler).

In addition to multiple conference sessions, notably a “DevOps Throwdown” between security industry veterans Caleb Sima, Chris Wysopal and Gary McGraw, there’s also the 2nd Annual DevOps Connect: Rugged DevOps Edition. If last year’s event is any indication (standing room only, venue literally at max capacity), this daylong DevOps security track at RSA will be an even more high-profile element of the entire week.

Further, if DevOps security—or, at the very least, the growing focus on adjacent segments such as container security—isn’t the unofficial theme of this year’s entire RSA show, I’d be extremely surprised.

Meanwhile (self-serving), my employer CA Technologies absolutely has a lot going on that complements and contributes to this growing DevOps Security market momentum.

For all of those reasons, here I find myself again, drawn closer back into the security space that I’ve called home for roughly the last decade. I guess that’s just how it goes, as DevOps and security are inarguably two of the most significant trends in IT these days and inextricably linked at their respective cores.

There are definitely worse fates out there. But, just when you think you’re out …

February 19, 2016
Performance Testing is a Team Sport

Performance testing is a vital component of any developer’s working process. The performance of your website or app can be judged by a lot of people in your organization, both technical and non-technical.

Developers, operators and performance engineers are on the front lines when it comes to the development of a functioning site/app, but there will always be people away from engineering with comments, too.

“Can we make this faster?”

“Is the UI making this a little too slow?”

C-level execs, board members and marketers can often make the journey into becoming amateur web performance testing experts and critics, but that’s okay!

Web performance is widely accepted as a crucial factor in user experience—for example, when it comes to the e-commerce buyer’s journey. Your application better be fast, or you’re going to end up costing yourself conversions and money.

Instead of having these conversations after the development and testing has been done, we should definitely include all the stakeholders in the process from the beginning. Sound familiar? (ahem, DevOps, ahem)

Assemble the Team

Here are a few simple ways you can get everyone involved in your performance testing cycle:

Test Early

In the business of collecting data, getting a baseline is key. Test your minimum viable product (MVP) as aggressively as you would test your “final” product. If the MVP can’t stand up to some pressure early on, it might be in deep trouble moving forward.

Test Often—and Automate

The idea here is to create a performance trend to show if you’re moving in the right direction. More data points is always better.

The best way to generate many data points is to programmatically run tests after each code commit, either using continuous integration software or by having a post-commit hook or something similar in the source code control system that makes an API call to fire off a load test.

This way, you’re arming yourself with the right information to know when new code is impacting application performance, and you’ll know when to jump in and investigate changes right away.

Give People Access

We’re living in the era of transparency in software as a service (SaaS), so it’s always a good idea to give your coworkers access to your testing dashboard data. While some people in the organization might not wholly understand the technical side, it won’t be difficult for them to grasp that an increasing load time can be a bad trend.

There are all kinds of dashboard SaaS companies out there who can help you do this—assuming you want to stay away from pounding numbers into a spreadsheet or table every day.

At Load Impact, we’ve received a lot of feedback from engineers who were looking for an easy way to share test information with team members, and from that, we created the “Team Members” functionality.

Testers simply invite team members to their organization, and the new invitee can see past results and even create scenarios to be tested. That way, everyone’s in the loop, and everyone can get involved to the extent of their expertise.

Play Nice and Share

Continuous testing is a mindset as much as it’s a methodology. It fits neatly into the ideals of DevOps, and a big part of that is an inclusive, collaborative culture with your team members.

So remember, play nice with your colleagues—technical and non-technical—and you’ll create a productive, creative environment for everyone to thrive in.

February 9, 2016
Testing: Mind the DevOps Gap
If DevOps is a continuum, it starts with planning and development, and ends with deployment into operations. Much of the focus in DevOps and within the DevOps toolchain has been on application build automation (development) and deployment of apps into operations. But, for DevOps to pay off, there must be a continuous flow and automation from development all the way through to operations, and that means providing automation for the steps in between these two.

What is in the gap between development and production delivery? Testing. Not surprisingly, when Gartner surveyed its clients in 2015, more than 50 percent of them ranked the need for test automation as one of their top priorities for DevOps success. This really illustrates the recognition that DevOps cannot succeed if only one part of the DevOps continuum is automated. Companies that have started to focus on DevOps recognize that they must “Mind the Gap.”

Closing the Testing Gap

Why is automating the various testing steps (regression, performance, stress, GUI, QA, interoperability and security testing) so difficult? There are five key components that must be automated successfully:
1. The infrastructure – It starts with one very important characteristic critical to testing that is not automated – the ability to mimic the infrastructure in which the software (or hardware) being tested is expected to run in when it gets to production. Having the ability to give each tester (or set of automated tests) their own personal replica of the target production infrastructure—a “sandbox”—is critical to ensuring the software coming out of test can be deployed into production automatically and continuously. Automating tests is not nearly as difficult as automating the underlying infrastructure in which those tests run.
2. The production environment workload – Often overlooked in automating testing is mimicking the workloads that would be common in production. This includes the network traffic, other application workloads and network security profiles. These activities need to be automated within the sandbox to create a more realistic environment for testing.
3. Test automation – Of course, the tests themselves also must be automated and there are a variety of tools to assist in this process. As software gets more complex, automation continues to be a challenge, with mobile testing replacing GUI testing as the most challenging type of testing to automate.
4. Reporting and result analysis – To provide continuous automation from development through testing to delivery, it is important to see and analyze test results automatically.
5. Tool Integration – Finally, automation into and out of testing needs to be enabled. This means that sandboxes and test automation suites need to be API-driven so they can be started by a previous tool in the DevOps tool chain and can also initiate a tool that follows testing in the tool chain.
DevOps projects tend to start in one part of the continuum from development to operations. There is nothing wrong with taking this approach. However, it is also important to “Mind the Gap” and work to expand automation across the continuum continuously. One way to ensure that automation can expand easily is to deploy sandbox technology across the continuum, beginning with development and continuing through configuration and release management. The sandbox ensures that at each step, the production configuration and workload is being replicated.

Sandboxing technology has been used in development in the past, but is relatively new to the entire DevOps process. As more companies move to continuous integration and continuous deployment, however, the need for sandboxes is growing.

The sandbox should provide the following:
- The ability to create an accurate replica of the real production environment that deployment is intended for. This can include the physical, virtual and cloud infrastructure as well as applications, network traffic and load.
- The ability to initiate a sandbox via APIs so that the sandbox can be triggered by DevOps tools as an integral part of a complete end-to-end DevOps automation.
- The ability to automate DevOps processes inside of the sandbox (e.g. automated regression testing), as well as to automate the teardown of the sandbox and trigger the next step in the DevOps cycle when those processes complete.
- Business intelligence about everything that happens inside of sandboxes so that DevOps processes can focus on continuous improvement.
- The ability for many people to have their own sandboxes running simultaneously with isolation in shared labs.
DevOps depends on continuous automation from development, through test and QA, to configuration management and production deployment. Sandboxes provide a common automation environment that can be used at each step of the DevOps cycle to close the gap between development and deployment.

About the Author/Joan Wrabetz

Joan Wrabetz is the CTO for QualiSystems. Most recently, she was theVP and CTO for the Emerging Product Division of EMC. Joan received her MBA at the University of California, Berkley, and MSEE at Stanford, and a BSEE at Yale. She has held teaching positions at the University of St. Thomas, St. Mary’s University and the University of St. Thomas, St. Mary’s University. Joan holds patents in load balancing, distributed systems, and machine learning classification and analytics. Her experience inclues: founder and CEO of Aumni Data, CEO of Tricord Systems, Vice President and General Manager for SAN operations at StorageTek, Founder and CEO of Aggregate Computing, Management and senior technical positions at Control Data Corporation and SRI International, Partner with BlueStream Ventures.
January 28, 2016