devopsy.kinsta.cloud

Author: David Geer

SolarWinds Powers Instacart’s DevOps Efforts

Instacart is an Internet-based home delivery service for groceries. Its mobile app and website enables consumers to leverage its service, which operates in 17 cities across the country. Customers order through the app, and Instacart’s personal shoppers fulfill those orders at local stores. Forbes magazine named Instacart the Most Promising Company of 2015, citing its growth since the company’s 2012 inception, notes Nick Elser, director of Engineering at Instacart.

DevOps touches critical business applications at Instacart, including the apps and APIs for both iOS and Android mobile operating systems. It also supports back-end systems such as processes rooted in machine learning that assign grocery lists to shoppers and a pipeline that retrieves retail partner product data about available grocery items nightly.

Key Mobile Development Challenges & Goals

The Instacart DevOps shop is tasked with maintaining infrastructure in software, expanding the current business apps through innovations, and scoping and building future software for the company and its services. “We needed better infrastructure-monitoring solutions in place to save time and help scale parts of the infrastructure as the company continued to grow,” says Elser.

Elser and his team looked at SolarWinds‘ capabilities, and were enticed by the Librato cloud-based monitoring tool’s potential to safeguard the performance and uptime of the company’s mobile APIs. “Our customers demand a high level of reliability from their consumer services,” says Elser. “Librato ensures the reliability of our mobile APIs by keeping a close eye on the health of our internal/external services.”

Instacart hoped to use SolarWinds to support its infrastructure-scaling efforts. By enabling Instacart to leverage aggregates of logging data and statistics reports, Elser and his team could focus on creating an increasingly innovative and fruitful grocery delivery service. “We wanted to concentrate on scaling our operations, fleet of shoppers and metropolitan presence, as well as identifying processes that would enable us to move faster as a team,” explains Elser.

Applying SolarWinds Tools

With the software libraries they needed already present in the programming languages and environments they had on hand, implementing the SolarWinds tools to serve Instacart’s intended goals was relatively pain-free. One tool, Papertrail, required only a minor adjustment to existing config files to forward log records.

Papertrail log management tool eased logging duties for the entire Instacart infrastructure. “I could have Papertrail on all my boxes, give it an endpoint and it would be aggregated into different buckets based on host names,” says Elser.

Instacart also added SolarWinds’ Pingdom solution to notify the team about website downtime issues. This enabled the team to gauge real-world user experiences with uptime and to assess synthetic transactions for issues that could concern customers. “Pingdom was the most flexible uptime monitoring tool my team could find,” says Elser.

Librato also proved worth the effort, thanks to its many value-adds. “We received a lot of free advantages with Librato because it came with Amazon CloudWatch integration,” says Elser. This made it fairly trivial to add instrumentation to our apps.”

Using Librato ultimately enabled Instacart to create graphs to identify correlations in event data—instances where some relationship exists but falls short of cause and effect. Instacart could then use that data to pinpoint and resolve security breaches. “We also use the Librato CloudWatch integration to monitor database loads, as these affect the infrastructure’s health,” says Elser.

By integrating Librato with Heroku and AWS CloudWatch and applying other SolarWinds apps, Instacart gained visualizations of important infrastructure elements and transparency into its entire software stack for monitoring stack health.

Other Tools

Instacart also uses other tools to help its DevOps, including Crashlytics mobile crash and usage tracking application, Urban Airship mobile push notification tool, Branch.io deep-linking application and the Rollbar exception tracking tool.

January 18, 2016
Boston DevOps Leader Pete Cheslock Talks Accomplishments

Pete Cheslock began his leadership role in DevOps in 2009 at cloud-based email archiving company Sonian. The company was founded by friends and former colleagues of his, who called Cheslock for help when they got funding for the Sonian venture. He filled in wherever they needed him—as did everyone else in the first few years of the 10- to 20-person startup.

“As we grew, I saw a need for leadership for Sonian’s five-person operations team. I went to our VP of Engineering about the need for ownership of operations and said I’d love to be the one to give it a shot,” says Cheslock.

Sonian was an early user of Amazon Web Services, deploying production code with customers by 2008. Sonian had been named an AWS Startup Challenge finalist, but despite its forward-thinking approach to the cloud and infrastructure as code, many of its processes were immature.

“The application had scaling issues and we were spending an inordinate amount of time on reactive measures, dealing with application downtime and using short-term fixes for continuing issues when I took over,” says Cheslock. The young DevOps shop wasn’t able to make headway in innovative, future-looking projects.

Smothering Fires, Forging Ahead

Cheslock transitioned its team into two groups—a proactive group for innovative projects with a view to tomorrow, and a reactive team to get the worst recurring fires under control. Cheslock added himself to the latter group while remaining head of Operations. “While we attacked the hairiest issues, the proactive group worked on tooling that we needed in support of CD (continuous development) and CI (continuous integration), which made deployments much easier,” says Cheslock.

It had been taking the Sonian team about a week to deploy software for all its customer environments. The process of deployment affected much of the software stack; it required testing as well as completing and checking off several items on a lengthy deployment list.

Cheslock directed the proactive team to automate much of that checklist. Soon, they had reduced deployment time to 20 customer environments in just three hours, from five three-hour periods over five days.

Weeding Out Costly Cloud Issues

Cheslock and Sonian next began to work on cost issues. “We built an application to analyze our Amazon usage and correlate it with our Chef server, which was our single source of truth for all instances,” says Cheslock. The app revealed unused and abandoned servers that had been used for development and testing, which were still running and costing the company money.

“We found occasional spot instances where we could pay fractions of the retail price for batch workloads to cover these. We leveraged the Amazon Marketplace to make better purchases of infrastructure to save money,” says Cheslock.

Cheslock was able to shrink 800 instances on a single AWS cloud with a monthly spend of $450,000 down to 400 instances across four public cloud environments with a much smaller price tag of $200,000 a month. He accomplished this in three months, even as Sonian’s cloud operations team doubled.

Now senior director of Operations and Support at Threat Stack, Cheslock is nurturing DevOps methodologies and seeing continued growth and success there.

BosOps

A few years ago, Cheslock and a few friends started a Boston-area technical operations meetup called BosOps, inviting people who managed operations at various companies to get together regularly to discuss their challenges and offer support.

“As BosOps grew, we wanted to reach a broader audience and started pulling in more people in operations or DevOps roles to gather on a quarterly basis,” says Cheslock.

Currently, BosOps has about 150 participants, who also participate in online chats using the free tier of Slack.com, a popular team-working messaging app among DevOps folk. “It’s great for people who are new to Boston,” says Cheslock.

BosOps helps by quickly connecting a professional network for people who are new to the industry or to the area, enabling them to meet like-minded people and to secure a better foothold—perhaps even find a job they like more than their current position.

January 14, 2016
Jenkins Founder Kohsuke Kawaguchi Talks DevOps Leadership, Part 2

In part one of this two-part series, we covered the early work of Kohsuke Kawaguchi, founder of Jenkins while he was at Sun (now Oracle) and also how he addressed the project’s challenges to growth by applying extensibility/plugins, incentivizing sharing in the community, and removing barriers that can result from apprenticeships. Now let’s continue as Kawaguchi shares the rest of his story.

Explosive Community & Tooling Growth

Today the Jenkins Project and its open source community boast more than 120,000 members and 1,000 plugins. It was by design that Kawaguchi spurred that growth. Here are the challenges he faced, his solutions and his good fortune.

Though continuous integration (CI) and continuous delivery (CD) have sprouted an immense amount of automation in DevOps, it was necessary to have diversity in the software development environment/toolkit to ensure that each developer and team have the tools best-suited to their goals and uses.

“This is where our extensibility features came in handy—by letting people reuse what they could and build the missing pieces. As developers, we prefer flexible building blocks over rigid software that does things in just one way,” says Kawaguchi. This diversity encouraged community growth by removing a tooling stumbling block and brought even more plugins to life.

Jenkins emerged at a time when IT began embracing open source projects and, as a result, benefited from that momentum. “Companies recognize that it’s better in a lot of ways to rely on open source software than proprietary tools. It gives them more control,” he says.

Jenkins as a Key Component for Build Orchestration

The DevOps movement has helped Jenkins become a key component of helping teams orchestrate their builds. From its roots of using time-based activity triggers and requiring a developer complete “this task, then that one” in a cascading, triggering fashion, Jenkins has moved on to model what people do and then pull that work into Jenkins as automated functions.

The automation capabilities of Jenkins kept expanding and Jenkins entered what Kawaguchi calls the “orchestration-idioms-as-plugins” era, with different people in the community developing different patterns for orchestrating/automating activities. “I created one such plugin called the Promoted Builds plugin, which basically was a generalization of how different teams collaborated within Sun to release software,” says Kawaguchi. By creating these idioms within the model of executable plugins that others can adopt, the Jenkins Project fostered even more efficiency in sharing, adopting and reusing proven orchestrations.

As automation spread from builds to tests, deployments and releases, these orchestrations gained new importance but presented still another challenge, which a plugin also solved. “Idioms were useful but different idioms created by different people weren’t really composable,” Kawaguchi said. “In recognizing that, we’ve built the Workflow plugin, which creates a single framework to describe any orchestration, whether simple or complex. We can share and compose these fragments.”

Kawaguchi’s Leadership Accelerates DevOps Acceptance

As both a leader of the Jenkins open source community and the CTO of CloudBees, Kawaguchi has helped the open source community and enterprise teams alike wade in to DevOps. The open source Jenkins ecosystem helped to move automation ahead. It gave developers a fast track to useful, functional software that would please ops and enter production quickly. At the same time, it enabled developers to innovate swiftly and improve on that same software to see more utility and recurring rewards.

“Now with CloudBees, we have allowed the project to have a number of full-time engineers, which increased its velocity. It also brought commercial support, an enterprise version of Jenkins, professional services and training,” Kawaguchi notes. “It made Jenkins more viable in the mind of large-scale conservative enterprise users who value those things and allowed Jenkins to be deployed at scale in those organizations.”

January 11, 2016
Jenkins Founder Kohsuke Kawaguchi Talks DevOps Leadership, part 1

Kohsuke Kawaguchi, founder 0f open source continuous integration tool Jenkins, was an early leader in DevOps, guiding the development of the Jenkins Project during his time at Sun (and later, Oracle) and plotting the course it would follow.

“This project started as a hobby while I was at Sun,” Kawaguchi says. “From the project’s inception, it was not popular. During those early days, my leadership was really just to keep the project going. It was easy for me because the project was so useful that I didn’t need any external encouragement.”

Once Jenkins steadily gained acceptance, Kawaguchi worked on polishing it and making it more user-friendly. “Writing software that works for you is much easier than writing software that works for other people, especially first-time users,” he says, noting he drew from his experiences supporting users to make Jenkins a better tool.

Kawaguchi applied his previous open source project experience to make Jenkins an application/project that others could join, invest in and enjoy using. “As the project grew even more popular, I expended more conscious effort into those initiatives, specifically into extensibility and incentivizing sharing,” says Kawaguchi.

Kawaguchi’s Vision for Extending Jenkins & Incentivizing Sharing

Kawaguchi decided to use a plugin model for extensibility to solve the potential bottleneck of using an increasing number of contributors while managing change. Sensible software development does not permit unbridled change; developers must propose changes or at least innovate in an environment where they can retract changes that create issues.

To attract more contributors into Jenkins, Kawaguchi decided to let people add whatever they wanted to it, through plugins. “Selling me on a software change need only happen when a change affects my use, so if we can split things up such that your change is in your plugin, I’m not affected by your change so long as I don’t use your plugin,” explains Kawaguchi. “This enables developers in the Jenkins Project to go from idea to implementation without getting blocked by someone, which reduces organizational overhead and encourages innovation.”

But while this approach solves the extensibility issue, it also can create a new one. Without any incentive to share plugins with other users, extensibility can lead to siloed development in which only the group that created the plugin benefits from it. It then becomes difficult for the open source community to locate, leverage and improve on others’ advances, leading to plugin abandonment should the developer of that extension relocate to another project.

Kawaguchi pressed for an update center and set a rule that plugins listed in the update center must be available to the entire community. “This created a positive feedback loop that encourages users to install plugins through the update center and developers to bring their plugins to the community,” says Kawaguchi.

Breaking Down the Walls

While extensibility and sharing accomplished much, the Jenkins Project faced another hurdle. Other projects such as the Apache Project use apprenticeships (the “committerships”), an approach designed to create a stable of tried-and-true contributors. But it also can be a hurdle, discouraging many developers.

The Jenkins Project, however, simply asks that contributors apply good judgment when considering whether a plugin/change is good for the larger open source project and its community. “This might sound scary to some, but it really strikes the right balance between the speed of progress and quality,” says Kawaguchi. “Besides, everything is version-controlled, so if we later find a problem, we can always roll things back.”

Stay tuned to devopsy.kinsta.cloud for the rest of Kawaguchi’s DevOps journey with Jenkins.

January 4, 2016
DevOps Leadership: Leading a DevOps Revolution in Pay TV

The State of DevOps in Pay TV

Pay TV is largely rooted in an ancient operator model. “Operators in this industry usually take a year or longer to release new code into software. Their Ops teams are focused on making sure that existing environments never fail,” says Mark Hydar, Head of DevOps, Ericsson TV Platforms. That is as opposed to moving to the CI and CD of DevOps with their continually changing environments.

Newer Pay TV models that revolve around sending TV shows to a web browser in exchange for a modest monthly subscription arrived with DevOps and evolve faster because of it. Older models where you offer a set top box for a consumer’s home TV are still transforming sluggishly, taking small bites at adopting DevOps.

“An operator such as a Vodafone has customers who pay a lot of money for the reliability that comes with the set top box model. It’s on a big screen, the customer uses a remote, and they demand 4K and to be able to connect it to their home audio system. The expectation with that is that it never fails,” says Hydar. Environments of constant change such as DevOps where the perception is that the risk of failure is high present a lot of apparent / assumed conflict for operators and ops teams supporting the legacy model.

Moving Operators to DevOps

These operators need a guiding hand as they take baby steps toward fast and ferocious innovation and improvement. “Our approach is to first educate the operators’ directors and above on what the consumer tells us they want. Our field research and R&D tell us that by 2020, there will be a lot more connected handsets and smart devices out there,” says Hydar.

The current generation, the Millennials, have “surpassed the Gen-Xers as the largest generation in the U.S. labor force”. This generation is spending less time in front of a fixed screen and a lot more time with mobile screens, says Hydar; operators need to figure out what they can do to stay aligned with what the current consumer wants while protecting and growing the revenue they currently have with “cord cutters”.

Ericsson next works to help operators roll innovations out to customers faster by adopting Agile methodologies and moving to a DevOps deployment model while learning how to sustain that model, according to Hydar. But this transition only happens as Ericsson aids its operator customers in dealing with an expected level of internal apprehension and hesitation by bringing each internal group along a bit at a time.

For example, Ericsson will recommend that operators and their CIOs take existing data center engineers who address trouble tickets and retool their skillsets so that they can look at software bugs and start to think more like developers.

“Once we suggest that, the CIO will often feel like, wow, and then there will be some push back, some resistance, and what-if questions. Over a period of a month or two, there will be a high volume of pointed questions that come to us, but that goes away,” says Hydar. Operator leadership eventually wraps their heads around it, see the benefits, and translate the changes to a lower cost of ownership, says Hydar.

As for the operations side who are still keeping the status quo up and running, Ericsson simply informs them as to what is coming down the pike in a very positive way. “We tell them how it is going to make their life better and how they can accomplish more,” says Hydar.

When approaching the test and development teams, Ericsson focuses on how they test and roll software today, introduces them to how they can innovate and deliver software faster, and leads them to want to align with that process. “They start to see the benefits and gain momentum both on the test and development sides,” says Hydar. Development begins to change.

Finally, Ericsson goes back to the Ops teams and brings them together in summits where they discuss the speed of delivery of some the software projects, and the test findings. “The Ops team starts to take notice. They want to learn about Agile. That’s where you introduce training in what it takes to be a Scrum master, what DevOps means, and how they as operations people can integrate themselves with development,” says Hydar.

December 1, 2015
BoomTown No Bust with VM Supported Parallel Testing Environment

An Introduction to BoomTown DevOps Processes

BoomTown uses a specific set of DevOps processes in order to deliver websites with the latest real estate listings and housing data as well as a CRM solution and analytics tools for real estate professionals. “Our development, operations, and QA teams automate everything in sight while offloading to cloud providers and other services the tasks that those providers are best suited to,” says Brian Baumgartner, QA Manager, BoomTown.

As only one example, BoomTown recently adopted Amazon Web Services (AWS) for hosting various aspects of its infrastructure. “It enables us to provide high availability and resilient service while allowing our development team to focus on revolutionizing the Real Estate software industry,” says Baumgartner.

Docker containers support regular deployment processes at BoomTown, containerizing the environment setup for a more reliable, orderly deployment experience, whether the firm is pushing code to QA or to production, explains Baumgartner. TeamCity, the increasingly popular and frequently appearing name in build management tools and CI servers courtesy of JetBrains is BoomTown’s go to build platform. New Relic is BoomTown’s performance management solution of choice, enabling live transparency into the production application and server performance that keeps BoomTown citizens (customers) content.

BoomTown Before & After Using Sauce Labs VMs to Scale Testing

Prior to applying Sauce Labs, BoomTown’s QA analyst team processed all software tests exclusively in a manual fashion. Every test run crossed their laps / machines / laptops. “Before a new feature release, an analyst would execute a full set of regression tests against the product or feature, often through several different browsers and versions to verify the new release and its function. This process alone took up to two weeks for large projects,” says Baumgartner.

BoomTown now uses a test automation framework constructed using several popular DevOps tools. The custom built testing automation leverages Ruby, Selenium WebDriver, Cucumber, MySQL, Sinatra, and Angular JS, says Joey Gryder, Senior Software Engineer in Test, BoomTown. The framework offers BoomTown views into historical and real-time test results as well as the capability to run manual tests through its dashboard front end software connection to its backend systems.

BoomTown applies a single web service to assert control and manipulation of test system automation features. A “test runner” library enables QA to run Cucumber features in a parallel execution approach. BoomTown uses MySQL to store test results. “When we combine all this with Sauce Labs’ VMs, we can schedule nightly test runs using our full testing suite on a variety of browser configurations,” says Baumgartner. This has enabled BoomTown to run up to 100 tests on software nightly.

Parallelization

Where time, resource, and person hours’ savings are concerned, the secret sauce in BoomTown’s software testing framework is its use of Sauce Labs VMs to support many automated tests in parallel. This has cut BoomTown’s longest test run time of an hour and a half down to less than 15 minutes.

“With Sauce Labs, it is trivial to spin up any number of parallel VMs, configured against your desired browsers to run through a set of Selenium WebDriver test steps. Without such a solution, BoomTown’s test team would have had to either execute its entire test suite locally through a single machine and browser, or build an in-house testing lab to achieve parallel browser testing,” explains Baumgartner. (Sauce Labs is obviously not the only player with competitors including BrowserStack and TestingBot; let your due diligence be your guide in selecting the best service for your needs.)

BoomTown saw offloading the work as the best option in order to accelerate otherwise time intensive testing tasks, preventing QA from becoming a sort of time-related bottleneck on CD. “We did this simply by increasing our VM count and running more tests in parallel,” says Baumgartner.

Adding VMs and increasing the number of parallel testing instances is an easily scaled approach that continues to speed tests and drive CD even as BoomTown adds tests and lengthens total test runtime.

Using automated parallel tests supported by Sauce Labs VMs frees up QA analysts to be more creative in their testing, finding more and more interesting ways to break the application under testing loads, says Baumgartner; meanwhile the automated testing ‘machine’ verifies the mundane and repetitive tests that the team has already established.

“We recently incorporated automation as a quality gate in our CI process. Before a pull request is accepted, it must pass a set of critical path UI tests that run up in Sauce Labs. This sets a certain level of quality expectations during our daily development process, which saves us time in testing,” explains Gryder.

November 19, 2015
Meeting Challenges in incorporating automated testing throughout the DevOps cycle

Enterprise development teams find it difficult to perform QA tests on software in short DevOps cycles where they cannot squeeze in lengthy regression tests. DevOps teams must insert automated testing approaches to resolve this predicament. “DevOps teams need to be more selective in prioritizing tests that occur at each stage of the integration to delivery pipeline,” says Marc Hornbeek, Senior Solutions Architect of DevOps, Spirent.

At each point between stages of the development lifecycle, there is an opportunity to insert the most applicable tests to determine whether the software has incurred errors. It is crucial to test early. The earlier in the development process that team members discover their errors, the sooner they can fix them, ensuring fewer error-related issues in releases. The earlier in the development lifecycle that errors occur and go unchecked, the more opportunity they have to grow and become increasingly debilitating to the software.

With best practices, DevOps shops can use the right tests in the right stages of development and achieve quality assurance without depending on large test suites that run at the end of the cycle, according to Hornbeek.

The best time to create automated tests is together with or prior to developing the code for the given software feature itself. This will enable development shops to keep pace with the speed of DevOps; the old approach of automating testing after the feature is developed has truly become ineffective, according to Hornbeek.

DevOps Targets Quality / Requires Testing

DevOps has the goal of creating better software faster. Apart from effective, continuous, automated testing at keenly selected intervals, DevOps is only faster. This would achieve only a form of dev similar to what we had before DevOps, i.e., the kind where coders believe they have done well even though operations staff sees a lack of fitness of the software for production use.

Enterprises must embed a culture of QA inside the culture of DevOps rather than separate out QA testing from DevOps using organizational silos; communicating only through inefficient bug reporting methods is no longer the state-of-the-art, let alone practical with the increasing industry expectations of DevOps.

Testing: Why Must I Be Misunderstood?

The impact of testing is often overlooked. DevOps shops that take testing for granted may not invest in or improve upon testing as testing needs change and grow. “This was true before DevOps and continues to be true with DevOps. Developers focus on feature creation, while the business leaders focus on delivery,” says Hornbeek. As a result, testing is often not a focus of DevOps.

Further, there is a common misconception that testing just the part that is done after the builds are created is all that is necessary; but enterprises must integrate continuous automated testing with DevOps throughout the development-to-delivery pipeline, according to Hornbeek.

“When testing is conducted a long time after the feature is developed, there are many inefficiencies. The developers lose context of the problem and have a harder time recreating, diagnosing, and fixing problems,” says Hornbeek. Automated continuous testing eliminates this delay, improves software, and shrinks time to quality software releases.

Slothful Testing Gone Awry

There are high-profile examples of slothful software testing where projects grew out of scope and way off target. A chief example is the original rollout of Healthcare.gov, says Hornbeek. “In another example, a network equipment manufacturer tried to implement DevOps twice without success. They finally realized that they had underestimated continuous testing,” says Hornbeek of a Spirent customer.

To prevent these kinds of outcomes, says Hornbeek, follow these best practices:

• Create automated tests as you develop the features.
• Do not accept features into the integration branch without automated tests.
• Ensure that the developer test environment is a good representation of the production environment.
• Select DevOps-ready tools that have Rest APIs.
• Test priority features so you can easily integrate them into the DevOps orchestration framework.

Follow these steps, according to Hornbeek, to incorporate automated testing for QA:

• Establish a clear leader with responsibility over the entire dev-to-deliver pipeline who is visible and will champion continuous testing.
• Assign an architect for continuous testing tools selection and to oversee the technical implementations.
• Determine best practices for Continuous Testing and decide what practices best fit the organization.
• Address the culture, roles, and responsibilities for continuous testing.
• Put some automated metrics in place, such as a goal to automate 85% of all tests.

Editors Note: I had a chance to catch up with Mark Hornbeek at the Enterprise Summit 2015. Here is my interview with Mark.

November 11, 2015
Ancestry.com Brings BlazeMeter into the Fold

In order to gather and organize your lineage and the branches in your family tree, Ancestry has spent more than a generation (the company was born in 1983) wading through the 16B+ images and records buried in historical society data, government archives, and information caches of like organizations, groups, and communities the world over.

Data in hand, Ancestry transcribed it and converted it into a searchable format for your investigation so you can connect the dots from great, great aunt Tildy to great, great, great grandpa Jebediah. “These historical records date back to the 11th century and form more than 30K collections that include birth, marriage, and death records as well as military service, immigration, and even incarceration records,” says Norman Warren of the Ancestry Performance Team, Ancestry.

Ancestry’s platforms enable it to unite descendants and the curious with true tales from 70M family trees that subscribers build up by adding their own branches. Interestingly, Ancestry data now relies on a million plus DNA samples and counting.

“Our servers respond to more than 75M searches per day,” says Warren, who coaches Ancestry’s teams in refining load testing practices as they release new testing scripts before and after sprints and based on their individual development timing.

Load Testing Before & After

When the Ancestry team went looking for deeper, broader load testing capabilities, they were hoping to add routine testing across the site 365-days a year. The folks at Ancestry wanted a malleable solution that would invite them to apply JMeter to vertical load testing and permit them to cut costs for load testing by leveraging the advantages of open source testing tools.

Ancestry had been using a load testing approach that did not allow Warren’s teams to use home-brewed scripts. But in order to benefit from a testing solution that did embrace such scripts, Ancestry had to ensure a flow of updated quality scripts from every applicable internal group. Because Ancestry ultimately opted for JMeter and BlazeMeter, that presented challenges such as JMeter and BlazeMeter learning curves, the need for users to make JMeter requests in parallel, and Ancestry apps that are harder to script.

Growing pains accompanied BlazeMeter adoption including the pains of stretching to embrace different devops teams in order to retrieve testing scripts. By pointing out the advantages of the new testing paradigm, making the argument for time saved by site wide load testing, and providing training in JMeter, Ancestry made the leap to the BlazeMeter supported approach. “BlazeMeter provided excellent JMeter training while we evangelized for teams to own their scripts,” says Warren.

Ancestry solved its technical and learning curve challenges with steps including installing the JMeter Webdriver Set plugin and learning to debug and test scripts before they would run optimally in the BlazeMeter cloud service.

“We had to work to get adoption, but it was well worth the effort to build accountability over different sections of the site. We now ask teams to take ownership of their own scripts to review, modify, or emulate user traffic flows,” says Warren.

Results

“BlazeMeter enabled more frequent load testing at a competitive cost and good site-wide load testing with the ability to add loads to stacks or to stop loads on one of the stacks during the test,” says Warren.

BlazeMeter helped the Ancestry teams to foster a strong confidence that they could support heavy traffic loads at their most hectic times. “The BlazeMeter team helped us to translate the metrics we use for measuring traffic into a number of concurrent users for loading the site. In this way, BlazeMeter helped us to measure load test results against a baseline, such as traffic on our most recently busiest day,” says Warren.

Ancestry leveraged BlazeMeter’s swift feedback about site throughput and the response times of each page to speed site responsiveness for each user. Ancestry was able to predict and automate the flows that are most commonly used by customers in order to streamline processes and speed the user experience even more.

“We created a new baseline from new site wide load tests of additional traffic flows by loading flows incrementally using a special BlazeMeter feature that enables us to dynamically add more concurrent users in the midst of a load test,” explains Warren.

November 6, 2015
Avik Partner’s Grok APM Uses Machine Learning to Find Anomalies Early On

Machine Intelligence-Driven Anomaly Detection

Machine learning—the science of getting computers to take thoughtful action without explicit programming—is the basis of Machine Intelligence-Driven Anomaly Detection, which is the core capability in Avik Partner’s Grok APM product, according to Casey Kindiger, veteran enterprise software solutions provider, IT automation expert, and founder of Avik Partners.

“Thoughtful action means using the type of faculties that a human being would use to make a decision and to take an action,” says Kindiger, who has more than a decade of experience in designing and developing process automation solutions for companies such as JP Morgan Chase, T-Mobile, and Allstate Insurance. Grok learns from your dev and ops environments and takes thoughtful action to detect anomalies, all in the service of Application Performance Management.

Grok works by analyzing real-time performance data from apps and infrastructure, looking for behavioral bumps in an otherwise smoothly-paved DevOps highway of deployed services and underlying infrastructure.

Grok’s Pre-Event Inferences

“Grok uses common statistical techniques to learn a pattern over a selected period of time in order to build a dynamic threshold of the continuously changing underlying patterns,” says Kindiger. In this way, Grok can establish a sort of dynamic baseline from very noisy patterns, which is something that would typically take a number of person hours of involved statistical and predictive analytics to accomplish.

Grok’s machine intelligence leverages the ability to make inferences based on this dynamic baseline information like a human being would in order to reason out a logical conclusion. Grok does this to detect the slightest of anomalies based on the difference between what reasonably should have happened and what actually did happen. Using this approach, Grok detects subtle patterns of significance in streaming data despite the fact that the data has patterns that are continuously in flux.

But rather than detect when something has gone wrong at a point in time, which is not scalable or sustainable, Grok looks for minute patterns that appear well in advance of any issue that you would normally consider looking at with traditional APM mechanisms, explains Kindiger. Rather than send alerts based on threshold-based exceptions, Grok makes a prediction on what the next data point or the next series of data points will be and then looks at deviations from those predictions, considering whether to flag those as anomalies, according to Kindiger.

Currently, Grok is particularly tuned to analyze streaming data and is, according to Kindiger, data-point agnostic. “We don’t need to teach the system what the context of a particular data point is. The system learns the context of the data over time and the subtle nuances of the pattern that produces over time, and creates the model based on that,” says Kindiger.

Watch Out for That Initial Two Week Delay

If Grok does everything that Avik Partners’ Kindiger hypes that it will, DevOps teams will soon be able to catch and act on infrastructure issues before they grow big enough for end-users to notice.

But Grok does have one wrinkle in its APM armor: it takes time for it to really learn a new customer system and get good at detection. Let’s assume that Grok’s machine learning causes it to get better at detecting anomalies in a given set of infrastructure over time. That means that there’s a point, at the beginning, given a brand new customer when Grok only starts to learn. At that point, Grok isn’t very good at knowing what’s anomalous compared to what’s normal for that infrastructure because Grok has no foundation for that particular system.

“Grok will take some time to learn the nuances of when to escalate an issue. We’ve learned from experimentation that it gets pretty good after ingesting two weeks’ worth of data,” says Kindiger. Because Avik Partners initially targeted AWS, it has taken two weeks of CloudWatch data and fed that to Grok to provide it with some backdrop so that when Grok goes live, it won’t be learning this environment entirely anew.

November 3, 2015
APM Prepares Prep Sportswear for Promising Growth

Prep Sportswear kicked off in 2003 and appeared either in the Inc. 500 or Inc. 5000 list of fastest-growing private U.S. companies six years in a row, from 2009- 2014, according to online data from Inc. Magazine. The custom sportswear vendor has achieved 130-percent growth over the past three years, says Mike Jones, Director of Technology, Prep Sportswear.

Prep Sportswear clothes more than 200K US schools including most middle schools and high schools with custom-printed sportswear for their school sports teams. Prep Sportswear maintains a catalog of each school’s logos so that teams can create their own shirt or jersey representing that institution and team using its colors and logo or mascot. “So you can create a jersey and for me it would be for the class of 1986, Lake Washington High School, and you would see that Kangaroo on the back or on the front,” says Jones.

The custom-printed sportswear firm serves local teams not affiliated with schools as well as golf courses, colleges, and religious groups. Each customer gets its own store, and Prep Sportswear now maintains more than 2M stores.

Prep Sportswear: Small Team, Big Technology Play

Prep Sportswear produces its own sports goods and maintains manufacturing and attendant technologies while also serving ties to firms such as Active Network and others that use its API.

“We support retail customers, consumers, and our own plants and equipment. On the technology side, we do that with a very small team of ten people. They cover the e-commerce site, our API, and manufacturing needs,” says Jones.

Prep Sportswear programs its codebase and platform using C# and .NET. It also provides for manufacturing components such as ink and vinyl printers and embroidery hardware. Prep Sportswear manages its supply-chain and its related processes and mechanics. “We do it all in-house despite the fact that it is such a large system with a number of components,” says Jones.

How Prep Propped Up DevOps

When Michael Jones arrived at the company, the CTO was deploying releases directly to production from his laptop. “We didn’t have a DevOps team really until Richard came on board,” says Jones, speaking of his colleague Richard Dominguez, Developer in Operations, Prep Sportswear; “before that we were running in start-up mode, just kind of out of a garage until recently.” Richard describes the technical setting when he came on board.

“A lot of the automation they had was not really automation but little scripts that they ran individually,” says Richard Dominguez, Developer in Operations, Prep Sportswear; “when we released something, we had no idea whether it would break.” At the time, there were few tests on software and there were no external tests at all.

The Arrival of Change

With Richard at hand, Prep Sportswear ultimately adopted TeamCity for CI for deploying to its test and production development stages.

The company also added the Selenium browser-based test suite, coded in Java to run the user-based interactions to determine what the user experience is like. “It checks to see that the page and elements load properly when a user clicks a link,” explains Dominguez.

Richard found Dynatrace installed when he arrived but not in use. He revived it as the main APM for front-end monitoring. “We also use another tool a lot for our internal monitoring of infrastructure on the back-end and that tool is PRTG,” says Dominguez.

DevOps Done Right Brings Cheers from the Sidelines

The combined revenue from all channels last year was about $23M, says Jones. “Since Richard came on, we’ve seen a decrease in time to resolution of more than 50-percent just by having the APM and other tools,” says Jones, with reference to Richard’s revival of Dynatrace and addition of the other aforementioned tools.

Up times have improved by .50-percent as well. “That stems from simply being able to find an issue and resolve it faster. For an e-commerce site where we don’t have any sales people, it’s just the website, that additional up time is very important to us,” says Jones.

October 29, 2015