devopsy.kinsta.cloud

Author: David Geer

A DevOps Metamorphosis Echoes Through Reverb.com, Part Two

Last time, Dear Reader, we discovered how the Reverb.com e-commerce platform, which connects musicians who want to buy and sell gear sought to automate its workflow software development process, speeding deployment and bug fixes while satisfying customers who want to trade in an online environment that is as efficient and error-free as possible. Now, the conclusion of how Reverb.com adopted DevOps to achieve its goals.

A Round Trip Through The Reverb.com CD Process

Today, Reverb.com focuses the DevOps mindset on its development process using Chef [https://www.chef.io/chef/] for configuration management and Jenkins [https://www.cloudbees.com/jenkins/about] for continuous delivery and infrastructure automation tasks. Developers use Capistrano [http://capistranorb.com/] for rolling out new code changes and running database migrations, CircleCI [https://circleci.com/] for unit testing code, and DataDog [https://www.datadoghq.com/] for metrics and monitoring. “We also use Logstash [http://logstash.net/], ElasticSearch [http://www.elasticsearch.com/about/], and Kibana in unison for log aggregation and visualization,” says Adam Enger, Senior Infrastructure Engineer, Reverb.com.

In a typical deployment for Reverb.com, an engineer clicks the build button in Jenkins, which triggers a Chef converge on all the web heads. This rolls out any configuration changes that the developers have scheduled to go out, such as NGINX configuration updates or app config updates.

Once this is finished, the system signals the application to perform a Zero Downtime Deploy in which the application reloads without dropping any requests. “If this process finishes successfully, Jenkins kicks off a code deploy using Capistrano and simultaneously deploys new code to our web application servers. If a database migration is scheduled to be run, that will also happen on one of our worker boxes,” says Enger.

This may sound like a standard deployment so far. But that’s where DataDog and the ELK stack come into play. “That’s where we really gain the benefits of CD/CI and DevOps,” says Enger. Every time the development team converges Chef, they use a Chef Handler to submit an event to DataDog. And once Capistrano starts and finishes, another event goes to DataDog. The team has configured Logstash to watch the application logs on the web heads and to send an event to DataDog once the system renders an error. “The single point of information now becomes DataDog because we can see into our config and code changes and immediately know when a bad change goes out the door,” says Enger.

The last thing the development team looks at is the performance logs in Kibana, to see how the application is performing based on the logs it is generating. “With the ELK stack in place, the team can answer questions like: Are there any 500 errors? If so, how many and how often are they occurring? Did development increase or decrease application response times? Do people like the new feature rolled out at /my/feature?,” says Enger.

Hurdles

The biggest hurdle in moving to DevOps is the mental shift you have to make from that of a traditional systems administrator or developer to a more DevOps-based mindset. This means the operations side learns coding and development learns about load balancers.

“If we propagate continuous change in our computer systems, we should also expect to be continuously changing our mental models to adapt to new ways of doing things,” says Enger.

Benefits You Can See Through

Transparency is the number one thing the Reverb.com developers gained from this new CD approach. “Capistrano tells HipChat [https://www.hipchat.com/], “Hey, code is going out!” and I know to pull up my dashboards in Kibana and DataDog to watch for any anomalies,” says Enger. Humans can’t always be expected to communicate to each other efficiently. Computers help the Reverb.com team communicate across the organization through programmed notifications as to when changes go out.

Code deploys are now predictable and automated. No more worrying about a failed deploy due to a bad connection. “When failure does occur, we are able to open up the Jenkins Build output and find the error pretty quickly,” says Enger.

A Nice Surprise

One really positive result that surprised the development team after implementing CD/DevOps is that they learned a lot about their user behavior and traffic patterns. “Now we know that 4.30am is the slowest period on the site,” says Enger.

The team knows exactly when is the best time to roll out big configuration or code changes. They know immediately how deployments affect users and especially whether they ever break critical paths in the site, such as the checkout or signup functions. And the automated approach that DevOps provides enables them to fix bad changes within minutes of their original deployment.

The Affirming Roar of Fan Feedback

Reverb.com recently introduced more automation and monitoring to prevent site downtime and abuse by bots. The feedback: Customers don’t care about DevOps; they care about sites being operational and efficient. But it’s DevOps that makes that happen!

February 2, 2015
A DevOps Metamorphosis Echoes Through Reverb.com, Part One

“Reverb.com is an e-commerce platform that connects musicians, enabling them to buy and sell used gear and instruments,” says Adam Enger, Senior Infrastructure Engineer, Reverb.com. As an ecommerce site, it is critical to Reverb.com that its workflows such as checkout, offers to purchase, and messaging work in tandem with its selling workflows, which include creating new listings and managing orders.

The Reverb.com team also develops a free, comprehensive online price guide and a complex in-house content and curation system that allows them to hand pick and recommend the best gear for their customers. In the Reverb.com marketplace, all software components must run smoothly for customers to have the optimum research, buying, and selling experience when conducting trade on musical equipment.

To create those components, Reverb.com developers code primarily in Ruby on Rails on Macs. The etailer runs Linux in its production environment. “Our backend stack includes PostgreSQL for our database, ElasticSearch powering our search engine, and Redis for queueing and key value storage,” says Enger.

Not So Good, Old-Time Deployment

Prior to DevOps approaches, Reverb.com deployed code from individual user workstations. “For configuration changes to our infrastructure we had to use SSH to connect into production. All while our fingers were crossed, of course,” says Enger. This loose-knit deployment process presented Reverb.com with some obvious challenges.

“If a deploy failed due to an engineer being on a bad internet connection, we could potentially have had two updated hosts while three were still running an older version of code,” says Enger. This was especially problematic because these deployments often included database migrations.

There was also no real way to know when a deployment failed or who started the deployment. “The process was devoid of any deployment log records, so errors and anomalies were harder to spot when the deployment was swallowed in the developer’s terminal,” says Enger.

These risks made it possible for Reverb.com to face an even bigger risk: a lapse in faith on the part of its customers. Reverb.com’s reputation as a marketplace hinges on the perception of trust among its customers. Every aspect of the organization generates trust. From how Reverb.com interacts with customers as people, to how its systems present to users, the firm wants to exude confidence. “If we experience outages and slow performance, we may erode our trust with our audience,” says Enger.

Goals For DevOps

Reverb.com had many goals for DevOps in the hope that it would meet their challenges to fast, scalable, reliable delivery of code and configuration changes. Reverb.com needed to know when a change was going out. The firm needed to be able to deploy at a moment’s notice.

“All processes must be behind the click of a button or an SCM post/pre trigger,” says Enger. All Reverb.com deploys must be logged and graphed. Scaling the team and the business should become easier if the teams have quick, repeatable processes that allow them to push new features and bug fixes quickly onto the new DevOps platform. This should allow for a scalable and dynamic workflow for the Reverb.com marketplace so it can be nimble, adapting to the users’ needs as they arise.

Reverb.com’s Reasoning: An Automation Proclamation

Reverb.com made its move to DevOps on the basis that the more the automation, the less the likelihood of human error. From that, Reverb.com development staff should create fewer issues for its customer base as consumers browse the website, make purchases, and sell hardware.

“It’s extremely important to look at root causes here: if we only treat the symptoms every time the site slows or goes down, we will be playing Whac-a-Mole forever,” says Enger. By using the DevOps mindset, initiating repeatable and automated deploys, and measuring performance and application metrics in rea-time, Reverb.com developers should be able to trace when something was released, how it affected users, and whether customers are having a poor experience as a result.

In the hopes that this would equip them to quickly pinpoint issues and fix root causes, the band of coders at Reverb.com set out on the road to DevOps, fired off its first continuous delivery power chord, and awaited feedback.

Conclusion: Cliffhanger

Tune in next time when devopsy.kinsta.cloud reveals how Reverb.com adopted DevOps and used it to meet these development challenges and goals in the exciting conclusion of “A DevOps Metamorphosis Echoes Through Reverb.com”.

January 28, 2015
DevOps Transforms Copyright Clearance Center

Copyright Clearance Center (CCC, www.copyright.com) works primarily with business, academic, and medical institutions, enabling them to access the vital content that supports their work. CCC manages fee payments to the license holders and content creators who own the content. DevOps approaches have transformed CCC, a global leader in content rights-licensing solutions, resulting in expanded service to an increasing market.

“In production, CCC is already processing millions of dollars in transactions. This came to life within a year of the inception date for the DevOps project. For a startup that would be fast. For a medium-sized enterprise, I think that is outstanding,” says Babis Marmanis, CTO & VP of Engineering, Copyright Clearance Center.

The DevOps transformation that took place in the internal structure of CCC’s engineering department continues today, allowing the organization to respond to the massive disruption in the publishing industry, which is due in part to digital content and the Internet. Using DevOps, Marmanis and CCC have accelerated product development, sped time to market, and provided a stronger consumer experience with the product.

Accelerated Product Development

“The Copyright Clearance Center vision is to create the most comprehensive portfolio of global licensing and content solutions (products) that make copyright work for everyone. My product platform strategy stems from that,” says Marmanis.

CCC has multiple software-based products, each performing the same or similar functions and acting on the same or similar data. “There are certain fundamental things that we must abstract appropriately and encapsulate into common components to make development work optimally,” says Marmanis.

But when Marmanis came to CCC, there were no common components. And what is more the internal systems lacked interoperability, skills were different across internal teams, and there was no cohesiveness to the development approach. These factors slowed software product development.

A DevOps mindset was crucial to changing the status quo, to treating infrastructure as code, and to bringing in automation, says Marmanis. A DevOps approach would help to speed the delivery of software projects into the marketplace.

“Now we use Jenkins (www.cloudbees.com) to glue together elements of the continuous integration process. We have Jenkins build jobs that retrieve the source code from GitHub (https://github.com). We publish software artifacts to Artifactory (www.jfrog.com),” says Marmanis.

For each release, Marmanis’s teams use Jenkins to execute Chef (www.chef.io) recipes. They deploy artifacts from Artifactory onto a pre-specified execution environment. “In addition, and this is key to the overall scheme, we execute the schema updates with Liquibase (www.liquibase.org), and everything is coordinated with Jenkins,” says Marmanis.

Further, CCC built its cornerstone platform from the ground up using DevOps-related tools, build detection software, and software libraries. The infrastructure that CCC added is based on best-of-breed open source libraries.

“Using programming languages such as Java (www.java.com) and Groovy (http://groovy.codehaus.org), Gradle (www.gradle.org) scripts, postgreSQL as the relational database, Liquibase for database objects, Tomcat (http://tomcat.apache.org) as a servlet container, Apache Camel (http://camel.apache.org) for messaging and enterprise integration, and countless other tools including Apache Directory Studio (http://directory.apache.org/studio) and Bootstrap (http://getbootstrap.com/)—the list is very long—we were able to create a harmonious ecosystem,” says Marmanis.

Stronger Consumer Product Experience

CCC offers its consumers a better product experience through DevOps approaches and through a homogeneity of standards and technologies, which is available when using open source tools.

“When we’re building the product platform, these initiatives allow us to embody timeless design principles such as crisp abstractions, but also allow us to scale in a number of dimensions including the number of works that we can represent, the number of publishers we can admit and manage, the number of customers that we have, the number of applications that we can build and deploy, and the number of market verticals that we can occupy,” says Marmanis. This is crucial to customer satisfaction.

One product that is a prime example of how CCC keeps customers happy is RightsLink for Open Access. “Using RightsLink for Open Access, we facilitate fee collection for articles published under an Open Access license, mediating between the publisher and the author to collect those fees,” says Marmanis. Then CCC aggregates those transactions and presents them to the publisher.

“Obviously, we give the money to the publisher, but we also create reporting for the authors, publishers, and institutions as well as the founding agencies that gave the money to the author or the institution respectively for funding that research,” says Marmanis. The RightsLink for Open Access product keeps transactions simple, integrating with publisher workflows to expedite publication.

January 6, 2015
How DevOps Eases The Holiday Shopping Experience

“I have been building retail software for 15-years,” says Harper Reed, CTO, Modest, CTO emeritus, Threadless, and former CTO, Obama for America. “With the advent of DevOps, development is now about how we can make the online experience better for the customer,” says Reed. It used to be about taking care of the systems.

Today, the infrastructure is in software with Amazon and the Cloud, with what DigitalOcean is doing with Docker, and with what CoreOS is doing with server deployments. Before cloud-based infrastructure, the enterprise used the data center, which was a fortress that operations had to protect. It wasn’t as easy to think about the customer with those kinds of concerns to look out for. Now everyone has time to think about the customer.

“With the cloud, you can start to have the same lifecycle with infrastructure that you had with the software that actually ran your ecommerce,” says Reed. Teams can leverage a uniformity of tools for measuring performance. Because everyone is using the same tools, it becomes easier, more uniform, to test the software. “We can all be thinking about the user,” says Reed.

With DevOps, infrastructure in software is making the online Holiday shopping experience easier by scaling up in the cloud using very simple scaling rules. “In 2007, 2008, I was CTO of Threadless and we had very good software, but we had to plan months ahead in order to scale things up during the Holidays,” says Reed; “today we don’t have to think, what is our capacity going to be, how am I going to add 15 more servers? We just turn up the budget on Amazon and away it goes.” Of course, scaling keeps shoppers from experiencing server time-outs and slothful transactions.

Because scaling is already addressed, teams can focus on development efforts on the front end, on making the user experience better. The time teams would have spent scaling the back end they now spend ridding bugs and adding and improving services and features, and doing it all much more quickly thanks to continuous deployment.

As a result of a stronger commitment on the front end, efforts become possible such as you see today where stores like Everlane , an online clothing retailer has partnered with Postmates, an on-demand delivery service to drastically cut order delivery times, including same day deliveries http://blog.postmates.com/post/104856354257/powering-on-demand-logistics using Postmates API.

“I visited a pretty well-known ecommerce startup in the Valley a couple of weeks ago. I can’t name the company, but their technology team had zero ops people because they are able to manage all of the ops that they need through software,” says Reed. This retailer hires developers who handle the ops and they don’t have to split their team up, which means more people are building better features.

They’re not spending their time managing hardware sitting in a data center somewhere. They don’t have to manage their bandwidth consumption as much. “They can do it through a control panel in software rather than by making calls to a data center,” says Reed.

And the faster the ecommerce site is, the better the conversions will be. “If your site is very quick, which goes hand in hand with scaling and how you handle capacity, the better the experience will be for everyone from the consumer who is purchasing the products to the retailer selling them to the shipper shipping them out,” says Reed. This all adds up to increased customer satisfaction.

Through continuous integration / deployment, DevOps is able put customer feedback into action very quickly, creating and testing fixes to customer experience issues, which teams can then immediately deploy. “Since we started using DevOps concepts in development, the product development cycle has decreased the time it takes to get things done and into the marketplace,” says Reed.

Before if you were a user and you reported a bug, the enterprise might well have said, “well, okay, we’ll get to it after the Holiday season because we don’t want to change anything at this critical time for fear of introducing more issues”. “Now,” says reed, “we’re able to do some testing and make sure it works and that it’s the right path for us. We’re able to get there very quickly and then we can make a more educated decision on whether now is the right time to make the change.”

Further, the online retailer has the opportunity to consider whether making the change immediately might save a significant amount of sales that would have been lost.

December 18, 2014
Choose Digital Chooses Jenkins For Continuous Deployment

Challenge: Shrink Time To Innovate

Choose Digital offers a white label media store solution (think iTunes) for enterprises that want to reward brand loyalty with opportunities to use miles, points, or cash to buy digital products. As a competitive media company, Choose Digital was eager to test and adopt innovations to its software platform as fast as its developers could produce them. The media firm was ready for a model that lifts the burden of maintaining tools in-house.

“We were looking for a way to move from a straight compute environment to a DevOps approach that would enable us to push lots of code and new builds out daily while quickly iterating a lot of new ideas for improvements to our digital marketplace,” says Mario Cruz, Co-Founder & CTO, Choose Digital. A managed service could also relieve the headaches of supporting all the development applications internally. Choose Digital chose CloudBees’ Jenkins , a Java-based pipeline and orchestration tool for continuous deployment, which was also available as a hosted service, to meet those needs.

Solution: Simplification

Once Choose Digital made its decision, CloudBees made it simple to get up and running. “We opened our account with CloudBees and were productive with our Dev and Stage environments in two days using a fully functional Jenkins trial implementation,” says Cruz. Choose Digital moved production into the cloud 30-days later.

Jenkins enabled Choose Digital to find and fix bugs and try new things fairly quickly, without waiting for monolithic releases. “Jenkins is how we innovate, test our code, and enable the coder to immediately fix what he breaks,” says Cruz. Jenkins provides continuous integration services for software development and enables Cruz’s team to run build tests and functional tests.

“Jenkins gives us a quick snapshot so we can find broken code right away, not five weeks from now,” says Cruz. Enterprises that are still using older approaches don’t typically have time to fix small bugs, and certainly can’t fix them right away.

Choose Digital uses Jenkins in the cloud (DEV@cloud), which CloudBees fully manages and hosts on AWS EC2 http://aws.amazon.com/ec2/. Cruz doesn’t worry about maintaining the environment or whether his media platform has enough space or capacity. “CloudBees manages all the updates and issues with Jenkins CI. They support all the plugins for Jenkins, even if it is not their own plugin,” says Cruz.

As a managed service, Jenkins simply works from machine to machine. “My developers can focus on the product. They don’t have to worry about whether they have the right software build, the right apps on their machines, or the right versions of those apps,” says Cruz.

“As a result, development is up to 5 times faster in our continuous deployment environment than it would be if we had set up our own hardware or used an IaaS solution. Things that could take months or years we can now do in days,” says Cruz.

A Bump In The Road

As Cruz and his team moved to Jenkins, the biggest challenge was communication. “The toolset is not the challenge, it’s having everyone thinking with and getting used to the new DevOps mindset,” says Cruz; “while we were fully functional in two days, it took two weeks to get used to the new mindset.”

Today, Cruz’s software team communicates across stakeholders and team members in a DevOps approach that leverages Slack https://slack.com/ (IRC) and HipChat https://www.hipchat.com/ plugins for Jenkins. “DevOps is more a culture than it is tools; DevOps is all about communication. We communicate to our team, to the QA department, and to product managers,” says Cruz.

Current State: Everyone Knows, So We Can Fix Everything

Jenkins brings transparency so that what only some could see—such as what a developer has broken—becomes everyone’s knowledge. “Things get flushed out pretty quickly. By using Jenkins CI, you see the hidden things that no one knows about quickly,” says Cruz.

Cruz has some realistic advice for his peers: The sooner you get to an environment and mindset like Choose Digital uses, says Cruz, the sooner you will see the benefits and flush out bigger problems. “It’s not perfect the first time around but it gets better and better on every build. You get rid of the technical debt you didn’t know you had,” says Cruz.

November 25, 2014
How to Enhance DevOps by Supporting the Work of the Build-Release Engineer

Support Through Freedom

As we’ll see, supporting the work of the build-release engineer enhances DevOps by adding consistency, easing minds, and freeing DevOps teams to do more. To support the work of the release engineer, the enterprise must set them free as well, by letting them automate everything, treat manual interventions as process failures, and configure the target deployment machine and its monitoring context.

Enable the Release Engineer to Automate Everything & Treat Manual Interventions as Process Failures

One of the best ways to support the work of the build-release engineer is to encourage them to become actively hostile to manual interventions, says Robert Fischer, VP of Engineering, Webonise Lab www.webonise.com. Release engineers should shoulder the responsibility for putting manual interventions out of reach.

An enterprise’s internal DevOps community should empower the release engineer to automate as many steps as possible, which may require the enterprise to give them programmatic access to environment credentials, says Fischer. “For the internal DevOps community, the reward for permitting this risk is that they will have a cleaner environment that everyone understands better. This will make it easier for them to respond correctly in stressful situations,” says Fischer.

“So, let’s say you have some Linux VMs spinning up as part of the release. The release engineers could be tempted to use some kind of global SSH key for accessing these VMs, which they would install nicely on their laptops or workstations,” starts Fischer. Unfortunately, this will make it easy to add undesirable and unnecessary manual steps such as “‘Log in and set this IPTables value’ or ‘Log in and manually determine this hostname’”, says Fischer.

And while it would be easier to enable and apply these changes in the short run, it would leave a monkey wrench under the hood of automation and discombobulate the efforts of DevOps true believers in the long run. And it would all start simply by enabling the global SSH key and making these manual steps possible.

These kinds of manual processes also crop up in branches in the release logic: ‘Login to the server and see if packages X, Y, or Z are up-to-date in the image. IF they aren’t, THEN…’ All of that needs to be automated. “If the release engineers make it hard to implement manual steps, then they are more likely to A) notice and B) avoid these manual dependencies,” clarifies Fischer.

Instead of making manual steps possible, the release engineer should create active blocks for manual intervention, such as putting virtual machines within a virtual private cloud or making them only accessible using the automation tools’ SSH keys, says Fischer. This will actively impede manual interventions like those first cited herein and force the release engineer to think. “The result is that you are more likely to build the appropriate automation steps and error handling instead of creating a dependency on manual intervention,” says Fischer.

Empower the Release Engineer to Configure the Target Deployment Machine and It’s Monitoring Context

Another way to undergird the work of the release engineer is to open up the target deployment machine and monitoring context configurations to them. To enable the release engineer to configure the target deployment machine, make tools like Docker www.docker.com available. To enable them to configure the monitoring context, automate the configuration of the virtual machine platform (e.g.: AWS http://aws.amazon.com security groups, load balancers, etc.). “This allows for tuning to be a part of the release planning,” says Fischer.

The enterprise DevOps folks can still make manual adjustments if they must, but they should resist the impulse to use that option, except when handling anomalies. “Leaders in enterprise DevOps approaches can audit these deployment instructions for sanity and compliance, of course,” says Fischer.

As is the case with automating otherwise manual processes, the enterprise DevOps team should have a lot less to worry about once they set the release engineer free to configure these things. “This should give them more time to tackle the hard and interesting problems instead of performing repetitious and error-prone manual interventions by letting the computer handle the brain-dead computational and rote part of DevOps, which is what computers are good at,” says Fischer.

October 30, 2014
Separation Anxiety & the Leap from Dated IT Approaches to DevOps

Never ASSUME!

I assumed that the drag on moving from IT as we knew it to a DevOps approach (with its continuous software deployment and other advantages) was due to the separation anxiety and fears of loss or change on the part of enterprises on the whole. As you may know from the 70’s TV show, The Odd Couple and its character Felix Unger, played by Tony Randall, you should “Never ASSUME, because when you ASSUME, you make an ASS of U and ME.” *

And so I was mistaken, but not far off from the truth. “I don’t think that enterprises as a whole fear change so much; what I think happens is that individuals fear change and erect a number of barriers to change,” says Alan Baker, President and Chief Consultant, Spitfire Innovations (link), a business process optimization consultancy. Let’s investigate some of those fear-based barriers.

Barriers to DevOps, Based in Fear

The first barrier is resistance due to a loss of knowledge. “There are individuals within IT who have built their careers around the current practices and approaches,” says Baker. These IT professionals have developed reputations as subject matter experts, which they truly are. This enhances their value, but only so long as the current subject matter retains its value. By moving to DevOps and making material changes to the way developers and IT operations accomplish their work, the enterprise will diminish their knowledge as well as perhaps their status and their usefulness to the company. They have a vested interest in the status quo. They would have to have more to gain by joining DevOps than by opposing it in order to move to a more progressive approach to change management, according to Baker.

The second barrier is political; it’s the loss of control. IT professionals who shoulder accountability for operations day in and day out are typically averse to loosening their grip on how change is introduced into their environment. “That’s understandable, because once something is productionized it’s their responsibility; they want to be completely sure that the deliverables are defect-free,” says Baker. For this reason, these individuals are going to need assurances that the new processes will maintain the integrity of the deliverables, says Baker.

The third barrier is structural. In an environment where some form of management by objectives is practiced there is typically little alignment of objectives between different groups. “Day-to-day operations typically have objectives relating to efficiency (such as throughput), whereas development objectives are driven by dates and cost,” says Baker.

Reaching for Change

If the enterprise is going to break down these barriers and change its practices, all the stakeholders need to come together and agree on a plan of action. The stakeholders, including anyone tempted to resist DevOps, need to fully and collectively develop the future state of the enterprise, focusing on shared objectives. “All groups that are concerned need to have a set of common goals to work towards that satisfy all stakeholders,” says Baker.

There also need to be necessary supports. A DevOps approach requires increased integration and communication between all of the stakeholders. The walls need to come down. “This openness doesn’t necessarily come naturally to all IT staff; the organization will need to invest in the appropriate education and expert guidance (as well as some patience) to ensure that the cultural change has a chance to take root,” says Baker.

The final item consists of two value propositions. These value propositions give the organization the opportunity to demonstrate its vision, transparency, and resolve. The first concerns the need of the organization for someone to take an approach that answers the who, what, when, where, why, and how questions surrounding DevOps activities and responsibilities, and then to clearly articulate all aspects of the change to everyone. It’s a necessary step in order to get the buy-in. The second value proposition is directed at the individuals impacted by the change. Help answer the question ‘What’s in it for me?’, particularly for those individuals working themselves out of a role.

*It was one of the show’s funniest and most memorable lines and is repeated to this day. Though proper attribution for the original comment may ultimately be out of reach, see http://en.wikipedia.org/wiki/Jerry_Belson for the Television Writer who placed it in the Odd Couple episode.

October 26, 2014
devopsy.kinsta.cloud Examines DevOps-as-a-Service

Should enterprises buy DevOps or build their own? devopsy.kinsta.cloud looks at the benefits and detractors of DevOps-as-a-Service.

To -As-A-Service or Not To -As-A-Service

Vendors are moving enterprise technologies to the cloud, adding “–as-a-Service” at the end of their titles. Such was the case when Disaster Recovery became Disaster-Recovery-as-a-Service. But not everything that can be done should be. Some say that DevOps-as-a-Service is a good example.

“If DevOps is a growth engine for your company, then outsourcing it is like building a car with the motor located elsewhere,” says Michael Riley, Co-Founder, Boxter http://contentboxter.com. Optimizing DevOps requires tight integration between differing and complex entities. If a service can wrap this system into an external package for an enterprise, then either that customer doesn’t have a real need for DevOps or this approach is going to hurt the business, concludes Riley.

But TriNimbus is forming partnerships rather than selling packages. The DevOps-as-a-Service provider educates enterprise customers, brainstorms with them, and develops a shared vision of DevOps, according to Goran Kimovski, Principal Cloud Architect, TriNimbus http://www.trinimbus.com/. The final product is external to the enterprise in varying degrees, depending on the customer’s comfort level and expertise.

TriNimbus’ Take on DevOps

On the process side, TriNimbus takes the stance that DevOps is not primarily about continuous deployment. “The way we see it, DevOps encompasses many aspects including performance monitoring, management, and being able to recognize anomalies and patterns,” says Kimovski. DevOps includes solid backup and restore and disaster recovery processes for all aspects of the deployment. “We essentially try to match requirements with processes,” says Kimovski.

On the tooling side of DevOps, TriNimbus introduces automation tooling, different services that support automated deployment, and the implementation of automated workflows. The enterprise may have homegrown tools or be running continuous integration. “We try to architect a solution that involves the tools that they are already using and take advantage of those by using the skillsets that exist in their organization. So, we architect a solution that is going to scale and that we can maintain,” says Kimovski.

But taking these kinds of home grown tools, services, and workflows into the cloud with a DevOps-as-a-Service provider is risky. “It can sound like a cost effective solution, but end up killing your company’s growth in the long run. You are taking critical infrastructure and handing it over to a third party. If they care more about your business’s growth than you do, then you have a whole other problem to solve,” says Riley.

Software Development Expertise

Creating DevOps-as-a-Service requires expertise in software development. “Our background is software development,” says Kimovski. TriNimbus uses solutions architects and software engineers with 10 to 15 years of experience. TriNimbus has a background in continuous integration and build and release management.

TriNimbus’ experience enables it to work with the customer to break down barriers. In one instance, the customer’s obstacle is patching. “We have a current project that we are working with a client. It’s fairly complex. Each key area has its own piece of infrastructure. Patching it all together is a problem of integration and registration,” says Kimovski. It’s not a single problem of pushing a build onto a machine. TriNimbus built an orchestration workflow that patches things individually and supports rolling patches back if anything goes wrong.

The Cloud Advantage

TriNimbus uses the cloud (Amazon’s AWS http://aws.amazon.com/) to give developers greater access to provision things like compute and storage. “AWS as a software arm seems to understand the need for embedding tools into the platform itself,” says Kimovski. Developers may want to increase the frequency of automated testing. They may want to do load testing. TriNimbus works with developers to enable them to clone a production environment to do that.

But simply cloning a production environment in the cloud for automated testing using a DevOps-as-a-Service provider is more complex than it sounds and carries some concerns. “Cloning a production environment for testing can help you solve problems, but does not help you create a production environment in the first place. It could also cause problems if it’s not directly comparable for performance,” says Riley.

Increasing Release Frequency

Though continuous deployment isn’t the only thing DevOps has to offer, an increase in release frequency is something most TriNimbus customers want. But even TriNimbus admits there are no guarantees. “Just because we have the tooling automation in place doesn’t mean that magically things will start to happen faster,” says Kimovski. A lot of it has to do with being able to test more frequently and get confidence sooner that what the customer has built so far is not going to break things. As a tool in the right hands, that can raise release frequency.

October 9, 2014
Targeting software development to SoCs / NoCs using a DevOps approach

Without hardware to run it, software is useless. Hardware influences software development, since the software must fit the hardware platform and take advantage of its strengths, abilities, and components. Network-on-Chip / System-on-Chip hardware is a relatively new platform enabling communications between network and system elements that exist on the chip at the molecular level. This is a fascinating area. Learning to target software development for the NoC / SoC platforms using DevOps approaches should prove equally enthralling.

The Top Three Keys to a DevOps Approach to Developing Software to an NoC / SoC Platform

According to Aaron Cois, Researcher, the Software Engineering Institute, Carnegie Mellon University, the top three keys to a DevOps approach to software development for NoCs and SoCs are as follows. First, nurture a collaborative culture where collaboration between the parties involved in the software development process can flourish. Second, leverage Infrastructure as Code (e.g. simulation of hardware, EDA or ECAD applications, and programmable environmental configurations). Third, use automated creation / testing components for both hardware and software to create a fully automated DevOps pipeline.

The enterprise must foster equal collaboration and engagement among the necessary participants. “Getting technical staff with specific areas of expertise (Dev, Ops, Security) fully engaged in the project from the beginning ensures that their respective areas of expertise are first-class concerns in all system design and implementation decisions,” says Cois. This is correlates with Conway’s Law, which states that “two software modules A and B cannot interface correctly with each other unless the designer and implementer of A communicates with the designer and implementer of B. Thus the interface structure of a software system necessarily shows congruence with the social structure of the organization that produced it” (http://www.melconway.com/Home/Conways_Law.html).

Therefore, says Cois, a team of software developers will design a system primarily to meet software development concerns (features, performance). A team that also contains IT Ops professionals will design a system that also meets operational needs (deployment, maintenance). A team that also has Information Security professionals will produce a system designed to address security concerns. “All team members have to be equally tasked, engaged, and accountable for the project, however, for this effect to work,” says Cois; “this is where organizations commonly fail.” In real-world scenarios, tasking is unequal such as when IT operations attends a few meetings to consult, but the primary project team is made up entirely of developers. Or, in another example, InfoSec does a system audit before release and identifies issues that development should fix, but because InfoSec wasn’t strongly involved in the system inception and design, development may never complete the fixes.

The enterprise must focus simulation techniques on the development process. “Simulation and mocking are integral in situations where you are developing custom hardware as part of the overall process. This is the case with SoC/NoC systems,” says Cois. By simulating the chip hardware, developers can script/automate a number of things:

1) Testing hardware configurations
2) Testing hardware interfaces to software systems
3) Testing software/firmware deployment onto hardware
4) Security concerns
5) Possible networking scenarios on NoC chips
6) Other quality attributes such as response time and load testing

“Simulating the hardware platforms enables the development team to perform the same automated/continuous deployment techniques that are common to DevOps in general on SoC/NoC development projects as well,” says Cois. This gives developers advantages such as the ability to automatically deploy the system multiple times per day, which builds confidence in deployment success (through repetition). As with all automated deployment scenarios, the parity between the internal deployment platform (the hardware simulation) and the production deployment platform (real hardware) defines how much confidence development gains from repetitive deployment (e.g., if the simulation isn’t accurate, you won’t be very certain deployment will work on real hardware).

Challenges & Solutions

There are challenges to developing SoC/NoC software the DevOps way. First, the organizational culture needs to change so that communication and collaboration between various areas of expertise can dramatically increase. This is difficult. According to Cois, the solution is to hire generalists who are T-shaped professionals with a wide T. “They will have their specialty (Dev, Ops, Sec) and an understanding of other project needs. This understanding leads to empathy and effective collaboration–to the benefit of the project,” says Cois.

The second challenge lies in adopting automation, which is a significant technical shift, especially on legacy systems. It requires planning and technical expertise. “Knowing your goals before you start is key–a lot of organizations may jump into implementing automation without knowing the benefits they expect to see–or how to measure them,” says Cois.

October 6, 2014
BlazeMeter Adds New Relic Insights to Drive Testing

BlazeMeter a performance / load testing as a self-service platform for mobile apps, websites, and APIs lets developers test mobile and API backend performance and view KPIs in a dashboard. BlazeMeter tools simplify testing, reducing the time to test while running many tests per day. Customers use BlazeMeter as a self-serve and on-demand service.

“All they need is a web browser to generate massive loads from various geographical locations and view the test results in real-time,” says Alon Girmonsky, Founder & CEO, BlazeMeter. BlazeMeter lets developers simulate any size and configuration of device (Android, iOS, etc) and network (AT&T, Verizon, etc) user groups from any country using their app or browsing their website in order to test the backend.

Developers can run simulations from minutes to hours in length, gaining real-time performance intel. Developers can test and fix the backend repeatedly until they are satisfied they have a version that is ready for release. Developers can then test continuously throughout the product lifecycle. Developers can script automated tests, test what-if scenarios, and examine user interactions with their backend technologies. BlazeMeter can scale simulations to satisfy very large customers who need to test for the kinds of loads that come with up to a million users.

Irdeto Perfects Multiscreen With BlazeMeter

Irdeto (http://www.irdeto.com/), a Naspers (http://www.naspers.com/) subsidiary and solutions provider for Pay-TV operators needed to performance test its Multiscreen mobile app, which enables TV viewing on a variety of devices and screens online. The Netherlands-based company had to examine broad simultaneous use of its app across many systems and complex, variegated usage patterns. Irdeto selected BlazeMeter to load test its Multiscreen technology backend while scaling user group size and changing test scenarios in real-time.

Irdeto must enable a seamless, low latency UX across all devices while enabling audiences to access media content on-demand on new mobile devices as they appear in the marketplace. To do this, Irdeto partnered with BlazeMeter for additional support to test the app against “real world” scenarios, simulated by JMeter scripts (Apache JMeter http://jmeter.apache.org/). The JMeter scripts had to be created and entered into the system, leaving a lapse in an otherwise automated process. Tested scenarios included surfing, buying, and running premium content and interacting with Google Wallet (https://www.google.com/wallet/). Irdeto tested ramp up using JMeter’s Ultimate Thread Group (http://jmeter-plugins.org/wiki/UltimateThreadGroup/).

Applying BlazeMeter and JMeter tools, Irdeto pinpointed and opened bottlenecks based on results from scaling tests to hundreds of thousands of users running a labyrinth of usage patterns. BlazeMeter enabled Irdeto to alter test schemes “on the fly” including adjusting weighting for one user activity over another. Irdeto was able to use BlazeMeter to drill down to less apparent bottlenecks and then resolve those as well.

Speaking The Language of DevOps

BlazeMeter is partnering with New Relic Insights (http://newrelic.com/insights) to add functionality to BlazeMeter’s testing and dashboard including the ability to query application data after testing using natural language questions, query data sets over time, audit test activity, and answer questions that relate to data coming from BlazeMeter and NewRelic combined. “New Relic Insights enables developers to create new dashboards that run in real-time. Insights can add five more dashboards to the 20 that BlazeMeter provides,” says Girmonsky.

“BlazeMeter and New Relic Insights speak the language of DevOps, the language of continuous development and improvement by automating the development process, removing bugs, and reducing the time to project completion,” says Girmonsky. Developers can use these tools to follow performance trending, to see whether and where performance is improving or degrading, and why.

The partnership helps operations, too. “The ops guy needs to see that all apps are up in production and not affecting each other or the overall user experience. These tools enable regression tests to do just that with every new software release,” says Girmonsky. The ops guy can use his New Relic Insights dashboard to build trends from continuous testing and improve response times as they discover issues using big data analytics.

According to Girmonsky, enterprises should expect ROI in time, cost, and person-hours savings using these tools. “We didn’t conduct these specific kinds of analyses, however, prior to the availability of these tools enterprises had to export data into Excel and do the calculations that way. Now, they can run automated calculations in minutes,” says Girmonsky. According to Girmonsky’s estimates, BlazeMeter offers the enterprise the equivalent of building its own infrastructure to do testing, which would require a commitment of about two professionals for a full year to build that out.

BlazeMeter alternatives include Flood.io (https://flood.io/), Load Impact (http://loadimpact.com), and HP LoadRunner (http://www8.hp.com/us/en/software-solutions/loadrunner-load-testing/). There are several other load testing platforms.

[*Irdeto data compiled from Irdeto / BlazeMeter customer reference. New Relic was not a participant in that case study.]

September 30, 2014