— ROELBOB
In the spirit of Halloween, this past month testing experts from around the globe answered the call to join in a “crowd-sourced” article highlighting their most frightening testing QA experiences. These are real-life examples of testing gone wrong—leading to all sorts of mayhem in some cases. We allowed these testers to hide their identity, and to keep their organizations anonymous to solicit the most behind-the-scenes responses.
In no particular order here are the top five Testing Nightmares, as told by the testers themselves:
The Importance Of Making Sure Your Hosting Company Gets Paid
By Ivar (from a medium sized company/ now acquired)
The company I was working for at the time had bet its existence on signing an enterprise contract that it had already been in negotiations for more than year with. We were building a digital asset management system that controlled the licensing and delivery of assets to corporate devices. (Think ‘private app store’.) I had written backup tooling that wasn’t ready for use but was included, unused, in the branch that was being deployed. Ten minutes after the final deployment (in preparation for client review!) all of our assets disappeared. Nothing new in the deployment was related to asset storage except my code. There was an eight-figure contract on the line and we had no backups. Fueled by the mother of adrenaline rushes, I reviewed my work and remained convinced that my code was not the cause of the loss. I was already into investigating other possible causes when another developer informed our team that after calling our asset host, he discovered that we had neglected to update our billing credit card. The host had turned off access to all of our assets ‘to get our attention’. They succeeded, but at oh what a price…
Not The Best Way To Start As The New Test Automation Engineer
By James Eisenhauer
This happened my first week as the new and only Quality Engineer at a health and wellness startup. I was on a mission to get test automation created and show them what I could do for them! My first test was to invite all the members of a test company to a new company wellness challenge. It just so happened the company President’s test account was a member of that test company. I scheduled my tests to run at 5 a.m. At 5:38 a.m. we all received an email from the President that read, “Emails every few minutes telling me that QA wants me to walk by the damn river. Make it stop!” “Walk by the River” was a copy of a real wellness challenge we had in our test system. Something in my auto-retry logic had gone horribly wrong. First impressions are a bitch!
Gaming Fail
By Abe Chiu
While working for a gaming app company, we pushed out a build with an engine update that reverted all of our users back to levels that were saved on their device. This as opposed to the one that was saved on our server side. (Example: If logged in to create an ID and was at level 50 on your device, that was the save that popped up even though on our server you had previously reached level 125). All of our users were affected. That’s not the worst part. Some of our users reverted back to level 1 when they were above level 100 and up. This all happened on a Friday and the build was pushed on Thursday night. On Friday we were only able to fix some issues with a small percentage of our user base but the ones that were reverted back to level 1, those were not able to be helped until the following Monday.
The White Screen of Death
By Anonymous
I work as a Software Engineer in test at a media company. This company has its own video player that is embedded either as a single standalone video or a playlist. The player was re-done completely, and I was responsible for testing it before it went out to production. I tested the player as a single video and found a number of issues so the release to production was pushed out. After the issues had been fixed, I tested it again as a video and then gave my “go-ahead” to release it to production. The problem was I had COMPLETELY FORGOTTEN to test the playlist! We had figured that everything looked good and since the release had already been pushed off once, we made a decision to release it to production at the end of the day, around 5:30 p.m. I double-checked it on production as a single video and it looked fine. An hour later, when I got home, we started receiving urgent messages and emails that the playlist wasn’t working!!! You could only watch the first video and then it became a blank white screen! Since the playlists are embedded so deeply, this was a major problem. At 6:30 pm that night only one engineer was around and online but he wasn’t sure how to revert it. Finally after a very stressful hour and a half, he figured out how to do it and I verified that everything was back and working again with the old player. I still can’t believe that I had forgotten to test the playlists, which is such an important part of the feature.
Global Co
mmunication Breakdown
By Joe Nolan
We were trying to create a mobile on-line conferencing feature to include in our education app, much like Google Hangouts. Testing had gone well in controlled environments. Then we pulled all of the teams together to have a large group call using and testing the feature. We immediately had reports of crashes and all sorts of issues. It was impossible to talk over the people trying to diagnose the problems, while at the same time others were randomly pushing buttons to try the feature. Even worse, participants were located around the world with different networks and connectivity issues. It was worse than herding cats!
About the Author/Sauce Labs
Sauce Labs is the leading cloud-based web and mobile application automated testing platform. Its secure and reliable testing infrastructure enables users to run JavaScript unit and functional tests written with Selenium and Appium, eliminating the time and expense of maintaining a test grid. With Sauce Labs, organizations can achieve success with continuous integration and delivery, increase developer productivity, and reduce infrastructure costs for software teams of all sizes.
The prior blog Continuous Test Monitoring DevOps Healthbeat discussed the importance and suggested practices for testing and monitoring test trends. This need is amplified when the system that is being tested and monitored is a network of co-operating applications, infrastructures and devices such as networked components of next generation Software Defined Networks (SDN) and Network Function Virtualization (NFV) systems. In networked systems a problem with any network component such as end-user applications, network interface devices, network control plane, network data plane, load balancers, firewalls, network administration, orchestration, and network virtual infrastructures can seriously impact user and network services performance. The number, variety, topology and versions of components and their host environments present special challenges for DevOps continuous testing and monitoring.
In an SDN and NFV world all production topologies and network variations must be tested and results must be monitored or the results will be uncertain. Despite the complexity, or perhaps because of it, continuous testing and monitoring must be optimized to ensure that testing and results analysis for each CI/CT cycle keep pace with accelerated continuous testing and longer term continuous test monitoring is necessary to assure overall delivery confidence for each release and longer term product performance trend in a positive direction. Individual test results or a fixed number of test campaigns can only tell you so much.
Unless the results of testing and test analysis are continuously monitored and results aggregated over multiple test and release cycles for all network nodes and all topologies then there is no way to build sufficient sustainable confidence in the long term health of the SDN network and NFV components. The combination of topology-aware continuous testing solutions together with continuous test monitoring tools that can aggregate results over the entire set of network resource can provide a longer term strategic view of test results that is necessary to collect, aggregate and organize test results data to gain confidence in network components and services for each release and evolution over multiple releases.
Below are some suggestions in a checklist format that have proven useful for continuous testing and monitoring for sustainable high performance SDN and NFV systems and services.
Regression test network systems and services even though there were no expected changes: Unintended consequences of indirect changes may impact performance so automated regression suites should audit all production network test topologies and version variations areas occasionally just to be sure. Typical examples that are often caught by this are features important to audit functions, system backwards compatibility features and upgrade/downgrade administration features.
Select continuous test and monitoring tools that collect and report trends:
Tools that can correlate and report test results across multiple network nodes and topology variations can find intermittent bugs or problem trends. Thresholds and email alerts and dashboards that highlight short term results views from long term results views are especially valuable.
Use continuous test monitoring results to diagnose and resolve problems:
Intermittent failures and problems that only become apparent when test results are presented as trends over a long term trend are easier to diagnose when a large data set is accumulated and filtered by most probable cause tags. Once a diagnosis is determined, the root cause can be verified through targeted retest cases that set all the conditions in accordance with the diagnosis. Once confirmed then the offending design can be refactored to handle or avoid the failure condition.
The above is a partial list of suggestions for continuous test monitoring that have been proven to support sustainable SDN and NFV network systems and services. At Spirent and Zephyr we think continuous testing and monitoring is critical for successful DevOps. You can hear more about our views on this topic by reviewing our joint webinar at:
Overcoming DevOps Challenges
What do you think of these suggestions and do you have others that should be mentioned?
Marc Hornbeek is Sr. Solutions Architect of DevOps continuous test solutions at Spirent Communications, Infrastructure Test Optimization (ITO) BU. He recently managed DevOps at Spirent. He has performed as the primary architect of test automation tools and champion of test automation for firms ranging from start-ups to large multi-national companies. He published more than 30 articles and has been a speaker at numerous conferences and user forums primarily regarding topics related to continuous automated testing and DevOps.
Sanjay Zalavadia is responsible for driving custom success at Zephyr. This includes training, consulting, customer support and client management. Most recently, as the Associate Vice President for Patni Computers Telecoms IT Managed Services Practice, he established IT Operations teams supporting mobile content providers. Sanjay brings more than 15 years of leadership experience in IT and Technical Support Services teams across multiple geographies for both large and small companies. Sanjay has a graduate degree from the Manipal Institute of Technology in India
I know we have mentioned it before but we haven’t officially announced our DevOps Connect: cdSummit SoCal event on December 3rd in Los Angeles. The event co-produced with CloudBees, the enterprise Jenkins company, IBM and DevOps.com will take place from 9am to 6pm at:
The Olympic Collection Banquet & Conference Center
11301 W. Olympic Blvd., #204
Los Angeles, CA 90064
We have a great line up of speakers and sessions for this event and I wanted to share 4 of them with you today:
1st Keynote/Headliner: John Willis, evangelist at Docker – That’s right, the one and only @botchagalupe will be kicking things off. If you have been lucky enough to catch John present before you know that he never disappoints. Partly because his experience and knowledge give him superior insight into the issues and problems that IT teams face today, but also because he is so passionate about what he presents on. John will be presenting:
Kata
Mike Rother in his book Toyota Kata implied there was an invisible side to why Toyota was so successful over the years. As we we have explored the transitions from Lean to Agile and now to Devops we often fight the visible versus the invisible in accomplishing our goals. This presentation will take a look at what Kata meant at Toyota, Alcola over the years and bring it full forward with companies line Netflix and Etsy.
2nd Keynote: Andy Pemberton, Sr. Director, Solution Architecture at CloudBees. I have seen Andy present several times now and he does a great job of showing how the technology we are using everyday in our lives is being pushed, pulled, stretched and shaped by DevOps, Continuous Delivery and Agile. Andy will be presenting:
Orchestrating the Continuous Delivery Process
Continuous delivery is an end-to-end process and orchestration is the engine that drives continuous delivery. Automating the continuous delivery pipeline starts with understanding the process and of the entire toolchain. It sounds daunting, but it can be approached in stages to keep up with the demands for application delivery speed.
Boyd Hemphill, Director of Evangelism, StackEngine. Boyd is one of the great Austin DevOps folks. He runs the Austin DevOps Meet up Group, helps with Austin DevOps Days, Container Days and so much more. Like John Willis and Andy Pemberton, I have seen Boyd present a few times and he also just really brings a great talk. Boyd presented on “HomeOps, lessons he learned about DevOps while doing work at home. It is funny, insightful and a really great presentation.
HomeOps – Learning DevOps At Home
DevOps is great. You can see the benefits. All the success stories have you champing at the bit to become your very own Unicorn. Oh, but you are smarter than that. You know that every one of those success stories comes with many, often hidden, trials and tribulations.
HomeOps is the way I have become familiar with many of DevOps models such continuous delivery, lean thinking and cultural impact. Bringing these models to work with the ability to anticipate problems helped me save nearly $1.0m in costs while opening $1.8m in revenue.
Come gain some of the missing insight to your continuous delivery journey.
Bob Reselman – Publisher and Managing Writer: The World in Which We Live. Bob is half of the ROELBOB team that has been creating some of the funny cartoons here on DevOps.com. Bob also has many, many (did I say many?) years of tech experience under his belt which gives him the perspective he needs in his role at The World in Which We Live. Bob will be presenting:
Making a Linux Container: My pain and agony in grueling detail
Allow me to share an intimacy: I an experienced technology writer. Also, I am a container newbie. But, I understand that containers are an important step forward in computer virtualization. Knowing the stuff is a whole lot better than not knowing it.
So I took the plunge and decided to get down and dirty by building a containerized microservice.
I do have a method. I am a learn by doing guy. First I get the concepts down, then I get hands-on and write about it as I go. I assumed there must be a lot of Hello World type stuff out there to get me up to speed with the basics. Then, I could take things a bit further and build my microservice container on what I learned in the Hello Worlds.
I mean, it can’t be that hard, right?
Wrong!
It turns out that making my first container was a journey into pain and agony. No human should have to go through such an ordeal. So I am going to share my story with you in grueling detail so that you don’t have to experience such torture… ever.
We have another 5 or 6 sessions/speakers that we are finalizing right now and will announce next week. However, there are still one or two slots open and if you are interested in presenting you can write me at editor@devops.com with contact info and we will contact you.
Besides the great sessions and speakers, we will be featuring demos of Jenkins, CloudBees, IBM BlueMix and more. Also lunch, refreshments and more will be served. There is plenty of free time built in for networking as well including caricatures by the ROELBOB team and more surprises in store. We will also be announcing additional sponsors in the coming days.
So if you are in the Southern California area and into DevOps, CD, Jenkins, BlueMix or want to learn more, this is a can’t miss event. You can get more info here. You can register for the event (there is a $99 charge for the day) on Eventbrite.
We hope to see you in Southern California on December 3rd.
Why DevOps Communications Need to Be a Two-Way Street
Today, mobile apps have become one of the most important components of brand awareness and opinion. Often considered a company’s “front door,” a mobile app is an important way for consumers to directly interact with a brand, so it’s key for organizations to deliver high-performing, feature-rich user experiences right from the start. The margin for error is low – according to industry statistics, 79 percent of end users will only retry an app once or twice if it fails to work the first time, and only 16 percent report they will give it more than two attempts.
With these requirements in mind, the role of an organization’s development team is more important than ever. In today’s hypercompetitive market, organizations must get applications to market as soon as possible, without compromising functionality and end-user performance. This has led to the emergence of DevOps, a new approach to software development emphasizing greater collaboration, agility and speed across the entire software development continuum – developers, testers and IT ops teams – in order to meet the joint goals of velocity and quality.
An interesting by-product of the DevOps movement is the blurring and blending of roles across the application lifecycle as a result of these condensed timeframes and pressure to deliver new functionality, particularly in the area of testing and quality. This can be a boon, however, as we’ll discuss.
“Shift Left”: Eliminating Defects as Early As Possible
The traditional application lifecycle followed this model – ideation, development, testing and finally, production. It used to be that testers assumed the bulk of responsibility for ensuring an application worked as it was supposed to, before it moved to the production phase. But today, developers are assuming a greater role in testing, ensuring that applications meet basic functional requirements before they move any further down the funnel towards production.
This is known as the “shift left” movement, and ultimately it helps prevent code defects from getting baked into applications as they progress through the lifecycle. The further along an application moves toward production, the more “baked in” defects can become, and the more costly and time-consuming they are to un-do.
Even with their newly added responsibilities, the world of agile demands developers to work within shortened testing cycles. It is a race against time. Fortunately, tools exist today to bring greater testing automation to developers, plugging right into existing developer environments and frameworks. Further downstream, automation tools for testers enable them quickly provision (and de-provision) resources, and tweak testing environments. By reducing sluggish, error-prone manual processes, organizations can better deliver velocity and quality simultaneously.
“Shift Left” Also Requires A “Shift Right”
Ideally, an organization doesn’t want to treat real end users as beta users. But as the saying goes in software development, “the last bug is a mirage.” The reality is, even with all the due diligence on the left, finding some bugs in production is normal, and the “zero bugs” goal – while sounding commendable – can actually kill motivation, hurt creativity and discourage healthy risk-taking. Many developers note they spend much more of their time modifying existing lines of code, than writing new code. In a sense, testing – and the pursuit of better and better applications – never ends.
This is when IT ops teams can start making own contribution to application quality – a “shift right” so to speak. There is a wealth of information in production data that can be fed back to developers, in order to further optimize application functionality and end-user performance. Forward-looking IT ops teams are going a step further by applying data analytics, yielding deeper insights that can then be fed back to development teams.
Consider the case of SpeechTrans, a leading speech translation company. SpeechTrans applies Big Data analytics to end-user experience data – for example, what screens are end-users getting stuck on the most; where are end users exiting the app – so developers can focus specifically on improving these app areas. Previously, it took SpeechTrans up to three weeks to fix an app issue; now, they can address most problems in less than two days. This approach has enabled SpeechTrans to achieve consistent 5-star ratings in mobile app stores, a key driver of new customers and revenues.
Testing will always be indispensable in the development process, but further value can be realized by releasing an app into the wild and collecting real end-user data. IT operations teams can help developers and testers optimize applications in other ways. For example, they can support Canary Releases (or “rolling roll-outs”), enabling developers to roll out an application to a finite number of end users, in advance of a wider release. In addition, another important component of high-quality applications is cost- and resource-efficiency. IT ops teams can share production data on application usage patterns (such as, when during the day does an application come under heavy load, and when does this load taper off). This enables development teams to code applications accordingly – for example, architecting applications to accommodate longer or shorter queues (depending on time of day) which can free up CPU resources.
So in this way, when “OpsDev” happens, IT ops teams are helping developers do their jobs, but developers are also architecting applications with IT ops in mind, helping them utilize resources better. The days of developers simply creating applications and “throwing them over the fence” to testers and IT ops teams are over. Additionally, with real-time insights into production-level data, IT ops doesn’t just point a finger at development when a code-level problem arises. Rather, IT ops can provide deep, data-based insights into problems, giving developers a chance to prioritize and proactively address fixes. Advances in Big Data analytics are an important part of this.
Conclusion: Application Quality is a Shared Responsibility
To be truly successful in the development and testing process, organizations are recognizing that application quality needs to be addressed as early as possible, to prevent as many defects as possible from getting further embedded and ultimately reaching production. But like many aspects of the continuous delivery model, ensuring application quality requires a continuous feedback loop across developers, testers and IT operations teams.
In a continuous agile delivery environment, no one can shift responsibility for application quality to one group or another. The entire DevOps value chain – developers, testers and IT ops teams – must be involved and committed to collaborating as closely and efficiently as possible. In the true spirit of DevOps, lines are being blurred, between not just developers and testers, but also developers and IT ops teams. Silos are being transcended and “left-to-right,” back-and-forth communications streams are taking precedence. When “DevOps” also becomes “OpsDev,” the traditional, linear approach to development is replaced by a model that better supports the common goal – bringing high-quality software and enhancements to market, quickly and efficiently./
About the Author/Eran Schitzer
Eran Schitzer is a director of product management in the Application Delivery Management business unit within HP Software, responsible for driving the direction and release of new and existing products within the Performance Testing and Lifecycle Virtualization product suite.
He has more than 15 years of experience launching cloud, mobile and e-commerce products for consumers and enterprises, at companies ranging from start-ups to large-scale public entities.
Since joining the company, Schtizer has led the acquisition of Shunra Software, a network virtualization company; defined and executed the cloud testing agenda, and driven the move from an on-premise to Software-as-a-Service consumption model, including the launch of StormRunner Load.
Schitzer holds bachelor’s degrees in Economics and Management as well as Information Systems from Technion – Israel Institute of Technology. He also has a master’s degree in Business Administration from the University of California, Berkeley, Haas School of Business
DevOpsQA NJ just held our October Meetup on NoSQL in Agile DevOps Model with Shawn McCarthy from MongoDB, Bhavna Sehgal from Syncsort and Suhas Joshi as speakers. The venue was provided by ROKITT at a state-of-the-art office space with breathtaking views of NYC. 
Suhas, a Solutions Architect with a vast DevOps experience, gave a great introduction to NoSQL, starting with the history of the journey from relational to NoSQL and then deep diving into NoSQL Data Models including Key Value (e.g. CouchDB, Memcache), Document (e.g. MongoDB), Wide Column Store (e.g. Cassandra, HBase) and Graph (e.g. Neo4J). He also
touched upon big data saying that the term “big data” is a misnomer, it should be “huge data” instead.
Bhavna gave a talk about how Syncsort clients leverage Hadoop and NoSQL database to achieve powerful results. Some of the use
cases included mainframe data offload to Hadoop, query of large data sets and real time analytics on Hadoop. Bhavna also covered various options for accessing NoSQL in Hadoop including JDBC/ODBC and Java API among others.
Next, Shawn gave an exciting overview of MongoDB, starting with how it got started and deep diving into the functionality. MongoDB is a cross-platform document-oriented database. While MongoDB is classified as a NoSQL database, it also supports relational structures. Instead of breaking up data into multiple relational structures, MongoDB can store it in the small number of documents. Shawn went thru the reasons why divelopers like Mongo, which include the following 1) it was built by developers for developers, 2) very easy to setup, 3) availability, 4)automatic partitioning of data, 5) view of the data is abstracted out from the application. MongoD
B does not have transactions or joints and should not be used for transaction based banking data, but it’s a great tool for storing log data, user profiles, user-generated content and location based data. Shawn also covered Rich Query model and how Mongo can be used for real-time analytics. If you would like to learn more about MongoDB, you can reach Shawn at shawn.mccarthy@mongodb.com
At the end we had a great Q&A session and a very warm open discussion. Looking forward to our next session in November.
At a recent Lean project report out to stakeholders, a piece of analysis was presented which showed the quantity of metrics being captured, in various areas, at different organizational levels. IBM’s Watson had been used to generate the chart, and it told an interesting story. But that story needed some translating.
‘Look at all those metrics under ‘Managers”, said one Director, with enthusiasm. ‘Seems like they’re looking at a lot of different performance measures for their people’.
‘Yes,’ said the Lean Coach, ‘but is that a good thing in itself?’
Another Director chimed in, ‘Well, when I look at how few metrics are being captured on timeliness I wonder how anyone knows how they’re doing.’
As an IBMer, and Six Sigma Blackbelt, I’m pre-disposed towards data. Information can be power. But with too much being reported, it can be hard to see the wood from the trees. Performance metrics are intended to allow teams and leaders to improve their performance. For many of those people, reading charts and interpreting data may not be a core skill and it certainly isn’t how the organization wants them to spend their time. So those teams generating masses of performance metrics may not be doing themselves, or their leadership, a service.
The key is to identify the critical few metrics. Ideally leading indicators of performance issues, rather than lagging ones. Leaving aside areas like SLA adherence and budget management to those responsible for governing them, these are the handful of metrics I would suggest a DevOps team needs to watch on a daily basis:
What do you think? What do you measure?
Michael Oubre – Agile evangelist, practitioner and trainer, and Marc Hornbeek – DevOps continuous testing evangelist and thought leader, share their views on popular questions regarding Agile and DevOps from their respective viewpoints.
Michael: From an Agile perspective, let’s consider your company decided to take the plunge and switch its software development process to Agile, formed Agile teams, got them some training and set them up with a coach. The team have been scrumming for a couple of iterations but there’s not a lot of work getting finished, and many of the stories are having to be carried over from sprint to sprint to get the defects resolved.
You might be tempted to start worrying about the quality of the code being written, or whether you have the right ratio of test engineers, but it might be more useful to look at your build/test cycle. One of the big benefits of Agile is the reduction of the time between the code being written and the identification of defects – the shorter that time is, the lower the cost is to get it fixed. Agile removes the process barriers, but if you only rely on unit testing and infrequent integration testing, you won’t achieve the results that you hope for.
Agile teams will have a very difficult time of ever reaching high performance without the benefits of Continuous Integration providing the ability to deliver new code in an integrated, testable build to the Agile teams as close to “instantly” as you can manage.
Marc: From a DevOps perspective development teams that are “doing Agile” are the primary customers of Continuous Integration together with Continuous Testing.
Both Agile and DevOps emphasize the importance of incremental development processes rather than large, serial waterfall processes because incremental processes facilitate fast feedback and reduce process waste compared to long waiting periods between software creation and validation. In fact the DevOps “movement” was born out of a system test consultant’s frustration with the disconnect between some Agile development teams and separate operations teams. (Reference: http://itrevolution.com/the-history-of-devops/) . Today the words Continuous Integration and DevOps, together with Continuous Testing and Continuous Delivery are practically synonymous.
The mere fact that there are plenty of successful Agile examples that existed prior to DevOps or Continuous Integration became a word tells you that Agile can indeed be done without CI, but clearly CI is a great complement to Agile and supports the goals of Agile while helping to tie it to operations. So the strict answer the question “If you are doing Agile do you need to do Continuous Integration?” is, well, “no” but to leave it at that would be incomplete, so a better answer is “ no, but CI is definitely recommended to complement Agile”.
Bottom line is, from both Agile and DevOps perspectives continuous integration is really important to achieving performance goals. At Spirent we think Agile and DevOps work together to help teams develop and deliver quality software efficiently. More information about Spirent’s views can be found at the following link: Spirent.com/solutions/devops
What do you think of these views and do you have other views?
About Michael Oubre
Michael Oubre is an expert in Agile practice, transformation and management and has spent more than 10 years as a practitioner, coach, trainer and evangelist. His long and eventful tech career has given him practical experience and insight into every aspect of building, deploying, supporting and managing software, systems and networks.
DevOps is an approach and methodology (even a philosophy) for developing and testing new applications and deploying them into a production environment. Because DevOps is first and foremost a methodology, tools to implement DevOps should be used as a way to implement the desired methodology, rather than being an end in and of themselves.
Of course, many tools exist for automating parts of the DevOps methodology, and many are widely adopted. For example, few development teams would operate without source code control and versioning software. Yet I continue to be amazed at the number of enterprises that think that the best approach to implementing their DevOps methodology is to write some of the automation tools themselves.
The DevOps methodology is a common methodology, based on longstanding principles first espoused by W. Edwards Deming, through Six Sigma quality processes, and later popularized by “Lean” programming practices. Nothing is new in DevOps. So, what would be the advantage for an organization to implement their own DevOps tools?
We hear some common justifications for “Do It Yourself” solutions. Let’s review these justifications to see if they hold water:
This justification is based on some common misconceptions. For example, it is often difficult to do a good side-by-side comparison of cost between a commercial tool, an open source tool and writing the code yourself. Clearly the commercial tool has the most concrete and well established cost. When comparing to an open source tool, it is very important for enterprises to take into consideration the ongoing maintenance and upgrade costs that they will undertake themselves.
Early in 2015, Qualisystems interviewed its enterprise customers and asked them if they thought that the cost of open source tools was lower than commercial tools. They unanimously agreed that the cost was about the same from a total cost of ownership perspective. Other factors were the key influencers in whether to use an open source tool or a commercial tool. Estimating the cost of an internal development effort is even more difficult. One would have to consider the upfront development effort, and the ongoing maintenance and upkeep. A number of other intangible factors include the likelihood that the project will run over time (and therefore cost); how well the code will be understood in the future by new developers; the risk of loss of the key developers, etc.
Finally, there is the interesting question of how efficiently teams will implement their own DevOps tools if they aren’t yet already using DevOps methodologies and tools. (This is a little bit like a Catch-22).
Another hidden assumption behind this statement is that your enterprise doesn’t need all of the features provided by commercial tools, so therefore the cost to implement will be lower. Of course, most commercial and open source software has been vetted already by customers and the list of features is usually a result of extensive customer use and feedback. Often this feedback is not obvious to a new team just starting out with DevOps tools. And, then there is the fact that commercial and open source tools spread the cost of development across their entire customer base, so no one customer is bearing the cost of actual development.
Well, that is like saying that since I have a driver’s license, I should be able to drive a semi-truck. Designers of software applications require specific knowledge of the problem space and solution space. This knowledge and expertise yields Intellectual Property. If we extend this concept to DevOps tools, we would conclude that creating DevOps tools also requires specific knowledge and expertise and it is different from the knowledge and expertise of most enterprise software developers.
There is also an opportunity cost to writing your own code to implement tools. That time and effort is taken away from developing applications and solutions that would directly benefit the business of the enterprise. That opportunity cost could be quite high. The simplest way to do this assessment is to ask yourself, “Is a DevOps automation tool going to add to the relevant IP of my company in a way that increases our revenue?”
This argument also falls prey to the difficulties of accurately comparing the costs between automation and equipment. Purchasing more equipment to eliminate sharing will be an ongoing activity. The extra cost will be replicated every time equipment is upgraded or replaced. The extra cost will also grow linearly with the size of the software development and test teams. And, any equipment cost will be doubled by the cost of management, operations, power and cooling, etc.
Even with completely dedicated equipment, software tools will be required to properly configure that equipment to replicate different operating environments, a common requirement in testing and QA. If these software tools also implement sharing of equipment, then the overhead cost is low.
This is true but will also require that you maintain these plug-ins yourself over time, including staying current with API changes in third-party tools. Typically, commercial tools will carry this burden for you.
Well, it is hard to argue with fear and uncertainty. On the other hand, if the tool is just a means to an end, then switching tools should not require massive changes in process or methodology. If enterprises keep their focus on acquiring tools that match their methodology and processes, then they really can’t lose, even if they end up switching tools over time.
So, what is the answer? Perhaps there is a “best of both worlds” solution. After all, DevOps is a process and it is possible and even likely that each enterprise will want to implement that process in a way that best optimizes their own internal pipeline from development to production. Furthermore, each enterprise has its own mix of infrastructure in production and must optimize their testing to that infrastructure. Each enterprise has their own mix of legal and compliance requirements that they also must test against.
Perhaps the right answer is to provide a commercial (or open source) software solution that is more of a platform that enables easy customization (even in the form of code) to support unique process requirements, unique infrastructure, and unique testing requirements. At the same time, a common framework for implementing process orchestration, and for describing and controlling infrastructure would address the other cost issues of “do it yourself” development and make it easy to integrate in a standard way with other DevOps tools.
The time for “do it yourself” tools for DevOps is over. In the same way that no one would consider writing their own backup tool today (remember the good ole days when people did their own backups with tar files?), writing your own DevOps automation tools is usually not the best approach.
About the Author/Joan Wrabetz
Joan Wrabetz is the CTO for QualiSystems. Most recently, she was theVP and CTO for the Emerging Product Division of EMC. Joan received her MBA at the University of California, Berkley, and MSEE at Stanford, and a BSEE at Yale. She has held teaching positions at the University of St. Thomas, St. Mary’s University and the University of St. Thomas, St. Mary’s University. Joan holds patents in load balancing, distributed systems, and machine learning classification and analytics. Her experience inclues: founder and CEO of Aumni Data, CEO of Tricord Systems, Vice President and General Manager for SAN operations at StorageTek, Founder and CEO of Aggregate Computing, Management and senior technical positions at Control Data Corporation and SRI International, Partner with BlueStream Ventures.
