Software testing and QA is sometimes given low priority during development. The idea of treating software testers as adversaries by the developers seems old-fashioned and is short-sighted. It’s hard to believe that such an attitude can still exist, and yet it still persists, even in a modern DevOps team. But a DevOps-focused software team benefits when it considers QA/testing as an integral part of developing high-quality applications.
In a business setting, the hidden costs of buggy or unsecure software are lost productivity, revenue and extra work by the development team (and even extra work by users, if the software isn’t running right). QA/testing helps test for system flaws, vulnerabilities and security issues, so why wouldn’t a DevOps team be supportive?
QA/testing can help an enterprise avoid big mistakes. In fact, the DevOps Institute notes, “With the advent of DevOps and the movement to break down silos between developers and operations, it becomes critically important that all members of an IT team—regardless of what tools they use, or role they play—understand the essentials of testing.” And, according to Puppet, a standard for automating the delivery and operation of software, high-performing enterprises build quality testing into each stage of their software development process.
Make QA/Testing a Priority
Whether your DevOps-focused developers are handling the QA/testing themselves or passing it along to QA/testing specialists, it’s something that must be done. Ideally, developers themselves would handle QA/testing in an integrated DevOps environment, and that’s why some developers see it as an unwelcome chore. It’s not something they’re usually trained in, and they might struggle with it or see it as a burden. That could be the reason why some DevOps teams hit a “software testing bottleneck.” As data from a Vanson Bourne survey commissioned by Appvance shows, 73 percent of 200 enterprise IT decision makers in the United States have adopted DevOps processes, but 57 percent of those adoptees experienced a bottleneck during software testing that impacts how quickly new and updated software can be released.
On the flip side, the developer’s company may have QA/testing professionals in-house, so the developers are hands-off in that area. Needless to say, the relationship between developer and the QA/testing pro could get a little rocky. Just think about it: The role of the QA/testing specialist is to find problems and point out issues with the developer’s code. Or, perhaps the developer or engineering staff is lagging behind schedule, so the team inevitably needs QA/testing to rush its processes, further creating a tense or adversarial relationship between the two groups.
At one of my startups a while back, we worked for months on our first software release of a web application for an online marketplace. Late on a Wednesday night, we let the CEO of our company try it out—and the app immediately crashed. “Gee,” I said, “It was working on my computer.” Of course, it really wasn’t. We had synchronization problems, which are common in multi-threaded applications that access the same information for multiple users. And we made the mistake of letting the CEO be the first simultaneous user to try it out (creating a bad impression), instead of making organized QA/testing a priority at our company.
QA/testing in a DevOps environment can help your team avoid mistakes, so passion for app functionality doesn’t supersede quality and robust bug-free software. In the end, what you need from your team is passion for both functionality and quality/testing. Otherwise, you’ll deliver a faulty product and lose customers or users—unnecessarily—because of a bad first impression by your software.
Two Approaches for Integrating QA/Testing with DevOps
When you’re integrating QA/testing into your DevOps environment, there are two different approaches or procedures to ponder. First is the more traditional test of the full system, sometimes called integration testing. The second is test driven development (TDD), which is popular in Agile and Scrum methodologies. The main difference between the two is in how software code is tested. During integration testing, developers review and test the code. Then, for critical facets of the software, a peer review of code is completed. Afterward, testers create scripts and deploy an array of integration and system tests. These could include scalability testing, load testing, stress testing and very importantly, security testing. Code vulnerability, SQL injections and malicious code would be things to think about.
Under TDD, the order in which testing appears in the cycle is reversed. Each developer first writes “unit tests” and then writes the code. This is done iteratively in such a way that every discrete section of code is automatically tested. When new code is sent to the integration server, the tests go with it. The server itself runs unit tests for all the code that developers deposit in the server. On top of this, testers design system and integration tests to run automatically by the integration server. Finally, testers perform any manual tests that are impossible to automate. In this way, the integrity and quality of the code is constantly guaranteed.
Academic studies and field experience suggest that TDD increases development time by approximately 15 percent … but hold on. Experience also shows that total cost of ownership of TDD-developed software is lower than software using integration testing only. It’s worth noting that by using TDD and integration testing, a company can deploy software of world-class quality that exhibits less than 0.3 errors per 1,000 lines of code during 18 months of use.
QA/Testing: A Partner in Your Software’s Success
If you’re hesitant about integrating QA/testing into your DevOps team and instead tempted to outsource it, you may want to push pause. It’s actually quite challenging if you do development in-house and expect to simply outsource what you think is independent QA with cheap testers. Outsourcing QA/testing and assuming that piece is done is not really outsourcing—it’s staff augmentation at a distance and it usually doesn’t work. You don’t want a “siloed team” to handle a critical component such as QA/testing. Instead, outsource QA/testing and bring your in-house and outsourced teams together around a common DevOps culture and mindset.
DevOps experts Bob Aiello and Leslie Sachs, in an IBM report, say quality software “… requires not only a shift in behavior, but a dramatic shift in the organizational culture as well.” So whether you outsource or have an in-house squad, your developers and the entire team should take ownership of QA/testing, and remain open to new ways to test code efficiently.
Ultimately, QA/testing should not be viewed as a time-consuming, disruptive chore by anyone on your team or in your enterprise. Think of it as a partner in your software’s success—not an adversary.
As enterprises move to adopt DevOps practices, there is a desire to measure where teams are on their journey. This sometimes leads to the development of maturity models used to assess teams’ levels and progress. While I am not going to pass judgment on what might be useful for certain organizations, I believe that the real key is to drive positive DevOps practices, which include continuous improvement through data such as lead time, frequency of deployments and mean time to resolution (MTTR) (i.e. the key metrics from the State of DevOps Reports).
Any static maturity model will become outdated. What’s more important is that teams are taught concepts that can help them continuously evaluate what is stopping them from delivering more quickly so that they identify areas for improvement. Key to this is the lean concept of value stream analysis. If you walk up to a team and ask them why they can’t go faster, they will typically say they are waiting for something. They could be waiting for more work to flow into their backlog. They could be waiting for infrastructure needed for development or testing. Or, they could be waiting for another team to develop a service they need to consume.
Many teams don’t spend time in retrospectives talking about how to overcome these types of blockers or wait states and developing countermeasures as continuous improvement initiatives. One reason for this is the lack of objective data that can be analyzed to provide insights on this. This is where the lead time metric comes into play. Historically, lead time has been defined as (last) code commit to deploy into production. While this is important, it only tells part of story because accelerated delivery and feedback really starts with a customer concept and measuring how long this takes until it is delivered into production so feedback can be provided.
One challenge to providing this kind of end-to-end metric, which spans the entire delivery value stream, is the lack of an integrated delivery pipeline from which data can be automatically collected. A customer concept starts in the portfolio space, which might result in data being entered in some type of PPM (portfolio management) system. Once approved, this results in features being created which eventually should flow into a product backlog. Stories are then created that can be pulled by the agile line supporting that product into an iteration. This information is typically in some agile management system (e.g. Jira or IBM Rational Team Concert). Then the team develops and builds the stories, tests them and deploys them into production via a deployment tool like UrbanCode. So, this flow of work goes through various tooling and generally is not visible or easily traceable across the value stream.
Our initial goal in measuring lead time was to shift left and measure the time from story card backlog until the story was released into production. This would then give us four discrete measures that would provide more insight into the time spent in different parts of the value stream.
Time spent in the backlog
Time spent developing the story
Time spent testing the story
Time spent once the story was completed (marked “done”) until it was deployed
We mapped the first and last categories into wait times and the second and third into process times to provide a baseline of this metric. While we have just started providing this information to teams, we are finding that scrum masters believe this type of data can be used in retrospectives to better understand why some lead times are high. They also believe that this will help the team better understand when blockers are encountered during development and testing and what contributed to them. We also plan to make that visible as part of our metrics (time spent in a blocked state). Tooling like TaskTop Integration Hub can help integrate your pipeline tooling to provide visibility and insight.
Once teams gain more insight into what is slowing them down, they can identify countermeasures that can be applied. Categories of wait-states include:
Waiting for a dependent area to provide functionality necessary to complete a feature
SLAs associated with a service that is provided by a centralized organization (e.g. performance testing or security testing)
SLAs associated with infrastructure needed for an environment
Contention for a shared test environment
Countermeasures associated with these types of wait-states are:
Architecture decoupling
Dark Launching / Feature Toggling
Expanding cross-functional teams to enable them to do self-service performance and security testing (at least some portion of it)
Automated Provisioning Capability for environment to avoid wait-states and contention
The “State of DevOps Report 2017” contains information on the importance of architecture and how decoupling dependencies (using APIs and microservices) can mitigate certain types of wait states. DevOps Case Studies provides more information on applying DevOps practices such as dark launching.
But again, at the heart of this is getting teams to think more about what their value stream actually looks like, how long it takes to deliver business value that can generate feedback and what is getting in their way of going faster. This is not a technology play. This is not something that somebody can sell or educate you on. This is something the team itself needs to take on.
A good starting place for a team is to educate themselves on some of these concepts for DevOps practices using a book club. A great place to start is to study the “Phoenix Project.” This novel approach combines aspects of both DevOps and Lean to drive home some key points in the journey to discover types of work, manage work in progress and ultimately make visible and gain control over the work being done for the business. Once a team has an understanding of their value stream and blockers, they can then study the “DevOps Handbook” that provides practical guidance for improving their delivery capability and speed.
About the Author / Carmen DeArdo
Carmen DeArdo is the director of Build Technologies at Nationwide Insurance. He is responsible for helping drive accelerated delivery utilizing Lean and DevOps techniques across the end-to-end delivery value stream. One current focus area is the creation of an integrated delivery pipeline capability.
Over the recent months many of our customers asked for our guidance and building their test suites properly. Performance, ease of management and scalability were key priorities for them. We decided to share our suggestions with the broader audience.
We came up with the list while building our own test suite. Since we strongly believe in dogfooding, Testim is tested using Testim. We also learned a lot working with a variety of companies from small startups to large enterprises.
Rule 1: Prioritize
Most apps include thousands of scenarios. Testing the functionality and correctness is merely one step of the way to shipping high-quality software and meeting users expectations. There’s performance testing (speed and responsiveness in a normal state or under load/stress), visual testing (checking that 1+1 = 2 is one thing, but many visual bugs can make it a very bad experience), and so on.
Start by listing the most important user flows, for example login, add to cart, checkout, etc. Order the list by those more vital for user conversion.
Try to sort by first adding those that must succeed e.g. create account, checkout and payment. You can use your analytics tools (e.g. Google Analytics, Mixpanel, Kissmetrics, Woopra) to find which are the common scenarios. We also found Segment.io to be useful.
Rule 2: Reduce, Recycle, Reuse
The next step is to break the user’s scenarios down to simple, single purpose, almost atomic flows. e.g. “Create user,” “Login,” “Send email,” etc. Create a test for each of these flows. When completed, compare this to the the user stories.
Naming convention – Defining a good naming convention is an important part. For example, including the component name in the flow name, which can result in flow structure (e.g “Account.login“ and “Account.logout”).
Reuse components. Don’t Copy/paste – for non-experienced developers/testers, a copy/paste seems like a reuse. However, the challenge is to update those steps when the flow changes. Even a single step/line that repeats in hundreds of tests is a huge hassle.
Rule 3: Create Structured, Single-Purpose Tests
Single purpose tests verify one thing only!
Single purpose tests are easier to create and maintain, they will also run faster compared to longer and complex ones. Ideally, they should only fail if the feature they test is broken or one of its dependencies (which we’ll try to minimize). If you test several features and the first one fails, you know nothing about the other features, since you haven’t reached them.
A test should consist of three parts:
Setup – e.g. login
Those steps gets the AUT (application under test) to the state required for testing. This could vary by the test: sometimes you only require a login. you require a complex structure to pre-exist. Note: Premature optimization is not a good practice. Implementing the login flow (user/pass fields) is an easy way to start, and if you reuse the same login in all your tests, it wouldn’t be difficult to change the implementation later on to a quick implementation (e.g. by using cookies), so there’s no need to start with the speed optimization if you know it’s easily fixed.
Actions – the flow you want to test.
Continuing with our aforementioned “login” example above, unlike the setup steps, which require the login to be completed successfully (don’t care in which way), testing the “login” itself is a different task, and should not be reused.
Validations – validate UI state, validate DB – the result you expect.
There are different validations.his depends greatly on whether you’re testing data which is being updated from production on a weekly basis, or whether it’s the same Test Data.
In UI testing, the basic validations are done on elements in the UI.
Validate an element is visible.
Validate an element comprises some text.
Validate an element is in a specific state.
Visual Validation – validate the is comprising the expected text, image and has the correct styling (e.g. font, layout).
Using a predefined dataset has the advantage of making the validations easier, as when we know the exact setup state, you also know the precise expected state after the stimuli steps were completed.
If the dataset is updated, you get different scenarios which you didn’t expect (as different user use your application differently), making it harder to debug failures (as the scenario might not be reproducible).
Example: A test for a login could look like:
Set initial state:
Open AUT
Navigate to the signup page
Actions:
Choose a random username and it to a variable named username
g. “user” + Date.now() + “@mailinator.com”.
Sign up for new account by using username.
Set username to username
Set random password
Click submit
Validations:
Open mailinator.com
Navigate to “inbox”
Look for email to username
Check that there is one email with the expected link.
Click on the link
Login with username and pass
Validate login was successful
Note: One of the biggest differences between unit tests and end-to-end tests is the overhead. Opening a browser and logging-in to the AUT used to be extremely slow, pushing testers to perform many tests after reaching the Setup state, saving valuable time. Today, with docker, it’s much faster to boot a browser up, the speed of the browser help developers to write applications that load quickly, minimizing the overhead of writing end-to-end tests.
Rule 4: Tests’ Initial State Should Always be Consistent
Since automated tests always repeat the same set of steps, they should always start from the same initial state. The most common maintenance issues when running automated tests are ensuring the integrity of the initial state. Since automated tests depend on that state, if it is not consistent test results won’t be, either. The initial state is usually derived from user previous actions.
For example:
Adding an item to a list that already contains it (from previous test run).
Views that behave differently for new users and returning users.
Application redirect automatically if user already logged in.
Possible solutions:
Create new user for each test run
Use a dedicated app environment for test automation, as opposed to using your production one.
Seed your application with initial data before each suite run, also known as fixtures.
Use conditions on steps/group to handle dual cases in test which are hard to set.
Eliminate dependencies between tests. The AUT’s state after “Test_1” shouldn’t affect the flow of “Test_2”.
Rule 5: Compose Complex Tests from Simple Steps
Complex tests should emulate real user scenarios, you should prefer composing those test from simple tests parts (shared steps) instead of recording the entire scenario directly. Since you already tested all the simple actions in the simple tests, the complex tests should only fail on integration issues (this is where Rule No. 2 comes in handy).
Make sure to order your suite to run the simple tests first. It will give you indication when something is wrong in the complex tests when running the whole suite.
Rule 6: Add Validations in Turnover Points
Validations are usually used at the end of the test to signal the test is in a passed/failed state. It is also best to add them at points of major changes as a checkpoint to stop the test if an action failed. Since validation will run until its timeout reached, like all steps, it is also a mechanism to wait for a long async operations in your app (e.g. fetching data from server, continuous view rendering, etc.) before running the next steps.
Examples:
On action completion – navigation, form submission, modal display, etc.
After major DOM changes add validations to ensure you are in the new state before trying to play next steps.
Following a client-server communication which change app state – e.g. data submitted successfully message, etc.
Rule 7: No Sleep to Improve Stability
Sleep with random duration (what I call magic numbers) is one of the main sources of flaky tests. Although all apps are asynchronous in nature, there are rare cases where a human doesn’t know when the page is ready and perform the next action. The same should be true for automated tests.
The reason to avoid static sleep is you rarely know the load of machine upon you run the test on. Only in performance testing do you have the machine to yourself (knowing for sure all CPU and memory are for you). Since you probably use virtual machines or Docker you will get timeouts if you set the sleep too slow, or slow tests if you set it too high.
In most cases you add a wait-for-condition step, which waits just like a human would, or sometimes waits for some internal state (e.g. javascript value).
At Testim we provide all kinds of built-in wait-fors, ranging from wait for element to be visible to wait for a specific text (you use also variables and regex), wait for some visual change (e.g. color change) or code-based changes.
Rule 8: Use a Minimum Two Levels of Abstractions
If your test is composed mostly of user interactions, such as clicks and set-text actions, then you’re probably doing something wrong. If you used low-level frameworks (e.g. Selenium), you might have heard of the PageObject Design Pattern, which recommends to separate the business logic (e.g. login) from the low-level implementation (e.g. set username, set password and click login button). This is true for Testim as well. Having layers contribute to reuse reduces the maintenance and allows other members of the organization to quickly understand what happen, as sometimes the testers write the tests but the developers run them (before merging new code). The developers in this case need to understand quickly what the test does, and what is the bug.
Rule 9: Reduce the Occurrences of Conditions
A test should have as few as possible conditions (if-statements). Those that have many are usually unpredictable (don’t know the exact state you’re in) or complex (you might even see loops, too). Try to simplify your tests by:
Start your test at a predefined state;
Disable random popups; and
Disable random A/B testing, and choose a specific flow.
Rule 10: Write Independent and Isolated Tests
An important methodology of test authoring is creating self contained, independent flows. This allows to run tests in high parallelism, which is crucial for scaling the test suites. If, for example, you have 1,000 tests that run for a minute each, running them one by one will take more than 16 hours. Minutes, at full concurrency, can cut this testing time to 1 minute.
Having a fast build reduces the feedback time between the moment that developers commit their code and the moment they knows their feature is production ready and can lead to true agile development cycle.
In Summary
Proper planning of your tests can make a big difference in your development process. The ability to get fast feedback, adapt tests to new user flows quickly and quickly analyze the results, identify bugs and get enough information to quickly fix them. Proper planning can also save a lot of time and expenses. Suites that are not properly planned require high maintenance and keep the R&D team away from building new functionality and increasing coverage. Finally proper planning can help expand the responsibility for quality beyond the Q&A organization and let others contribute.
About the Author / Oren Rubin
Oren Rubin is CEO of testim.io. He has over 20 years of experience in the software industry, building mostly test-related products for developers at IBM, Wix, Cadence, Applitools, and Testim.io. In addition to being a busy entrepreneur, Oren is a community activist and and the co-organizer of the Selenium-Israel meetup and the Israeli Google Developer Group meetup. He has taught at Technion University, and mentored at the Google Launchpad Accelerator. Connect with him on LinkedIn and Twitter.
Trying to integrate applications on your own can eat your project alive. A seemingly simple project may look harmless when you start, but in the end do-it-yourself software delivery tool integrations can devour your entire software process.
The problem may be that building simple integrations appears easy. How hard can an integration be? But let’s look at what typically happens with integrations down the road using the example of one developer’s story.
A developer needs to send defects from Zendesk to JIRA, so he builds a one-way integration based on a workflow action (just a trigger). Soon enough, he realizes he is now managing two tools, and keeping them updated has become a big issue—a very manual process and a huge time-sink. So he designs something to support two-way updates. But that’s just the beginning; each tool changes every few months and an integration that seemed so simple on the surface becomes very complicated.
Then our developer tries to modify a simple JIRA workflow. Again, it seems straightforward, but it’s not, because the tools involved are constantly changing. There are complexities with workflows, and organizations undertake endpoint system upgrades. As a result, one change to the tool breaks everything he’s done. On top of this, it’s difficult now for him to know the hierarchy or how one item relates to another.
He starts to consider some of the challenges he’s facing. He needs to consider how the description is written in each tool. They are all different, so how does he make sure he conveys the information correctly every time? And each “simple” integration seems to create more pain points, creating auditability and traceability problems. Meanwhile, he has to manage other tools that are continually added to the mix.
He’s taken a look at both API and custom-tool approaches. He’s found that API integrations are brittle or they become too popular and then have to be maintained for an entire organization. But the alternative—custom tools—typically fall apart quickly.
And what happens when our developer decides to leave the company? There’s no one left who knows how to maintain the integrations he created. Meanwhile, new tool replacements mean someone has to build new integrations—and the puzzle becomes even more complicated.
This short scenario illustrates the unexpected challenges developers face in creating and maintaining integrations. The cost of building and maintaining tool integrations in an always-evolving landscape is deceptively high. And when an integration fails and the organization suffers critical downtime, costs (and tempers) skyrocket.
Many of the common pitfalls are readily evident, such as rich text problems, but the following three areas may be the most challenging to developers addressing integration issues and can’t be easily avoided:
Endpoint system upgrades and schema changes
Workflow rules and differences
Relationships
Endpoint system upgrades and schema changes: Much of the time, a different team from the one that developed an integration will perform the upgrade. Perhaps a new field was added. Regardless, simple changes to APIs can blow up an integration. API responses may change formats or the expected values for a field may change. When the team upgrades the system the URL may change, so avoiding duplicates is tough. So, too, is ensuring that the change is seamless. Changing some of the underlying features could result in unexpected behavior (and it will be hard to detect where the problem actually is). It may be that JIRA made API changes that caused custom select fields and comments to fail when using the “old way”; however, other features may continue to work as expected, making this failure silent.
Workflow rules and differences: Workflows are the complexities between systems. It is hard enough to reconcile teams, let alone the tools they use. Tools may require you to enter required fields, and this can constantly change. And workflows may not match up at all. How can you handle these differences without bouncing field values and allowing complex mappings to reconcile these? What happens when a system has a special action to change a status and fields are required at the same time, but the other system is open and doesn’t reflect this?
Because of these issues, it is critical to get the teams being integrated (e.g. testing and development) to discuss workflows and where the two teams align. These teams need to understand where workflows overlap so they can undertake proper integrations.
Relationships: These are not always apparent. As you grow and want traceability, you need to have synchronized relationships. How does this requirement trace to a test? How will you manage those relationships? The tools don’t match the way they manage relationships. Some allow one value, others multiple. Some allow relationships across projects and types, others are more restrictive. In addition to the rules of the system, how do you know what items to link once you have synchronized the artifacts? What happens if only one side of the link is synchronized or the items are not synchronized in the order that you expected?
Getting this all right requires intelligent systems that can look at one system and translate properly to another
Avoiding Integration Pitfalls
The first obvious step in avoiding pitfalls is to identify what they will be and when they’ll pop up. To do this, focus not just on the current problem, but also on the future. Consider the bigger picture, understanding that systems constantly change and teams work differently so there will be additional layers of complexity in any integration project down the road.
As such, my best advice is for development organizations to focus on the software their organizations are counting on them to deliver, not creating and managing tools to build that software. Put internal resources to work in IT areas where you can create competitive advantage for your company. And consider using third parties to address issues such as integrations.
About the Author / Shawn Minto
Shawn Minto is the senior director of Technical Services and Support at Tasktop and one of the original team members working with CEO and co-founder, Mik Kersten, on Mylyn prior to Tasktop’s founding. Shawn began his career at Tasktop as a software engineer, working on all aspects of the product and directly with our customers to support their needs. Currently he works with customers to understand their tool environments and offers suggestions on how to best leverage Tasktop through deployments and support. In his free time, he enjoys being outside, cooking, and renovation projects. Connect with him on LinkedIn and Twitter.
The fact you’re reading this article on devopsy.kinsta.cloud suggests you’re already a DevOps fan. Or maybe you’re visiting this site because you’re considering moving to DevOps in your organization, or you’ve been told you must move to DevOps and are figuring out how on earth to do it.
Whatever your view on DevOps, let’s get my position clear from the outset: I am a big believer in the power of DevOps to deliver continual innovation to an organization. I advocate DevOps in every project I do because I believe those organizations that innovate and change the fastest become the leaders in their sector, and since almost every business innovation relies on a technology innovation first (“every company is becoming a technology company,” after all), DevOps is the best way for an organization to adapt to, and benefit from, new technologies and gain a lead in the market.
However, as anyone who has managed the transition from waterfall to DevOps will know—or, in fact, anyone who has overseen any change management project will know—the transition is rarely a smooth one. So, what is behind this reluctance to innovate and work in new ways?
First, the Technical Arguments
At some point, most organizations have been burned due to a lack of quality control in technology, whether a security breach, bugs or implementations/changes that went horribly wrong. We all know stories like these. They can have an immediate and catastrophic effect on the organization’s ability to do business. Therefore, there has been a tendency to inhibit change in favor of “stability” and known entities, and to adopt a “Better the devil you know,” strategy.
DevOps has rocked this view of the world, and teams are reticent to move away from tried and tested approaches and processes because they are held responsible for operational stability. In addition, those people in a quality-control role believe their very purpose and job is under threat. There is also a view that DevOps is system- or application-centric, and can introduce risk to an interconnected IT estate.
While these are all valid concerns, it is worth bearing in mind they are all technical arguments against DevOps, and they have all been countered before. Implemented correctly, DevOps does not jeopardize stability or undermine quality (in fact, many would argue it does the exact opposite), and it does not undermine a person’s role in development or operations, but rather liberates them from their silos.
Culture is the Real Reason
From my experience, innovation of any kind cannot be held back by technicalities alone. While the technical arguments are certainly valid, in most instances, the reluctance to adopt DevOps comes from something much more basic: the organizational culture borne from natural human behavior. Most people are simply risk-averse and resistant to change, so the resistance to DevOps is just a manifestation of human nature.
The only way for DevOps to succeed is to change the organizational culture to be more open to change and to adopt an “innovation mindset.” The change must also encompass the whole organization. If you only bring half of the business with you, the innovators will become frustrated and find ways to circumvent processes and policy, resulting in additional operational risk and shadow IT.
The influence that senior managers can have in facilitating—or undermining—the move to an innovation mindset must also be factored in. It is common for senior management to inadvertently foster a hero culture, where those individuals who get things back on track after a catastrophic event are rewarded, yet the well-structured and “stable” DevOps deployments are not given the same weight or focus. Again, it is human nature to be drawn to the dramatic event than the non-event. But you must ask yourself: As exciting as firefighting is, is it a sustainable way to run a business?
Instilling a new mindset into a business is the same as any other transformation project. It requires you to tackle the four elements of process, organization, technology and information to bring about meaningful and sustainable change. To bring DevOps into your organization you must:
Align operational teams with development teams, embedding where possible for early involvement of Ops.
Ensure that operational/non-functional requirements have the same weight and priority as functional requirements—no user story is complete without a stable and secure operational environment.
Automate where possible to reduce human error and increase the speed of delivery.
Improve knowledge management to reduce “hero culture” or bottlenecks where knowledge is concentrated across a few people or one team. Never make exceptions for “heroes”; you need their support more than anyone else because they hold the biggest sway over the rest of the team.
Have accurate and comprehensive configuration information. Having a good understanding of the IT estate (including interconnectivity) can help move an organization from a one-size-fits-all governance model to a tailored risk assessment approach through better understanding of the impact of changes.
DevOps often fails when people can revert to their default behavior. Only by actively identifying and supporting a project in a DevOps manner (considering process, organization, technology and information as previously explained) will you be able to change to a new “business-as-usual” innovation mindset and behavior. But this needs strong sponsorship, guidance and patience.
DevOps must become the culture of the organization, driven from the top as well as the engineering teams, and people need to understand the implications for their role and responsibilities. CEOs and leaders need to actively support and drive the change, while providing the necessary working environments (tools and training) for those people who will be held accountable for operational stability. It is no good investing in technology and training for engineering to innovate if your ITSM functions are not brought along with the change, for example. But if you bring everyone with you, and address all concerns along the way, you will soon be reaping the rewards of the DevOps approach.
About the Author / Romy Hughes
Romy Hughes is director at Brightman. Romy is a business change specialist with expertise in service integration and management (SIAM) for managed services, business transformation and IT service management (ITSM and ITIL). Connect with her on LinkedIn.
In the technology world, we are in a time of incredible disruption and innovation. Many years ago, my friend and “DevOps Handbook” co-author John Willis predicted that a decade from now, academics will call this period the “Cambrian explosion” — it was an incredibly vibrant 25-million year period that resulted in an incredible diversification of cellular life forms, including the appearance of complex, multi-cellular organisms, setting the stage for the diversity of life we see today.
Similarly, over the last decade, we’ve seen an explosion of incredible innovation and disruption in almost every area of technology: cloud computing, containers, virtualization; Agile, iterative development, continuous integration and deployment, and automated configuration management.
One can be forgiven if all this change is overwhelming; I think we’ve all had moments in our careers where we think, “I’m not sure if I I’m up to learning how to tackle this next big thing.” Even someone who loves learning can become cynical, or maybe just fatigued by change.
But most of us, maybe with some calm reflection, will recognize that there’s never been a more fun time to be in technology. And many of these new techniques and practices lead to massively better ways of working, where after an initial learning curve, we’d never go back to the old way. I believe this actually calls for excitement! And, I’m not alone.
We recently hosted a live chat with multiple leaders in technology who’ve spoken at past DevOps Enterprise Summit events. As we are in full prep mode for the next event in San Francisco this November, we asked our esteemed panel to share about their career journeys in tech and their passion for learning and change. It was a great pleasure to be able to pick the brains of Dominica DeGrandis from LeanKit, Dr. Nicole Forsgren from DORA, Erica Morrison from CSG International, Terri Potts from Raytheon and Robin Yeman from Lockheed Martin. During the course of the discussion, each person shared how they got their start in tech and many of the lessons they have learned over the past 20 years in the industry. It was echoed throughout the discussion that while things are rapidly evolving, it’s an exciting time filled with continuous opportunities to learn, plenty of room for growth and time to experiment with new tools and processes. For the full video replay and transcript of the discussion, please see below.
Also, don’t forget to register for the 2017 DevOps Enterprise Summit San Francisco to hear more from leaders in tech and get a glimpse into their organization’s journey into DevOps. Early Bird pricing discounts are available until July 31!
Gene Kim: Hello everybody, welcome to this episode of the DevOps Enterprise Summit live chat. We are assembled here today with a group of people I’ve known for years, and people who I genuinely admire. They’re all experts and leaders in the DevOps Enterprise community who are helping shape much of the practices that are going to almost certainly be commonplace in technology organizations over the next decade. Over the last 20 years I’ve been studying high-performing technology organizations, and more recently it’s led to studying transformational leaders and trying to understand how they became transformational leaders. What I find fascinating is there’s evidence to suggest that these are traits that you are not necessarily born with, but instead they’re actually learned.
Over the next hour, my goal is to learn about these leaders on this panel, their careers and experiences that shaped who they are today. My desired outcome is that we’ll all learn something that we can integrate into our own respective journeys. As you’ll surely find out over the next hour, these are all great people. As a reminder, the upcoming DevOps Enterprise San Francisco event is taking place on Nov. 13-15, and for more information visit events.itrevolution.com. And just a reminder, Early Bird pricing ends on July 31st.
Let’s go quickly around the panel, and everyone, if you could just introduce yourselves, name, company role, when did you present DevOps Enterprise and what did you speak on, and free to speak to share any professional accomplishment that you’re most proud of. Dominica, let’s start with you.
Dominica DeGrandis: All right. Well, I was a build engineer, spent 20 years bringing visibility to what version of what file was on what server in what environment. Currently, I am the Director of Training and Coaching at LeanKit. I’ve spoken at DevOps Enterprise Summit for four years in a row now, mostly speaking about experience reports and metrics and making things visible. And I guess, my most professional accomplishment, I think it would have to be finding my courage to get up in front of a group of people, and explain or present data, and demonstrate how to get leadership buy-in with metrics.
Kim: Awesome, thank you. Erica Morrison.
Erica Morrison: Hi, everyone. I’m the Director of Software Development and Operations at CSG International in Omaha. We’ve spoken the past couple years at DevOps Enterprise Summit chronicling our journey through the DevOps space, and we’ll continue to tell that journey. Something that I’m most proud of, probably the things that have brought the most challenges in my career have actually been in the DevOps space. We’ve undergone a fairly major transformation this past year, brought a number of teams through that, but one team in particular who have made substantial improvements as far as stability and automation. That’s been really something that’s been fun to be a part of.
Kim: By the way, following your experience reports has been my favorites among every year at DevOps Enterprise, fantastic. Robin.
Robin Yeman: Hey, I’m Robin Yeman, a Lockheed Martin fellow. I’ve been working in engineering about 23 years in the defense industry on everything from submarines to satellites, and everything in between. I think my biggest accomplishment is really being able to see the change. When I first started doing Agile in DevOps, you really couldn’t see it, but now I start getting contracts and RFPs from the customer repeating verbatim what I told them before. It’s cool to see that the influence is there. It might not be as fast as I’d like it, but it’s there.
Kim: That’s awesome, and then you spoke last two years of DevOps Enterprise San Francisco and London.
Yeman: You’re right. Yeah, the last two years on transforming large legacy organizations way back. A combination between Suzette Johnson from Northrop Grumman and myself from Lockheed Martin, because we both had very similar journeys.
Kim: Awesome. Thank you, Robin. Nicole.
Nicole Forsgren: Yes. Hello, I’m Nicole Forsgren, CEO and chief scientist at DevOps Research and Assessment, otherwise known as DORA. I spent years doing software development, enterprise storage at IBM, and then I went and got my Ph.D. that might be my professional accomplishment that I’m most proud of is that I’ve been studying and researching what’s basically known as DevOps for the last 10 years, and then applying it and bringing what I found to the industry and in my work with Gene and Jez Humble, and the team at Puppet on the State of DevOps Report. We all lived through that 2000 dot-com bust, and we saw many companies fail and I don’t want to see that again. Really helping to make an influence in the industry and make things a little bit better is an accomplishment for me.
Kim: Awesome, thank you Nicole, and you spoke in I think every year at DevOps Enterprise. Fantastic. Terri Potts.
Terri Potts: Hi, I’m Terri Potts. I’m from Raytheon Company, and I’m the Technical Director of the Software Development Organization. Like Robin, we are focused on doing mostly defense- and Intel community-related contracts. I started in the software development business actually in the Air Force and spent 11 years there, moving through the ranks and came to Raytheon, and moved on to be an architect after doing development for several years and have been a champion for technology transformation since the early 2000s.
I’ve spoken a few times at DevOps Enterprise Summit. Once about our legacy transformation since we have been around for long, and a lot of our code is quite old. Last year, I spoke about how you build up your staff of people that can support your DevOps transformation because it’s hard to just go out and hire those, some ideas around how to go do that.
Kim: Awesome, thank you much Terri. And by the way, I remember meeting you at the IBM Interconnect Conference in 2014, and you were actually one of the first announced speakers at DevOps Enterprise, very grateful for that.
First, let me just state an assumption I have that there’s actually never been a more fun time and exciting time to be in technology even though things are rapidly changing. Actually the pace of change is what’s causing much fear for many professionals about our livelihoods, our vocations, but I genuinely believe that the best times are ahead of us not behind us. Do you believe that? And if so, why? Erica could we start with you?
Morrison: Sure. I think about when I came out of college with a computer science degree, and I was focused on writing one language and writing that language well, and now we’ve got the proliferation of numerous different technologies, different languages. Like you said, the pace is moving much quicker. We actually hosted a hackathon here. We’ve done two of them for college students and just a number of different technologies that they use was absolutely mind-blowing to me, but I find it extremely interesting to continue to evolve for all these different things. It’s not just this one little dedicated space anymore, it’s constantly learning these new things which I love to do. Its new challenges everyday which keeps work interesting for us, makes work about being lifelong learners. I love that continuing to learn throughout our career, and it’s just focusing on learning different things now.
Kim: Awesome. Robin, how about you?
Yeman: I absolutely agree. I’d really think it’s the most exciting time. Just like Erica was saying, the college kids are coming in with 15, 20 different languages, they’re bringing all kinds of new technologies, they’re bringing gamification. They’ve really brought some different leadership styles. It’s very exciting. I know that some of our engineers get nervous, but if you just grab on and join the ride its fun.
Forsgren: Honestly, that’s one thing I love about technology. I mean, it kept me engaged, it kept me excited. Back when I was doing a bunch of development work, there was always something new to work on. As you said though, there’s always that other piece. There’s always the other side of it. We’re working in systems with growing complexity and that does make it really, really difficult, and that’s something that we’re seeing when we work with companies with DORA, in our assessment. You have to make sure that you can focus, but you need to know what your constraint is, particularly when we’re working on technology transformations. We need to make sure we can focus on the most important thing. We need to make sure that we can limit our whip, we can focus on our constraints, we can identify what the most important thing is that we don’t have sprawl and just contribute to technical debt. But, I think that’s what makes our work challenging and exciting and meaningful.
Potts: I love change, and that’s why my career has taken me where it has because the more different things you get into I think the more rounded you are, and the more you’re able to adapt new technologies. And the way I see it today is all the technology changes especially with the automation that many of us are going through that transformation, and I have always seen that as drudgery. If you’re working in some factory, let’s automate all the drudgery out, and then that gives us time on our teams to explore more around analytics or machine learning. That’s exciting for me.
Kim: Yes, you make it sound exciting. Dominica, you’re smiling.
DeGrandis: Well, I’m just thinking back at my first job out of school after I got my computer science degree. It was more configuration management back then, that was getting everything on the mag tape from IBM and putting that in a library, physical library somewhere safe in order to deploy it into production. But I think for me, it’s just the amount of opportunity there is now. There is this massive amount of opportunity for everyone whether they have a degree in computer science or not to go out there with all the sharing. The ability to share and learn now is much greater than it has been previously.
Kim: That is true. I mean, there’s never been a better time to be a learner. There’s no cost to entry it seems these days.
DeGrandis: Absolutely.
Kim: I suspect that this isn’t the first time that you’ve seen this dramatic change in the industry in your career. I’d love it if you could tell us about that time and how it affected your thinking about how you viewed your skills, your career, and where it took you. Robin, would you mind starting?
Yeman: Sure. When I first started we had a lot of ToR trend and ADA going on, and I was worried about the next generation of ADA, and whether they would even be able to keep up. So, just like that Dominica and the rest of the folks have said, I spent a lot of time at night trying to study, make sure I could stay on top of it because I was working with these amazing engineers and I just didn’t want to fall behind. I do the same thing now.
Kim: And by the way, I mean in a scale of 1 to 10 what level of fear did you feel? One is like it’s just another book no fear, 10 is existential fear of, “I might not be able to finish the book and therefore my career is over.”
Yeman: I work with some engineers—they’re smart that I was terrified. I’d say way up at 8 or 9 out of 10 because some people just get it quick and I have to work at it—I’m just not a natural. I was terrified.
Kim: by the way, it reminds me of learning Clojure very recently. It’s my first script language and I think I had to do 40 hours of reading before I could write a single line of code. It was just, it was very challenging, but very rewarding but all very intimidating. Nicole, how about you?
Forsgren: My first job in tech was when I was still in college, and it was on a mainframe. I was coming out of my associates in like ’99, and that was the big Y2K, everyone’s panicked, and it really fundamentally shaped what I thought about tech in many ways. Everyone was panicked, there was this big dot-com bust, and I alluded to this earlier, “Survival isn’t mandatory, companies can fail.”
That’s one piece where it really shaped what I do is that I don’t want to see companies fail. I’m sure you all have talked to me at conferences, and suddenly I just dig in, and I’m passionate. I have this drive to make sure we know we can find ways to help companies not fail, but it fundamentally affected me in a different way in that in the late ’90s, there was almost a tactical way to address the problem and it was teach everyone fundamental things. People look at me, they’re like, “How do you do mainframes?” It was mainframe, COBOL, Fortran and that was all we were teaching many students because we desperately had a very tactical need to make sure that when we hit 2000, the world didn’t end, technologically.
And then I did a bunch of consulting, and then I got a Ph.D., and then I was a professor. We’ve almost taken this other stance of we only want to or we’re very purposeful now, and I think it’s led to complexity sometimes in technology, but we want to make sure that we understand the purpose behind what we’re doing. We want to make sure that we have flexibility, we’re strategic, and we aren’t just tactical.
So, we designed the State of DevOps reports. They’re vendor and tool agnostic. We tell you about the principles that are important, the architectural designs that are important, the principles behind using version control are important. We aren’t going to tell you to use Git or Subversion, we’re going to tell you to use Subversion control. That’s the other thing that’s really impacted my career is that I need to make sure that if I walk into a situation I can tell someone from a conceptual level what’s really fundamentally important to improve their technology practice, and it won’t be tied to only a specific technology because that can end up being limiting. If you only tell someone teach someone a language, but not the fundamentals behind that language, it won’t work.
Kim: Principles are timeless and those things are not. Awesome, Terri.
Potts: When I think about this, the thing that comes to mind for me is probably about 20 years ago when systems software architecture started becoming a first-class citizen, at least at Raytheon. I remember seeing the first book, and one of my mentors saying, “Hey, I’m going to have a study group on software architecture do you want to join?” And I wanted to join and started seeing it as a way to grow my career, and be able to influence bigger parts of the system, and try to make things more maintainable, and really make good decisions going forward instead of just sort of stumbling into things. I wouldn’t say I was afraid, but I was really curious, and certainly felt like it was something that I needed to keep up with and did a lot of studying on my own outside of the office with study group, learning everything I can from them.
Kim: Sounds very interesting. In this case, it was like a mentor who actually sort of directed you down a certain direction. There seems to be a pattern of having great mentors. Fantastic, Dominica.
DeGrandis: When it comes to dramatic change, I have to say hands down 2005 or 2006. I was working at Corbis at the time managing environments, and we went from building out maintaining 25 servers to building out maintaining 250 servers over a period of 9 to 12 months. It was just impossible to manually build out servers. We had to start automating, and developers were like, “Builds take too long.” They wanted to put their own changes in their own dev integrated environment, and they took it seriously. They said, “Okay, then you get to own that dev environment like you build it, you maintain it. Then, I’ll just go build out maintain QA and staging inner prod.” That was huge dramatic change I think, and just being able to adapt to that and quickly was the biggest lesson for me there.
Kim: And specifically, was it dramatic because it was such a different way of doing things or it was dramatic because the role change and responsibilities for you, or what made it dramatic?
DeGrandis: Because for the previous six years we have had this process where we just dole out responsibilities, who’s doing what and how we’re doing it, and now that was turned on its head because we had to deliver faster, we had to deliver better quality, we had to get these environments up and running, and everything just exponentially grew with the number of servers and the number of environments. The way that I had been working for the last six years which had been fairly successful was completely different. That’s the second law of thermodynamics, everything is always in constant change and motion.
Kim: I love it. This is the famous story that’s in David Jay Anderson’s book, and in your upcoming book, Dominica. Fantastic, and Erica your dramatic changes in a certain point of your career.
Morrison: In addition to the ones that have been mentioned, there’s a couple that come to mind for me. First of all, the introduction of Agile. I started my career very waterfall. I probably wrote code for three to six months before I even chucked it in to the point that other people were using it which now is obviously mind-blowing to us, but that’s how we did software development back then. That’s been a major upheaval in pretty much all of our processes is around how we do software development.
I think overall, all these items point to the fact that software is constantly evolving, it’s not a static thing. When the next thing comes out we’re used to this evolution of change. It’s something that as developers, we just accept it, and look to learn the next best thing and how we get to evolve and continue to get better.
Kim: And physically what’s different or dramatic about the Agile transformation from your perspective?
Morrison: Pretty much everything about it, just the concept of batching work up into small little pieces. I’ve participated in helping roll out Agile with other teams, and it’s somewhat mind-blowing to these teams that you can take this big amount of work and you can even slice it up into one week or one day increments, and chuck that in and have this daily accountability with my team members. That test is now my responsibility and how am I going to automate that because as a developer, we want to automate that test as much as possible. It’s just much more of this integrated, fast-paced, all of us owning all the work and focusing on flow through the system.
Kim: I do remember that it was around 2005, we had just started our first experiments on Agile and even finishing something in a week was mind-blowing. That’s how long it took us to fill out the PRD forms usually. I love these amazing stories that you’re sharing. If you could compare yourself now to the person you were 20 years ago, what were the most important and significant differences between yourself now and yourself then? What skills did you have to develop? Nicole, let’s start with you.
Forsgren: Twenty years ago, as you think back, there’s no Facebook, there’s no Twitter, there was barely email. It existed, but not really. I mean, definitely all the technical skills, but for me learning how to be a leader and how to interact with people, and how to exert influence in a soft power sort of away.
Again, going back and talking to people that I knew 20 years ago there was learning how to do things in more of a nuanced sort of way, which you need to do when you’re managing large projects. For me, it’s been a soft skills thing. I always joke that technology is easy and people are hard, even as complex as technology can be. That was probably my biggest piece of learning.
Kim: In many parts what we work on, they’re all volunteer projects. And even the softer skills are necessary when everyone’s volunteering their time. Great, Terri.
Potts: Twenty years ago I was finishing up my last year in the Air Force and doing some leadership things, but much more as an individual contributor. I have a good view of what it was like to be on the Air Force side managing big contracts, and testing systems and operations. But since then I think that there have been a few things that I really had to develop. One was my network, the group of people that I learned from, and that has definitely grown in the last four years outside of Raytheon. I had a pretty good network within Raytheon, but once I met Gene and started participating in DevOps Enterprise Summits and the Forum, I just love how that career or that network has grown from there.
I’ve also always been interested in really staying with modern technologies and finding ways to make our teams more efficient, I think that’s always going to be something that’s changing. You always have to stay on top of what the latest thing is that’s helping your teams be more efficient. And the Agile transformation too because definitely we were waterfall all the way in the Air Force, and for the first several years that I was at Raytheon.
Kim: That’s interesting you mentioned network. One my favorite quotes is, “You’re only as smart as the average of the top five people you hang out with.” But I think what is interesting is when you can actually bring that network to help other people with their problems. It becomes a really powerful asset and the ideas of everyone.
DeGrandis: Well, I think 20 years ago, I assumed that everybody else knew what they were doing, and that if they were doing the right thing and I intended to feel like I didn’t know what I would. I didn’t know what was up, I didn’t know what I was doing. I was just inundated with trying to learn quickly. And now I recognize that most people really don’t know what they’re doing. Nobody has all the answers and it’s absolutely okay to ask questions, and that in order to have your voice heard and become the voice of reason, how important metrics have been in order to present them in a way that others can understand.
Forsgren: Plus one on metrics. I will cheerlead for metrics all day long.
DeGrandis: I used to rant we don’t have automated testing, smoke tests take too long, and it just has got me nowhere. Now being able to present data calmly and confidently, to be the voice of reason in the organization is one of the greatest learning changes over the last 20 years. That and learning much more about Flow and Lean and Kanban and different methods out there—it’s why I wrote my book, Making Work Visible to help others see how to become the voice of reason. I think that’s super important one.
Kim: Awesome, we have a calm voice of reason. Excellent, Erica.
Morrison: I’ll echo the comments made about people and relationships, and, I think in college we don’t get an understanding that there’s nontechnical problems in addition to the technical ones. Really, we all look at the world differently and we’re going to have to understand that everyone that we work with isn’t going to have the exact same perception as us, and we need to partner together and get these diverse viewpoints, build these relationships, build these networks. That’s something that I probably didn’t get proper appreciation until maybe the midway through my career journey far. And I just wish someone had told me that earlier in my journey.
And then, I think appreciation for the ecosystem that we operate in I talked about that when I first came out of school as a computer science graduate. I was focused on my view of the world and the code I was writing, and I thought the big picture was understanding what the rest of the developers on my team were working on not understanding how do we deploy this code, how do we maintain it, what’s the infrastructure that we’re running on, how does our customer use this code. All those different sorts of things, which I we work on now to get that bigger picture for people, but that appreciation that that bigger picture makes you good at your specific piece of that. And then just that need to continuously improve yourself. It’s not about just being good with what you’re doing right now, it’s about looking at what’s next, how can I continue to evolve myself and improve myself that I can help move our company along.
Dominica: Well said Erica. Hear, hear!
Gene: I can resonate with all elements of that, but just to share a story. I was talking to a younger developer, he must been like around 25 hanging out with a bunch of ops people and I asked him, “Well, what are you doing here hanging out with a bunch of ops people?” He said something that I thought was hilarious, maybe not the most politically correct, but I couldn’t fault his curiosity. He said, “I just want to know where my code goes. To me it’s like I flushed it down the toilet and I just I’ve been doing it for two years, I just wonder where it goes.” Which I thought was unfortunate imagery, but man he was like, “Well I’m the minority of developers actually caring about the system as a whole.” Obviously, that made an impression on me.
One of the things that, if you go back to the topic of change, one of the things I hear a lot these days is how people in our profession often feel very uncertain about their own future, and specifically about how to maintain their own relevance. I’m thinking a person I met that I was actually able to interview on my blog, and yeah, he’s my age and despite having an MBA, despite having a Master’s in engineering management, despite being an expert in the theory of constraints, despite being an operations lead at Boeing, he wrote asking for career advice saying, “What do I do to make sure that I’m as relevant in five years as I am five years ago?” Specifically, what advice would you give someone who’s in operations and feeling like if they don’t take drastic measures they won’t be as relevant, and what actions would you recommend? Terri, why don’t we start with you?
Terri: I was thinking about this question, much of it depends on where he wants to take his career you have to know where you want to go, in order to put together a plan for how to get there. Given all of his degrees, and what he’s doing, it doesn’t seem like he really wants to go off and start re-architecting software or it didn’t sound to me like he would necessarily need to get a whole computer science degree. But if he wants to really understand how the automation works maybe he takes a few classes in Python or Ruby or one of the scripting languages to help with that, and get involved with that. But it seemed like it would be pretty drastic to go out because there’s lots of opportunities to lead transformations and he’s got those leadership skills apparently. I would say, “Look for the opportunities that really fit with your personality and your skills rather than going back to school for the answer.”
Gene: Indeed, and by the way, I just realized, sorry Robin! I didn’t forget your answer—could you say what advice would you give yourself 20 years ago, and what skills are most important? Then we’ll get back to our friend Pat Birkeland. Sorry, Pat.
Robin: Just like the others, I would say people and people skills. I was really type A and really driven building my code, and I thought the harder I work the more impact I would make, but as I got older I learned really the more that I build people around me the bigger impact I make. Being able to build those co-workers or people that work for me actually magnify that tenfold, and it just didn’t occur to me at that time. I would really focus on what you share. We tend to get rewarded for what we know, but if we rewarded people for what they share, you’d be amazed at what can happen.
Gene: Thanks Robin, and one of the things that I found impactful about your presentation even just in this last London event was you shared the story about how you had created this very informal community of people just sharing, and you met weekly or monthly, and 15 years later you’re still meeting, and then you’ve all sort of ascended together. Am I misremembering that story?
Robin: No, no, we’re actually still meeting and we’re building people. I think both me and Suzette [Johnson] are in the same boat because at the same time we were we were looking for others that would help us because we’ve started Agile pretty early in our careers, and we really needed any information. There just wasn’t a lot of like-minded groups to get together and build each other.
Gene: Awesome, and I guess if I were to sort of speculate that all are sort of very curious, all ambitious, all good at networking, and what a great collection people to learn from and collaborate with.
Gene: Awesome, thank you Robin. Let’s go back to this non-fictitious Pat Birkeland, he was generous in writing up some of his thoughts and experiences. Terri, your advice was the answer is not to go back to school, but maybe more crystal clear about where he wants to go in his own career and find opportunities. I love that. Dominica, your thoughts on Pat Birkeland.
Dominica: First of all, I could definitely relate because I had the same thoughts myself earlier, “Man, I should go back and get my Master’s. I need to. I just need another degree, make more letters at the end.” But really can’t do that because of working full time with little children at home. It’s just really hard. I didn’t have a nanny. I think that what somebody can do immediately is to learn the business better. Maybe just brought it again by a bit from the business perspective and see what the business challenges and risks are. And then, figure out how to reduce some of that risk maybe by making things a little bit more predictable to you. That’s going to be such a huge growth experience to go around, and maybe visit some of the other parts of the value stream and go look at the whole value stream, and see where things are stocking. And you got making that fall in faster.
Gene: That’s such an interesting piece of advice. And by the way, it reminds me, one of the things I was always envious, if I had to do my career over again, I would have loved to spend a couple years at a big four audit firm just because I was envious my friends who could actually see the big value statement, procure-to-pay, order the cash. It was something I was envious about, just even the language they spoke in. That totally resonates with me, Dominica. Erica.
Erica: One thing that I’ve told a number of our operations personnel that I’ve had this similar conversation with is “Good people will always find a way to be successful.” I’ll hire good people any day, and they’ll acquire those new skills. When it comes to being nervous about the situation I think as long as people are eager and willing to learn new things, they’re going to be successful. And then we as companies need to set them up for success. So they’re just personally able to talk with their manager and see if they have specific areas where they see that there’s opportunities for improvement. And then like Terri said, find your passion and invest in that particular area.
Gene: There’s no shortage of problems to be solved, I think is what you’re suggesting, Erica?
Erica: Exactly.
Gene: Yeah, that’s actually great. Pat, I hope you’re listening. I’ll have this transcribed for you. Robin.
Robin: The first thing I did when after the DevOps Enterprise Summit in London is to come back and tell everybody about Nicole’s research which stated that, “Transformational leaders really improve the likelihood of your success.” We’re short on those, I would think being able to build those folks around them and really drive, help the technical folks get things across to understand the value stream as Dominica said. I think there’s plenty of work to be done there, and that would be amazingly helpful.
Gene: That was great. In fact, I mean I do love the literature that says these transformational leadership traits are learned behaviors not that the old school was like these are things you’re born with. The great man theory, that you had to be born that way, otherwise you’re just, yeah, you’re just not wired that way. I think that’s exciting. Nicole, on Pat Birkeland.
Nicole: Yes, I would start by asking in an ideal world right, what would Pat be doing in six months, and in a year? And then, work backwards. What’s your ideal job? What do you want to be doing? And then what skills do you need? Because again you take a look at what he’s been doing, look at his credentials. He’s got an MBA, he’s a theory of constraints expert, and he has a master’s in engineering management. It sounds like he’s setup to be doing incredible things right now in leading a technology transformation. If he wants to be doing that, he might need to brush up on one or two specifics of a new technology in order to like really be able to speak to the developers or the operations staff on a few of his teams, but he might be there.
And it’s funny I say that especially as I was listening to what Dominica said she wanted a master’s so she could add letters. Some people walk up to me and they’ll say, “I want a Ph.D.” because I have a Ph.D., and I’m like why? And they look at me like like I want to be the only one at the Ph.D. I’m like, “No, like why do you want a Ph.D.? Don’t get a Ph.D. just for fun.” A Ph.D. is a research degree. If, for example, if Pat says I want to be doing research in six months or a year it’s going to take more than a year, then get a Ph.D., but if you want a Ph.D. because or if you want to degree because you want someone to know you have a degree that’s not the reason to get a degree. But if the job that you want in six months or a year is to be writing code, okay, then yeah then you’re going to need to either get a degree or do some Coursera or some Udemy or something. But if what you want to be doing in six months or a year is leading transformations, you’ve got it, you’re set.
Gene: There’s many easy ways to learn these days and Ph.D. is just one of them. By the way, if I could just go back to this and this more for me, but I remember a point in my career this must have must been, must be about 2005. And, I hear all these talk about “It’s easy to learn things, yeah, rah, rah. Change is good.” I remember when Microsoft changed their database interface – it would change from ODBC to ado.net, and the reaction I had was like, “This is crap. I do not want to learn another way to access database. This is horrible.” I had no appetite for that. I mean, sure you might have had encountered people who had that reaction, what advice would you give me in 2005? I’ll take any advice from anybody, otherwise I’ll call on.
Dominica: I’ve had friends, developers, who have been extremely frustrated at the continual change in the learning. But if they feel like they learned something really good, and then it just all changes again, and how frustrating that was for them, and it made them want to go be a business analyst. But, I think that is just the nature the demand. That’s the reality that we live in, and that there’s a need, we have to learn and adapt and changes are always there.
Terri: Right, I’ve had interesting reactions from people in our organization who seem to think that learning something new is going to threaten their job right like if you move to automated test then, perhaps they won’t have a job anymore. It’s always surprising to me when people don’t want to learn because learning is sort of what keeps me motivated and excited. Although sometimes, you feel maybe it’s a step too far beyond what your current skills are. But, yeah, there’s lots of, there’s lots of ways to do that these days and to figure out. And worse, if worst comes to worst get a few of your colleagues or friends that are in a similar boat, and you all want to learn about the same thing and you go through a course or a book together and maybe a group effort might make it a little more interesting, less intimidating.
Gene: It always helps to have a buddy is what I learned. Any other advice for Gene 2005 who is enraged by ado.net?
Robin: I would just say that it’s constantly changing, and that if you’re if you’re enraged by it you’re just going to be unhappy all the time. It’s almost like it’s just a given. You might as well try to learn to like it or pretend you do.
Gene: Awesome, Robin.
Dominica: Do you think though that there’s one thing that hasn’t been talked about yet, is this sense of fear that people have. They were really good at something and they knew this technology and they were at the top of their game, everybody went to them to ask advice on this one thing and now that’s changing. That’s scary for a lot of people like “Wow, just a year ago, I was awesome at this and now everything’s changed and what the heck do I do now? I mean look at some project managers now with the whole coming out of waterfall and PMI and Gantt charts. And now all that’s being shifted upside down and where are they going to go now?” I think addressing that fear of what people used to be really good at and now that’s gone is real.
Terri: Those are really good points Dominica.
Gene: And by the way, I had no illusions that I was good at ODBC. I mean, I was barely competent. I felt like their people are just randomly shuffling chairs just to force people to upgrade from Windows 95 or whatever it was just, I saw no value in it. Yeah, but I loved that it is a fact of life and something genuinely miraculous. It’s like, who would want to go back to the old way?
We did a webinar. It was on the topic of how to get your DevOps Enterprise submission accepted to the conference, and one of the people who was on that panel was Margo Cronin. She was an Enterprise Architect at Zurich Insurance and she’s now at AWS, but we actually accepted her through the CFP process. It was fantastic to see her perspectives and the advice that she gave, but one of the things that she said really made an impression on me. She said, “Have confidence that you have an interesting story to tell.” I think she even said, “Just because you’re not the lead developer or using the new cutting-edge technologies that’s not necessarily as interesting as someone who’s helping drive behavior change and transformation in large organizations.” Do you agree with her? And if so, what advice would you expound to anyone who has a story to share, but maybe are hesitant to. Let me call on you Dominica. And Dominica this is great because you’re on the programming committee.
Dominica: I hear this a lot! People don’t feel like they have a story to share, but everybody does. It’s a different perspective that they can bring and there’s a lot of similar nuances that we all go through, but I think to hear, especially somebody that we haven’t heard from before, a new fresh perspective and their challenges always makes for an interesting story.
Gene: Maybe you can shed a little light outcomes of the process right now. We’re looking at every one of these submissions and we are looking for interesting stories.
Dominica: Yeah, how many submissions did we have for San Francisco?
Gene: It’s over 250 almost 300.
Dominica: Yeah, going through all of those is a challenge and one thing that I really look for is original, real wording of people telling a real story and not shying away from using their own language. I don’t think people feel should feel that they have to conform to using the book-written perfect language in a submission. Just say whatever—what’s the problem, why it matters, what they did about it. The real story that’s what I want to hear what worked, what didn’t work, what was some of the failures, and where they are right now, and where they hope to go.
Gene: Great. Yeah, the lull of deliberation. In fact, one of the things that maybe most people might not know is just the level of the intensity of some of these debates of who we accept in the conference. I mean is that something that gets very, very heated. I’m not sure it was actually hung up and out of the meetings, but sometimes very close. Erica, how about you?
Erica: So, yes I actually had a conversation with a co-worker of mine after we attended a local DevOps meetup here, and she said, “I would love to at some point work towards a goal of presenting at this [event].” And she actually comes from a non-technical background and she has participated in a lot of the same things that I have in our DevOps journey and I said, “You know, I think it would be interesting to hear the same story told through your eyes, and I think it would be a different story than my version of it, and just the different things that stick out to you and were amazing to you.” I really encouraged her and I said, “I’d like to hear this story.” Even just the two of us telling that same journey, different perspectives on that are very, very interesting.
Gene: And are you referring to the portion of the presentation you gave last year, which was when a dev manager takes over ops?
Erica: Yes, yup.
Gene: I would love to hear that story. In fact, three of your colleagues actually did submit through the CFP. I’m eager to see where that goes. Thank you for any encouragement that you gave them. Thanks Erica. Robin.
Robin: Just like Erica, I think everybody has a story to tell, and through another person’s eyes it’s a whole different perspective. Sometimes the things that we do every day seems commonplace. That happens to me all the time, but then, when you tell somebody else they’re like, “Wow, I didn’t even know that existed or how would you even do that.” I think, again, just having confidence in yourself and realizing you actually have something unique to sell other people is what you need to do.
Gene: Interesting, thanks. Nicole.
Nicole: Yes, sometimes when I think about what might be a good topic to talk about when I go to conferences, I think about things that I do pretty commonplace or things that are pretty easy and think about what used to not be super easy, right? If it’s a process that I’ve done or a process or what we’re working through right now, and then what that journey has looked like. Right? Because if I can do it now, and it’s pretty easy, and it’s almost habit or reflex, when was it not a habit and reflex. And then, how did I get here? What were the steps that I got here? Did we have a checklist? What were the crazy hurdles? Because if it’s easy for me now and it probably wasn’t easy for me at some point, I had to learn to get there. That’s probably going to be useful or at least interesting to someone.
And then, write that up. And then, that’s the story, but then writing it up I think is another piece. And like Dominica said, I’m on a budget program committee, write it up that when I’m reading 300 submissions or a 1,000 submissions it stands out to me and it’s interesting I can tell you’re going to have some personality onstage and let the audience know what they’ll get out of it. That’s when they read it in the brochure, they know like which talks to go to. What takeaways am I going to get from this? Am I going to have a few tips and tricks? Am I going to know what to avoid? Am I going to know where to start? Those are some tips in terms of like writing submissions. And sometimes I forget when I write my own submissions, so none of us are perfect.
Gene: Awesome.
Nicole: Keep trying. I mean, I still get CFPs rejected. We all do.
Gene: No, it’s so true. Terri Potts.
Terri: Yeah, I do this a lot with people in my organization because moving up the career chain at Raytheon often requires you if you want to stay technical to start presenting. First maybe internally within Raytheon and then externally.
Nicole: I love that.
Terri: Yeah, so a lot of times people think, “Wow, what I do? That’s boring.” I try to just have a conversation with them to get them to tell me some of the things that they’ve done recently. And some of the things I really like to hear about is, “Okay, you started using Puppet or you started using Chef, did that just go perfectly the first time?” “Well, no we tried this and that didn’t work. Then we did this.” I’m like, other people would love to hear that they don’t make those same mistakes. So write something up to say these are things that we found that work really well and these are things that we found that didn’t work so well, so don’t repeat those mistakes.”
Gene: That’s great. As a professional conference goer, I mean I love one of my favorite variances like the internal technology conferences because this is a hope of the shared context that’s already shared. There’s a mercenary reason why you want to help people within your own organization that isn’t present in a more industry-wide conference. I mean I just really I got to attend the one at Capital One and at Nationwide Insurance. I just, they really stand out in my memories. Just incredibly unique experiences and so great for internal technologists to be able to show off the amazing things they’ve done.
All right, we’re almost out of time, one last question. I think a theme that’s come up over and over again is we’re lifelong learners. Learning is part of the game, in this game that we’re playing. Learning is definitely part of it whether we like it or not. Hopefully, we’d like it far more than we don’t. What are your top resources for learning? Maybe answer two ways—of all time and maybe right now here in the moment. Erica, let’s start with you.
Erica: Sure. Of all time, once upon a time I had like my C++ book that I referenced all the time before I really even used Google, so there was those sorts of things. But, much more recently, I rely heavily on Twitter, and a lot of the things that I see on Twitter and on leaders in the DevOps space, and what are they reading, what are the things that they’re tweeting about. There’s obviously books like The DevOps Handbook. And The Phoenix Project and I try read probably about one book at a time, but then I like to just see what’s going on space. Yeah, I’ll see whether there’s a conference going on and I’ll go look for that conference and what’s going on with that conference and the slides that are coming out of that, and that helps me stay current. And then just rely on co-workers as well. They’ll send out emails and different articles that they’ve read.
Gene: That’s awesome. By the way, one of the things I just want to punctuate is several of you mentioned, it always helps to learn with co-workers. I mean I remember teaching myself JavaScript in React, and it was incredibly gratifying, but it was horrible beyond words because there were like 40 tools I had to learn what they were and I didn’t care. Just that it always helps to have a buddy in learning things. I love that pattern. Thank you, Erica. Robin, top resource for learning.
Robin: So, I again I follow Twitter just like our counterparts here. I am a big fan of LinkedIn, I read LinkedIn and read the latest things there. I have a membership to Safari Books. They always have the latest technology books which I just love. And I read the blogs. It’s interesting because you get many different perspectives on the different blogs you hear and what Mike Cohen thinks is cool and Ken Schrader, and then you turn around and hear David Anderson. Maybe they don’t all play nice in the sandbox, but it’s interesting to hear what they have to say.
Gene: Twitter, it is interesting and we always used to dismiss it as a toy, but more and more I find it interesting just to some of the most interesting reading comes from what other than what my friends are reading. Interesting Robin. Nicole.
Nicole: For my overall best learning it’s probably my Ph.D. because it taught me how to do a lot of critical thinking, statistical thinking, pulling things apart, how to design, and critically pull apart research and think about things in a really concrete, consistent, detailed way. And then overall in today, Twitter, news sites. I try to make sure I keep it as well-rounded as possible. New York Times, Wall Street Journal, Al Jazeera, and I try to keep up on what my friends are reading, but then I also try to occasionally do a random search to get exposure to things that my friends aren’t reading.
Gene: Great. Thank you, Nicole. Terri.
Terri: Let’s see, far all time, I mean I’ve kept a membership for IEEE and ACM for years just trying to keep track of what’s going on, but now things are a lot more electronic. Some of the things I do regularly is I try to listen to most of the Software Engineering Daily podcast episodes.
Gene: Oh, great.
Terri: And all the SE-org podcasts, which they’ve had a tremendous amount of podcasts that I found really, really useful. And of course, I do go to a few conferences every year, so I’ve learned a lot at DevOps Enterprise Summit, and this year I presented at the SEI’s Architecture and Technology User Network Conference, and it was really, really good to see the amount of architecture work going on around figuring out how to transform legacy systems and architect for automation in this assembly.
Gene: That’s great. And by the way, I echo your endorsement of the Software Engineering podcast as they have great people in there.
Nicole: Yeah, and I’ll plus on IEEE and ACM. ACM Queue has been really fantastic.
Gene: Yes, and then last is Dominica.
Dominica: Yeah, I’ve learned much from watching TedTalks. That’s one of the greatest sources of learning for me for years now. I mean Hans Rosling, “Let my dataset change your mindset”—in 20 minutes what you can learn from that. Or David McCandless in “The beauty of data visualization”—just phenomenal. And then the DevOps Enterprise Summit videos over the last four or five years, from the tech side great, great learning resources straight up on YouTube, easy to get to.
Gene: Awesome, yeah, because there’s a YouTube channel you can just subscribe to it, and then you’ll be notified whatever there’s a new one. In fact I was going through them last night. Awesome, well, hey thank you much. I so much appreciate your time and you sharing all of your experiences. This is a topic that is near and dear to my heart, and I think we were actually going to be writing this up that people can actually access it in written form and the summary of it.
I look forward to seeing you all soon, one way or another. At the very latest at DevOps Enterprise in San Francisco. I hope you’ve learned as much as I have today.
Thank you to all the panelists who are doing amazing work in all your respective organizations, and you will be able to see them too Nov. 13-15 at DevOps Enterprise in San Francisco. It’s at the Hilton Union Square location and look for updates on the website at events.itrevolution.com. In fact, we’ll be announcing more speakers in the next week or two as we start processing the CFP submissions. And just a reminder, last reminder early bird ticket pricing ends on July 31 be sure to register soon for a 20 percent discount!
With that, thank you much Erica Morrison, Robin Yeman, Nicole Forsgren, Terri Potts, and Dominica DeGrandis. See you next time, all.
Let’s see if this scenario resonates with anyone: Your department, organization, company, etc. is beginning to embrace DevOps practices, which is, of course, an amazing step in the right direction! One of those key practices is the “You Build It, You Run It” (YBI/YRI) mantra that helps facilitate a higher level of quality and accountability. This common practice also solves the mentality of throwing your solution/code over the fence for some other team to fix.
I’m a huge proponent of YBI/YRI. However, in the case of existing brownfield solutions, there are several things you need to consider when transforming into the YBI/YRI model, including:
What if you have an existing solution where the collective “Y(ou)” in ‘You Build It’ does not include any or many of the original team that built it? This would be anyone who was part of the end-to-end creation of the solution and was empowered to make decisions impacting the design, development and delivery of a solution. In this case, it becomes, “Someone else built this mess, now you run it,” [SEBTM/NURI] which not only makes for a horrible acronym but is also a very realistic scenario that requires special consideration we’ll discuss below.
What if you are still heavily siloed and have a squad spread across multiple organizations, each with different goals and objectives? If you are still running outside of a true squad model, it can be difficult to ensure everyone’s priorities and goals are aligned to produce quality solutions with the right functions customers want.
What if you have varying definitions of “You” within your organization? Often in the YBI/YRI model, the interpretation makes some roles or key decision makers exempt from the glory of pager duty or answering to an angry customer. This can be especially true when you’re still trying to get to a point of having fully cross-functional, empowered squads.
I think it’s fair to say that most DevOps transformations don’t necessarily happen on greenfield solutions, which means, especially at the enterprise level, the reality can end up being a beast of a monolithic solution(s) built by others who may or may not still be around.
How do you fix it? My suggestion is that every person who was empowered to make a decision that contributed to the “B” (Build), becomes a “Y”(You). Of course, this doesn’t work if “Y” is no longer with the group or company, in which case I think it becomes a different issue of ensuring the new “Y” is 100 percent empowered with the ability to make decisions that enables that team to successfully run it.
One more complication that adds to the mix of a YBI/YRI strategy is putting that strategy in place without a solid foundation of agile practices. Ideally when you are adopting DevOps practices, you already have a solid agile foundation with empowered cross-functional squads; however, in practice we often see the agile foundation still being built and/or refined at the same time as incorporating DevOps practices. As a result, there still exists a lot of hand-offs while two transformations are trying to run in parallel. If you have hand-offs combined with team’s with separate business objectives, those blurry lines of accountability will become an issue sooner than later.
During transformations, if you find yourself in that in-between state of not having fully empowered accountable teams and are working with solutions your team may not have actually built, below are a few suggestions to get through those initial challenges:
Ensure you allow for enough time to address technical debt during your sprint planning. Continuing to add features and functions on a poorly designed and/or broken solution will only cause further issues and likely increase the level of frustration on the team.
Ensure that EVERY person who contributes to key decisions on a solution is included in the pager rotation in some way. Until you are in a state of fully empowered cross-functional squads, it’s important to drive the importance of accountability. As an example, if you have an architect in one organization who is working with a development team in another organization, that architect needs to be included on the pager as well to be aware of any problems resulting from a key architecture decision.
Get in the habit of operational retrospectives. Begin (or continue) the habit of operational retrospectives that bring together the end-to-end squad into a cross-functional review of the operational incidents, root cause assessment and stories/tasks that need to get added to the backlog and prioritized.
Although the transition into YBI/YRI for brownfield solutions can be muddy, empathizing with the mountains of technical debt and fully empowering solution teams can go a long way in truly helping your team’s succeed through the beginning phases of your DevOps transformation.
About the Author / Shelbee Smith-Eigenbrode
Shelbee Smith-Eigenbrode is a senior software engineer/IT architect at IBM and has been in IT for 19 years. She has worked across varying roles spanning the software development life cycle. Her experience across functional roles as well as performing that work deeply inside traditional silos has contributed to her passion for embracing the DevOps culture and transforming the way teams are delivering technologies by adopting DevOps practices. She is a member of the IBM Academy of Technology committed to driving innovation and technical advancement.
I am often asked, “if we are using a continuous integration (CI) server and adopting a continuous delivery pipeline, where does continuous deployment fit in?” Good question. In a nutshell, continuous delivery engines such as CloudBees or Blue Ocean are designed to trigger the execution of CI workflows across the dev, test and prod environments. Continuous deployment is called as part of the CI workflow to do the actual release of code out to endpoints. Continuous deployment can be managed by application release automation tools or supported by one-off scripts. Regardless of how sophisticated the deployment step of your CI workflow is, it is still continuous deployment.
To understand both continuous delivery and continuous deployment, one must go back to their roots, continuous integration. CI is a key component of Agile practice. Agile developers understand the importance of adopting a methodology that can manage smaller incremental software changes to release software, and therefore business value, to end users quicker and cheaper. The activities of deploying to testing and production historically have been in the hands of testing and data center teams, separate from Development. But in the Agile practice, these activities cannot be separated from development. The concept of separate development, testing and production teams is a waterfall concept, not an Agile concept. In Agile, everyone is on the same team. This means that testing and production teams must become part of the Agile process.
Mastering Agile’s last Mile
Ultimately, the goal of continuous delivery is to accelerate release cycles and get new code out to production fast. However, for most companies this is not happening. Developers and testers may have embraced continuous delivery, but when it comes to production, the code is moved to a staging area where it languishes until production is ready to perform the update. The last mile of Agile is not achieved. And we are back to where we started: waterfall.
There are many reasons for the production roadblock. Lets’ review the top four.
Agents: Both open-source and commercial CI servers are agent-based. An agent based system works fine for developers and can be tolerated by testing. However, production teams are not that excited about installing yet another agent on to the production end points. For this reason, production team use their own method of deploying code. Therefore, the deployment scripts used for releasing code to dev and test are not used by production. The way they see it: Production does not want a deployment process that requires them to re-configure their data center environments with agent from a development tool.
Scripts: Production teams want input and visibility. One-off scripts obfuscate the deployment process. For this reason, production teams will write their own scripts. Scripts simply do not have the intelligence needed to meet the audit, rollback and transparency requirements needed by production teams. Production teams perform many steps manually. This takes time and cannot be done in an agile cadence. The result: Great code languishes in staging areas. Production stays several releases behind what continuous test has approved.
Cost, cost, cost: Most of what developers create in a continuous delivery workflow comes from open-source or consumption-based licensing models (think Jenkins and Jira). Tools that can achieve continuous deployment such as application release automation solutions are far to expensive for one development team to acquire and incorporate into the CI workflow. These types of tools are sold at the CIO level and cost upwards of $250,000. Production teams are often hesitant to purchases these tools, as the high cost equals high risk. And ‘one ring to rule them all’ does not work. This type of operational tool acquisition is waterfall-based and does not fit into the leaner Agile DevOps tooling. The result: One-off scripts continue to be used and dev and test. Production does not adopt agile.
Sustainability: During waterfall, a production release was performed as infrequent as two times per year, or at most once per quarter (considered aggressive). In other words, production is staffed to support a production release on an infrequent basis. With agile, the demand for production releases increases 10-fold. Instead of doing at most four releases a year, they may be required to do four releases a month. Production cannot keep up the agile pace. They are staffed for waterfall.
Automating Continuous Deployment: Key to Maturing Continuous Delivery
Continuous delivery cannot be successful without continuous deployment. If you want to roll code out to end users faster, the deployment process itself must be modernized. Using the same one-off scripts that were used in the waterfall approach will not achieve your CD goals.
Continuous deployment automation is performed by a new set of agile DevOps tools called application release automation, or ARA. ARA solutions are designed to support a model-driven software deployment that allows development, test and production teams to use the same process without the need for one-off deploy scripts. ARA adds all of the features and functions needed to support continuous delivery and data center deployment requirements. ARA is designed to fully perform the heavy lifting of software releases including application packaging, release versioning, database updates, server configuration management, calendaring, roll-forward, rollback, security access and auditing.
ARA solutions are called by a templated CI workflow that is defined by the development team. The workflow is moved up the pipeline and adapts to the changes at the various pipeline stages (System Test, Integration Test, Pre-Production, Production, etc.) Because the process is driven by the ARA tool and not a script, the changes are automatically addressed across the pipeline reducing the workload for all teams and supporting frequent changes that can be easily traced and audited down to the server inventory level, including the production environment.
Developers, Get Your Hands on Deployment Automation Tool
To achieve a repeatable CD pipeline that supports production, developers must embrace the automation of continuous deployment at development. The dysfunction however is the outrageously high cost of ARA tools. Developers don’t buy expensive operational solutions. More ARA solutions must be offered on a consumption-based model or offered as open source with the ‘out of the box’ integrations into CI/CD. When considering the purchase of an ARA solution, look for agentless models that can be implemented one project team at a time. Embracing this level of corporate process change must be done project by project. The operational big bang approach does not work well in an agile practice. Not everyone can use the same solution. However, if done correctly, there are big benefits.
ARA is a process that strengthens the foundation of the continuous delivery pipeline and is core to the agile DevOps initiative. It facilitates agile practices by allowing the testing teams and data center teams the tools needed to become more agile and accept the frequency of changes being pushed across the pipeline by Development. Ultimately, the goal is repeatability across the pipeline, pulling together Dev and Ops.
About the Author / Tracy Ragan
Tracy Ragan is CEO and Co-Founder of DeployHub. Tracy has extensive experience in the development and implementation of business applications. It was during her consulting experiences that she recognized the lack of build and release management procedures for the distributed platform that had long been considered standard on the mainframe and UNIX. Tracy has worked with hundreds of development teams in implementing a team-centric standardized build to release process. Connect with her on LinkedIn.
It’s time to start implementing the practices of Continuous Testing
Earlier this year, Computing Magazine conducted a fascinating study around the adoption and benefits of DevOps in its DevOps Review 2017 publication. The findings that stood out most fascinating to me where around quality assurance and testing.
In this survey, the question was asked, “Where are the main holdups in the software production process?” Of all the self-proclaimed DevOps practitioners, a whopping 63 percent said the biggest delays occurring in the QA/Test phase. In fact, this cited bottleneck was more than 2 times greater than the next proclaimed bottleneck, which they felt was in planning (29 percent). In this same survey, another question asked, “What tends to lose out more in DevOps environments?” While 43 percent said, “Nothing,” nearly 30 percent of these DevOps practitioners said that quality was being sacrificed. When asked which area they wanted to focus on improving in the next year, over 40 percent opted for quality.
Why is it that quality has taken such a hit? The study cited two issues: 1) Testing and QA is often siloed from the other parts of the organization, and 2) Testing tools haven’t matured much in the past five years.
Testing must become a cross-functional operation across the software development lifecycle (SDLC) spectrum, one that involves all teams. Agile developers and testing teams don’t have weeks to test, but hours. Testing hasn’t kept up because it hasn’t applied the methods and concepts of agile development have not been applied to testing and QA processes. By removing the constraints that force testing to the end of a development cycle, developers and QA teams can achieve accurate, automated testing from day one. They can create an integrated pipeline, capable of taking an idea from design to deployment, at speed, without compromising quality.
Open-source, cloud-based solutions can be the catalyst that brings the speed of agile to testing. Here are a few examples.
Better In-Sprint API Testing: Since APIs have become the major component of applications, organizations are shifting their approach from starting testing when the user interface is complete to shifting the focus of testing at the API level. But the hard thing is finding enough time in a development sprint to test them sufficiently.
Today’s applications are a collection of microservices and API calls. UI testing alone just won’t cut it. UI testing is very brittle, and whenever it changes, testing breaks. Testing needs to shift left into the development cycle—into sprints. But if you shift testing left to be a focus of the sprints, what kind of API testing can you really do? How comprehensive can they really be? Creating those API tests has to be quick, and simple.
Dev and Test teams must utilize automated tools that not only generate API tests, but create an exhaustive set of test cases from the API, then execute those tests. These tools also need to support the most popular open-source testing frameworks allowing developers to run functional API tests as performance tests without any proprietary conversion issues. And, vice versa, developers must be able to take open-source performance tests and be able to run them as functional API tests.
Orchestrating Testing: A Continuous Delivery Pipeline Manager: “Faster” is becoming table stakes with automation. The facts show that companies who have widespread agile adoption and are deploying more frequently with significantly reduced lead times. But, this too often leads to release failures, quality issues and disorder with multiple—if not hundreds—of applications rolling through the SDLC. Plus, things slow down significantly with time-consuming handoffs and rework in the release process. DevOps teams, who release features and fixes frequently while significantly reducing errors and manual processes, need toolchain-agnostic pipeline planning, orchestration and optimization solutions that enable them to eliminate spreadsheets, automate test processes and feedback, and continually improve the delivery of revenue-generating features to end-customers. New CD SaaS tools help make this orchestration process easy.
Performance and Functional Testing Using Open Source Frameworks: More and more organizations are trying to move from legacy testing tools. It’s not so much that these don’t adequately test, but they were never designed to meet the needs of agile development approaches. Rather, they were designed for releases that occurred two to three times per year. But that won’t cut it anymore.
These are just a few examples of how continuous testing eliminates the testing hurdle. With continuous testing, organizations can:
Automatically convert user stories from requirements management tools like CA Agile Central or Atlassian JIRA into the minimum number of test cases required for complete code coverage, and being able to adapt automatically to changing business requirements.
Reduce test cycle prep time by requesting test data from within requirements.
Request, reserve, and create test data at any time, even from requirements phase.
Virtualize APIs for better testing, while managing API access to developers.
Institute easy API testing that can generate comprehensive test cases (positive, negative, edge case, and SQL injection) and execute those tests automatically.
Democratize testing with SaaS-based testing tools so that developers can test early and test often.
Speed testing by feeding test data directly and securely to virtual services.
Use open-source testing tools such as JMeter, Selenium, Jenkins, Jenkins, Appium, Taurus, Gatling, Locust and more.
Orchestrate the application delivery pipeline to promote releases from one environment to the next as soon as all the tests pass and enable faster feedback loops.
This is testing unleashed, going hand in hand with agile development to accelerate quality into production.
By adopting next-generation testing practices, DevOps teams have the ability to test early, often, automatically and continuously. To learn more about continuous testing and why it is critical to DevOps and Continuous Delivery initiatives, visit this Continuous Testing informational page.
One of the perks of being editor in chief of devopsy.kinsta.cloud is I get a chance to meet with so many leaders of the DevOps and IT industry. You can really learn a lot by listening to what these folks say. This DevOps Chat is a perfect example. I sat down with Flint Brenton, CEO of Collabnet. Collabnet is not some johnny-come-lately. The company has been helping developers and IT professionals for a long time. Flint talks about the fact that while DevOps is a great path for many organizations, it may not be the right way for every situation. Waterfall is not going away entirely.
Have a listen to my discussion with Flint and hear for yourself what he has to say on the subject. As usual, the streaming audio player is immediately below with the transcript below that. Enjoy!
Streaming Audio
Transcript of Audio
Alan Shimel: Hey, everyone. This is Alan Shimel, devopsy.kinsta.cloud, here for another DevOps Chat. Happy to be joined in this DevOps Chat by Flint Brenton, CEO of CollabNet. Flint, welcome.
Flint Brenton: Hey, Alan. It’s great to see you again.
Shimel: Great to see and hear you again. Flint, the last time I think you and I talked was back at the DevOps Enterprise Summit in San Francisco. I guess that had to be October or November.
Brenton: November. Actually, it was the same week that our new president was elected, if you remember.
Shimel: Yes. Yeah, a red letter day.
Brenton: Right.
Shimel: So, Flint, without getting any further into that, let’s turn the conversation to CollabNet. What—well, first of all, maybe some of our audience is not familiar with CollabNet in general, so why don’t we start with that? Can you give us a little CollabNet background?
Brenton: Yes, absolutely. So CollabNet is a SAS-based, cloud-enabled development platform focused exclusively on enterprise that delivers application lifecycle management capability for both Agile and waterfall development processes.
Shimel: Mm-hmm.
Brenton: The version technology to structure those builds and to maintain high quality and high quality control throughout the process. And then we also provide the DevOps infrastructure to build and deploy automatically whatever gets created in Azure or waterfall, with high quality, on time, and, to the extent we can help, under budget. So that’s really our focus.
Shimel: Sure. And you know what, Flint? I remember this about CollabNet. You guys have been in business now for a number of years, certainly before DevOps was what it is today, let’s call it, whatever it is today. How long have you been—
Brenton: Yeah, we’ve been in business—this is our 18th year.
Shimel: Absolutely. So, you know, DevOps is another technology innovation, if you will, that’s come down the road, but I think what—not unique, but what is one of the special qualities of CollabNet is is that you’re not out here saying that the whole world needs to go DevOps today, right, that everything—we should be all DevOps all the time. But you have customers who are still on waterfall and may very well remain on waterfall for the foreseeable future.
Brenton: Right. And we really invest very heavily in both waterfall and Agile development platforms and—although the growth rate for Agile’s higher than waterfall in enterprise, we still see roughly 70 [percent] to 75 percent of enterprises using a waterfall methodology. But the fastest growing segment in that area is clearly Agile.
Shimel: Sure. And I—obviously. But I think what’s important for our audience to recognize is even the biggest DevOps and Agile advocates or biggest—whatever you wanna call ’em, should recognize that there are still projects and places where waterfall might be the best way to skin the cat, to complete a project correctly. And it doesn’t have to be an all-or-nothing type of environment, if you will.
Brenton: That’s exactly right. You know, the industries that tend to be highly regulated, with a high need for security, compliance, and scalability, tend to lean more towards waterfall development. And, of course, once you started a process where you’ve got an established team that’s been using waterfall for a long period of time, they tend to prefer to continue with that. But it’s new projects, new projects where maybe time to market outweighs the need for compliance and security and scale—those tend to go Agile in an enterprise. But, either way, regardless of development methodology, the things you need beyond the tool chain or the process of building software is fairly similar, right? The way you use version control is fairly similar. The way you use DevOps technologies is also very similar.
And what we found is customers who are well grounded in Agile and waterfall—typically, in enterprise, they have both—getting your act together in DevOps and making sure you have a strategy, you created a culture to embrace and execute that strategy, then having the processes and tools in place to bring it to reality is critical because you don’t really get the return on investment for anything you develop, regardless of the methodology, until you put the build into production. And that’s what DevOps, at the end of the day, is all about.
Shimel: Sure is. Excellent. Excellent point, Flint. Flint, let me turn things a little bit ’cause, as I mentioned to you off camera, off mic, 15 minutes here goes pretty quick. Want to talk about some recent news around CollabNet in the industry. I understand you guys were just included in a Gartner Magic Quadrant, correct?
Brenton: Yes. Yes, thank you for bringing that up. Yeah, we’re very excited about it. It’s always a reflection of the investment our team has made in Agile planning and execution. So we’ve always been considered a leader when it comes to waterfall development, but, as we just mentioned, for enterprise, our enterprise customers are really demanding that we provide both, both equal value in waterfall and Agile. So we invested heavily in the last few years in Agile, and, as a result, we were just included in their Magic Quadrant as a visionary for Agile planning and execution, and we’re very excited about it.
Shimel: That’s nice. That is great. And I know everyone always wants to be in the upper right, but, really, having my background, being entrepreneurial and having started businesses, I’d almost rather be in that visionary quadrant than the upper-right leader quadrant and certainly in the visionary quadrant over, let’s say, the challenger quadrant, which is the upper left, because I think, with visionary, I mean, the recognition is that you’re setting a—you’re pushing the envelope, right? You’re pushing the vision—hence, “visionary.”
Brenton: Right.
Shimel: So congratulations to you guys. I know, in speaking with your team, you have rights for download of the quadrant, so what we’ll try to do is put into the show notes here a link to that where people can download the quadrant report and see if—check it out for themselves.
Brenton: Yeah, that’d be perfect and I’m sure that your listeners would very much appreciate that.
Shimel: Sure will. So, Flint, beyond the quadrant inclusion, though, what else do you see that’s happening with CollabNet or in the market in general?
Brenton: We’re starting now to get a number of customer wins that are embracing our motto of providing a cloud-based service where you can consume our integrated ALM, version control, and DevOps solution either in Amazon or on our own private cloud and, in many cases, a combination of on-prem and off-prem.
Shimel: Right.
Brenton: So we now have a number of customers who have completely moved their development process into one of our cloud-enabled environments, and it’s very exciting.
Shimel: That is exciting.
Brenton: It’s very exciting. And the great (thing) about that is, if it’s in the cloud, it’s actually easier for us to include, Alan, both open-source and compare it to components that we don’t own or control because then the need to test to get everything in your data center is lessened and our ability to innovate to your advantage is much greater, so we’re very excited about that.
Shimel: Absolutely. So, Flint, interestingly enough, I was reading an article this morning—I forgot where I saw it but it was a recent survey where 65 percent of enterprises still keeping the majority of their infrastructure on their own data center, not on a public cloud, not a hybrid situation. Is that about what you guys are seeing but is the arrow pointing up to the cloud? What’s your take on that?
Brenton: Now that’s a great question, Alan, and it’s absolutely what we’re seeing. So a thing to keep in mind, though, is “What is the variable that’s changing and what is the velocity of that change?” Right? So I would say, two years ago, that 65 percent number was probably 90-plus, right? So, as enterprises get more and more comfortable with the security and compliance options in the cloud, whether it’s in a public cloud or private cloud, they can see the tremendous advantages of driving down the cost of development and actually increasing the speed of innovation by going to the cloud.
I’ll give you an example. U.S. government, we do a tremendous amount in federal, and so both the Obama administration and now it’s transitioned to the Trump administration has really been pushing the department of defense to do a couple things. One, innovate faster, and so, if you just think about the U.S. government is barraged by cyberattacks from all over the world. Right?
Shimel: Mm-hmm.
Brenton: And so your ability as a professional, in NSA or CIA or the FBI, to deal with that is you want a complete toolkit of things you can use, and you want to choose the tools you need to deal with those cyberattacks, but you also need to do it quickly. So if you can use open source, it helps. If you can deploy it in a cloud, it dramatically increase—or decreases the time to deployment. And, three, it also tremendously helps you on the economics side. So even something as highly regulated and structured as the US federal government, there are advantages sometimes in doing things in a public cloud to help, and so we’re starting to see variations of that theme also in industry, primarily around enterprise, where there’s flexibility, increased ability to innovate, and improved economics by moving workflow to the cloud.
Shimel: Yep. You know, Flint, living in the tech world and you lived the world—you live that life; I live that life—I think sometimes we get jaded in that we’re in this bubble of, “What’s the hot new thing?” and it’s DevOps and it’s whatever’s next after DevOps, and then sometimes you get a reminder—65 percent still not in the cloud. And you say, “But I thought everyone’s in the cloud.” And I think it does good for us to remember that there’s a big world out there that moves at their own pace and their own—
Brenton: Right.
Shimel: Same thing with waterfall. Right? I think people would be—people in our audience from devopsy.kinsta.cloud would probably be surprised to find out how many organizations still either haven’t adopted DevOps or are primarily waterfall shops.
Brenton: You know, the key, when you’re a solution provider like CollabNet, is you need to meet people where they are. Right? So, if they wanna be on-prem, you need to be able to deliver them superior value on-prem. If they want to move part of their workload to the cloud, you need to be able to help lead them and sponsor that transition at that pace by which they’re comfortable. Right? So I learned a long time ago the key to success in this business is to give people what they want, right? He who can do that best with a quality solution at a great price is gonna win. Right? So that’s just the way it works, right?
Shimel: Absolutely—and that hasn’t changed, by the way, through all of these innovations, and I think it was true 100 years ago and it’s true today. Or maybe even more than 100.
Brenton: You bet.
Shimel: Anyway, Flint, listen, I think we’re about out of time here. I wanna be respectful of your time, so what we will do is let’s have you on again soon when we have some more time and we can delve into this. Are you gonna be at DevOps Enterprise Summit maybe in London?
Brenton: You bet. Yeah.
Shimel: Or San Francisco?
Brenton: Well, I always enjoy speaking with you, Alan. You’re a great guy and you’re—
Shimel: Thanks.
Brenton: —you’re a fun guy off camera too, I will add.
Shimel:[Laughs]
Brenton: So I look forward to our next adventure together. Thank you for the—
Shimel: Absolutely. Hey, Flint—
Brenton: All right.
Shimel: Flint Brenton, CEO of CollabNet, this week’s guest on DevOps Chat, thanks for joining us. This is Alan—
Steve Mezak is founder and CEO of Accelerance, a leader in global software development outsourcing, as well as the author of “Software Without Borders,” co-author of, “Outsource or Else: How a VP of Software Saved His Company,” and a CIO columnist.
Carmen DeArdo is the director of Build Technologies at Nationwide Insurance. He is responsible for helping drive accelerated delivery utilizing Lean and DevOps techniques across the end-to-end delivery value stream. One current focus area is the creation of an integrated delivery pipeline capability.
Oren Rubin is CEO of 
Shawn Minto is the senior director of Technical Services and Support at 
Romy Hughes is director at 
To understand both continuous delivery and continuous deployment, one must go back to their roots, continuous integration. CI is a key component of Agile practice. Agile developers understand the importance of adopting a methodology that can manage smaller incremental software changes to release software, and therefore business value, to end users quicker and cheaper. The activities of deploying to testing and production historically have been in the hands of testing and data center teams, separate from Development. But in the Agile practice, these activities cannot be separated from development. The concept of separate development, testing and production teams is a waterfall concept, not an Agile concept. In Agile, everyone is on the same team. This means that testing and production teams must become part of the Agile process.
Continuous deployment automation is performed by a new set of agile DevOps tools called application release automation, or ARA. ARA solutions are designed to support a model-driven software deployment that allows development, test and production teams to use the same process without the need for one-off deploy scripts. ARA adds all of the features and functions needed to support continuous delivery and data center deployment requirements. ARA is designed to fully perform the heavy lifting of software releases including application packaging, release versioning, database updates, server configuration management, calendaring, roll-forward, rollback, security access and auditing.
Tracy Ragan is CEO and Co-Founder of 
This is testing unleashed, going hand in hand with agile development to accelerate quality into production.