devopsy.kinsta.cloud

Tag: developers

CD 101 Podcast: How to Build the Modern Software Factory with Agile Development and Continuous Delivery
And now, for Episode 3 … you decide if it beats “Star Wars – Revenge of the Sith.”

I think we accidentally just set the bar pretty low, but …

… here we go. Let’s begin with a prerequisite pithy quote from an objective industry observer:

“If agile software development was the opening act to a great performance, continuous delivery is the headliner,” according to Kurt Bittner of Forrester Research.

Kurt explains this concept further by saying, “The pace at which consumers expect change is causing a crisis in application development and delivery … agile was a good start, but it wasn’t sufficient to drive better business results; continuous delivery forges the broken link in the value chain, connecting business strategy with business results.”

So there, you have it. At least according to Forrester, continuous delivery (CD) is one of the most important evolutions of agile methodology within the modern software factory. But, there’s more.

Andrew Binstock from Dr. Dobbs fairly well echoes this sentiment, saying: “It’s becoming clear that agile alone is no longer sufficient to keep up with the demands of the present day … software complexity and the need for constant updates are making it difficult for leading edge organizations to keep up without moving beyond agile and embracing continuous delivery.”

Episode 3 of our podcast series discusses the importance of Agile Transformation along WITH Continuous Delivery to automated some of the development, test, and release efforts to remove some of the bottlenecks downstream in the software development life cycle in the build, test and release phases.

Lean Agile + CD can create these 1+1=3 types of results.

Episode 3 of the Continuous Delivery 101 Podcast can be found by:
- Listening to it on your computer on YouTube
- Downloading it from iTunes store
- Downloading the podcast from your Podcast App by searching for “Continuous Delivery 101 podcast”
Thanks for listening!

— Scott Edwards
March 13, 2017
Release Automation: Delivering the Big Quality Payoff

Have you ever gone down the path of testing the latest release of an application only to find out that you had the wrong version?

Maybe you accidentally grabbed the wrong build or you didn’t have the correct configuration file for the environment you were testing against—or your database wasn’t updated correctly, or the script you were using was still pointing to a prior version of the software.

Whatever the case, we can agree that keeping track of all the various artifacts, database schema changes, configuration parameters and other components needed to accurately test and deploy an application is both time-consuming and prone to errors.

Today, many companies are looking to DevOps, continuous delivery and agile development methodologies to shorten their application delivery cycles and reduce the number of errors and the costs associated with application deployment processes. Nearly all enterprise companies are software companies, and their mere existence depends on delivering differentiating products and services via software to their customers.

It’s imperative to be able to react to customer demands and competitive pressures quickly, if not immediately, to stay ahead of the competition. Gone are the days of yearly release cycles.

Today we see companies deploying new capabilities into production environments on a monthly, weekly or even daily basis. At some companies, hundreds, if not thousands, of deployments are occurring across dev, test and production environments daily. The old methodologies we once used simply can’t support this type of volume and cadence.

Employing Release Automation

So, how are these companies transitioning to this new way of thinking? In a word—automation; specifically, by automating early and often.

Everything about your release process is a candidate for automation. This includes environment provisioning and configuration automation, build automation, test automation and release and deployment automation.

A customer of ours recently implemented CA Release Automation, and within eight weeks had automated 90 percent of its deployments. Deployments that once took more than five hours and a team of people now take a matter of minutes.

“This not only saves us a huge amount of time, but it also mitigates the risk of human error being introduced by manual processes,” said Tim Dallmann, platform engineer at Jewelers Mutual.

Another customer, a major airline, implemented CA Release Automation and immediately reduced their production outages from an average of 237 over a three-month period, down to 15, seven, six and five over the following four months.

And Molina Medicaid Solutions reduced their cost of human error by 90 percent, all the while responding quicker to emergency deployment requests and improving deployment quality.

Automation is critical, but I believe these results prove that implementing a framework to manage and deploy release processes in addition to automation works even better.

Increased Flexibility and Visibility

Continuous delivery helps break down barriers often seen between development and operations teams. It provides, even forces, a more collaborative approach and automation is a key enabler of this.

“One of the best things about automation is that it gives us the flexibility to skill-up end users—deployments in our test environments are no longer solely the responsibility of the operations or IT teams,” explained Dallmann.

With an application release automation (ARA) solution, artifact, build, configuration and release processes are no longer tracked with spreadsheets or embedded in complicated scripts.

Increased visibility into these processes ultimately enables all stakeholders to take a more active role in the release process, and access to real-time information can significantly reduce the amount of time and resources needed to resolve and recover from problems when they do occur.

Automation leads to:

Integrating with the Ecosystem

An enterprise ARA solution can help you create a single control point to streamline all your application delivery processes for better predictability and reduced risk. A critical component of this is the ability to integrate and orchestrate tools and technologies required along the continuous delivery pipeline, including source control, automated testing tools, provisioning and configuration management solutions and change management and performance management solutions.

Integration with an application lifecycle management solution can provide even further alignment, governance and traceability between development projects and what is ultimately being tested and promoted to production.

“By integrating the two solutions, we will have a complete audit trail of deployments to help us understand how, where and by whom stories have changed,” said Phil Nickolai, director of application services at Jewelers Mutual.

“What we really needed was a solution to streamline our whole way of working,” he said. “We had numerous people working on projects, competing for resources and causing complexity. Now we’re back in control and getting smarter every day.”

The Time Is Now

Let’s get rid of the problem that exists between the keyboard and information kept in spreadsheets, scripts or emails. Not only will you stop wasting time testing the wrong version of an application, but you can also ensure you have the right test data, the appropriate test cases and the correct configuration parameters for your environment.

Stress-inducing release weekends and nights become a thing of the past. Developers can spend more time on developing innovative capabilities that generate revenue and keep customers coming back for more.

— Tim Mueting

March 7, 2017
How Developers Can Spend More Time Coding, Less Time Fixing

I don’t think anyone became a developer because they loved to spend their time debugging. As frustrating as it may be, making sure your build works is critical—and this burden is now falling on developers. If your organization wants to deliver highly rated apps faster, building quality into the development cycle is imperative. Even though agile development has become a popular approach, it’s execution has a ways to go.

A recent trends report in mobile and web development shows 25 percent of time is spent addressing a bug fix backlog. This can slow time-to-market and limit the amount of new features your company can roll out—not ideal when your top priority is to draw in new users and boost engagement to stay competitive. The good news is, the time spent debugging throughout the development cycle can be minimized. How? By shifting QA into the development process, you can spend more time coding and less time fixing.

Mastering Agility in Practice

The question for devs today isn’t whether they test, but when and how to do it best. We’ve all heard the vague, top-line answers before: “Adopt Agile development,” or, “Shift left.” The problem is, these terms and strategies can be challenging to execute in practice, especially without proper tools to make it happen.

When we talk about “shifting left,” there’s still an important balance to be found as to which stage is right for dev teams to test—it can’t be too early, when there is nothing meaningful to test, and it can’t be too late, when the test batch pile is too big. Once your coding and testing processes are harmonized, you can effectively maximize your time and resources and get the job done faster and better.

For development teams, if testing is done in bulk at the end, identifying the specific point in the build that needs to be corrected is like finding a needle in a haystack. When we shift quality left, defects are found earlier among a much smaller set of changes, and are therefore easier to fix. That’s why continuous integration (CI) calls for building, integrating and testing the system you’re developing upon delivery of even the smallest change.

By testing continuously, uncertainty is reduced and feedback loops are made smaller, ensuring you’re building in the correct direction. The “shift left” trend is ultimately going to condense the dev cycle by getting quality back into the ingredient list. The earlier we deal with quality issues, the less focus is taken from our primary goal: building valuable, innovative technology.

Find Tools that Mesh with Your Workflow

Whether you use Selenium, Appium or Android Studio, new products need to integrate seamlessly into existing developer environments. Having to switch between building tools and testing tools is a big waste of time, and it’s unrealistic to expect a high adoption rate if the product is disrupting, rather than enhancing, your workflow. If you’re expected to shift testing into your development cycle, look for testing tools that seamlessly support the continuous integration of testing.

Once your testing and building processes is aligned, the testing feedback and analysis becomes a new bottleneck. To minimize inefficiencies and feedback gaps, you need to carefully select tools that will improve visibility into build health, and help avoid batching up issues.

Empower Developers to Satisfy Users

At the end of the day, smaller batch sizes are only half the battle. No matter the batch size, quality only can be built in if it includes full test coverage, and if all the defects can be fixed in a way that actually meets user standards of quality.

With tons of potential end user environments, injecting reality into the development process is crucial if you want to meet the high expectations of today’s consumers. With different phones, operating systems, levels of connectivity and apps running in the background, you never know what your apps will be put through once released. Leveraging stable test environments capable of recreating all potential user conditions throughout the developments cycle will maximize efficiency in development and accelerate the speed of innovation at your organization.

Through a better delivery pipeline, mobile app and web developers are simply more effective. Once your test process is automated, aligned and maximized, writing better code can become your main focus. By taking a strategic, practical approach to shifting quality left—and giving developers the opportunity to focus on what matters most—you’ll see happier developers, and in turn, more successful businesses.

About the Author / Paul Bruce

Paul Bruce is Developer Evangelist at Perfecto, focusing on the value of quality and velocity throughout the software life cycle. He previously worked as an advocate for API development and testing practices and a full stack developer before that. He now writes, speaks, listens and teaches about software delivery patterns in modern enterprises and key industries around the world. Connect with him on LinkedIn and Twitter.

March 3, 2017
What Happens Post-Deployment: The Rotating Roster
DevOps has become that celebrity who’s always the talk of the town but is largely misunderstood. It’s even quoted in the wrong context.

A few days back, while speaking at a conference, I was interacting with audience and one of the attendees said DevOps is like the backbone of Agile. I think that’s an excellent way of describing DevOps. In an industry where we are still struggling to implement Agile—and even more so to keep people from drifting—is DevOps ahead of its time? Maybe not.

DevOps is the set of practices necessary to sustain the fast pace of delivery. The best part of DevOps is there are no levels or certifications one needs to achieve. It’s the stack of practices that you pick and implement and reap benefits. The 7Cs—continuous planning, continuous integration, continuous delivery, continuous testing, continuous deployment, continuous monitoring and continuous operations—help maintain sanity, though.

One teams had to sustain fast-paced releases with a big customer base. We saw our users increase from .1 million to 2.4 million in three years. Agile, continuous integration and continuous delivery are not just fads for us, but needs for survival.

Using the Rotating Roster Method

When we talk about continuous delivery but then focus on just delivery, it’s not enough. It’s equally important to work on what happens post-deployment, such as continuous monitoring and continuous operations.

Rotating Roster is one practice we use in our organization and this has helped us pull together the whole ecosystem of:
- Resolving customer issues,
- Feedback system pumping into out technical pipeline/debt for next releases,
- Knowledge transfer and learning ecosystem to address attrition, and
- Broken silos.
Here’s how we do this. First, we define the scope of the Roster. We picked the following:
- Production Outage
- Customer issues/P1
- Ad hoc requests from stakeholders
- Data extraction (only tasks out of scope of a separate Data Analytics team)
Next, we publish a schedule of every two to four weeks, designating at least one person to be on the Roster. It’s essential that the Roster rotates to all or most in the team to build a bus factor or a knowledge transfer ecosystem in the team.

For each request received, we asked the Roster person to log it in JIRA. This was not just for bookkeeping sake, but as an essential step in line with the Roster objective.

After every Roster period, we sat down and reviewed these JIRA requests for possible targets to:
- Automate the task or the effort involved, such as account creation requests automated by writing php scripts, for example.
- Delegate the task to the right team to cut the effort. For example, for a new data extraction we set up SQL slaves and replicate tables on analytics machines, then delegate the task to them for all future instances.
- Mitigate the reason for the request, such as the root cause of an outage.
The mitigation/automation steps had to be picked by development team in their Sprints. Otherwise, usually they become technical debt and lay around in the backlog.

Since the development team was taking turns on the Roster, they understood the priority and necessity of doing the task. Thus, the technical pipeline was chosen and fed actively.

Another advantage of the Roster was it showed the team the “big picture” and helped catch early trends.

The Rotating Roster system turned out to be a big success for our teams; thus, we replicated the model through the rest of the organization. It’s a small but essential step to adopting DevOps culture.

In one definition, Jez Humble explained that DevOps is “a cross-disciplinary community of practice dedicated to the study of building, evolving and operating rapidly changing resilient systems at scale.”

I could not have put it in better words.

About the Author / Kiran Chhabra

Kiran Chhabra is Vice President of Engineering at Infoedge India Ltd. She has 14 years of industry experience designing and implementing highly scalable and available architectures with optimized continuous integration and delivery in technologies including LAMP, Java and mobile technologies.
Kiran holds a U.S. Patent and has published technical papers. She is certified scrum master and certified scrum professional, and speaks at various Scrum Alliance & technical conferences in India and the United States. Connect with her on LinkedIn and Twitter.
February 20, 2017
Reality Check: Bringing Continuous Delivery Dreams to Life

Every developer shares the same dream: to write a piece of code or app that changes everything.

Whether it makes your company do things more efficiently, attracts hordes of new customers, simply makes existing clients happier or maybe even changes the world a little bit, that code is what we strive for. Some of us don’t merely harbor that dream; many of us actually achieve that moment of brilliance. For those who have, it’s remembered as a moment of pure joy.

So, where does this magical process begin? First, you talk to your manager and, if you’re lucky, she or he agrees that it is a fantastic idea and starts to sell it internally. Sometimes the idea is so good that you get the go-ahead on the same day and even get a small team to work on it!

Then, soon thereafter, reality kicks in.

You need data, inevitably from applications running somewhere in the cloud and/or some other data from on-premises data stores. You need to interface with APIs, some well-documented, some not. You need some user experience work done, and the standards your company has adopted don’t really match your vision.

You also need to interface with many of the standard services your company has already implemented because, after all, your app doesn’t run in isolation—you work with existing billing systems, a CRM system, a fulfillment system … and all of them run on servers that can be anywhere. Soon you find yourself fighting for test data, systems to test with and applications and APIs to interface with. Before you know it, you’re banging your head against the wall.

Fortunately, your teammates are flexible, creative people. And because your internal systems seem to work against you, you go around them. You download a trial version of some testing software; you install a copy of some internal systems on a virtualized test server (if you can find one) and start working with these.

Since your app needs to be snazzy and snappy, you download some open-source software to help you test the performance on mobile devices. After three sprints, you’re now ready to move your app to an environment where you can start performing real tests!

Most times this is the moment when the hardest reality kicks in. The copies of the systems you tested with were not up to date. Data has changed. APIs have changed. The real system responds three times slower than the copy that sat on the server right next to you. The data in the cloud has changed. The performance testing tool you used is not available in the official testing environment. … The list goes on.

This doesn’t even include the difficulties of moving all the components of your new app to different environments. Trust me, this is not a fictional nightmare scenario; it is reality and I see it happen all the time.

Now, what if you had the test-data generators necessary to create the data you need and a server that ran virtualized services, mimicking all the (up-to-date) external systems you need including the APIs to talk to them—services that would allow you to say, “What if this application responds three times slower when I request data?” or, “What happens if this system suddenly goes down?” What if, when you’re done building, you could simply migrate the app and all the components that go with it to another (test/QA) system with one click?

Everybody wants to support people with good ideas and turn these ideas into working software without the frustration that very often distracts and demotivates people. To do so, you need to create the infrastructure that supports this. Is it difficult? Not really, the tools to automate services, create test data (and test scenarios) and migrate complete applications from A to Z are available.

However, just like a great app, this too starts with a dream—the dream of allowing people with great ideas to work on these ideas without the overhead of an infrastructure that allows them to focus on the coding and not on the infrastructural issues; the dream to create a software factory where the creation of the app takes 90 percent of the time and the infrastructural burden only 10 percent, not the other way around.

If you’re serious about developing software and want to create an environment where the infrastructure allows people to develop and release software faster instead of working against them, find the tools that support it and jump in, because they exist today.

Enacting this change now and maybe the next time someone comes to you with a great idea you can be confident that they will have all the support needed to get there efficiently.

— Marcel den Hartog

February 17, 2017
Is Cloud Adoption on Your Company’s Horizon?

Under pressure to increase the speed of innovation in the digital space and outpace competitors, companies large and small are turning to the cloud. Moving to the cloud provides flexibility of infrastructure, transparency of infrastructure costs and many other benefits.

However, making the transition to the cloud is far more intricate for regulated industries—especially large financial organizations—where security and compliance concerns often trump innovation efforts. According to Cloud Security Alliance, the financial industry is still in the early stages of cloud adoption. In fact, 61 percent of financial leaders say their cloud strategy is in the formative stages. Those financial companies that can shift to the cloud properly will see long-term gains, from both a performance and economic standpoint.

In 2015, while working at a large international bank, I assisted on a project to help transform how the bank approached digital for retail banking and wealth management. The bank was still approaching most projects as large, monolithic endeavors, where six-month delivery cycles were standard. Infrastructure was owned by IT Operations, and provisioning lacked automation. As such, almost every project began with designing environmental needs from scratch—adding time to both creation and delivery. It was quickly evident that cloud was the bank’s best option, allowing for increased transparency, greater scalability, infrastructure on demand and improved organizational efficiency.

Shifting to the Cloud

After working on this project, I learned that to effectively move to the cloud, organizations must understand how to maximize benefits and minimize risks. Here’s how:

Apply a two-pronged approach

We realized right away that we needed a system where we could meet our project objectives, while simultaneously exploring the cloud options. To avoid jumping in too quickly and running into non-technical challenges, we adopted a two-pronged approach: we’d build the platform internally using Cloud Foundry PaaS, while exploring the use of external cloud in parallel—without impacting the delivery of the internal platform. To avoid re-architecture and rebuild later on, we had a guiding principle: Whatever is built internally has to be available in the external cloud as well.

Navigate legal and compliance hurdles

Regulators are still on the fence when it comes to certain industries adopting the cloud, with regulations still lacking specific policies. For example, in the financial space, lack of close regulation leaves many institutions on their own to sort out contracts with cloud providers based on their individual risk appetite. When regulations are published, they are very high-level and generally state that managing the risk and responsibility for integrity of data is with banks, not with cloud providers. Instead of creating an equal playing field, this tilts the scale toward smaller fintech partners with larger risk appetite±and who more regularly fly under the regulator’s radar.

Expect roadblocks

In so many organizations, IT operates in silos. For example, all infrastructure is owned and operated by the IT Operations group, which then recharges that infrastructure back to the businesses based on complex metrics. For cloud to succeed, we need to answer the question of ownership. But that’s not all: Many folks may not believe that organizational issues are a serious problem, and thus, will lack a commitment to be involved. To ease this problem, strong sponsorship from senior leadership is essential.

Prepare for technical issues

When implementing a technology solution, things don’t always go smoothly from the start. For example, there might be approved ways to connect with third-party providers that were created before cloud providers came into play. While this kind of issue is easy to solve, it requires a change in internal procedures that must be accounted for.

Focus on development and testing first, then on production

While working toward final approval and negotiating the agreement, which can take significant amount of time, conducting a trial run is an effective way to work out any potential problems before full implementation. Signing an agreement to use cloud for development and testing only is less of a hurdle, as no customer data will be transmitted or stored. It’s also important to train anyone who might be involved throughout the process—no matter how big or small their role.

Conduct a formal project launch and set clear goals

To drive continuous improvements, it’s essential to establish clear objectives and goals. This provides metrics for benchmarking success and ROI, and encourages the brainstorming of new ideas on a case-by-case basis. Prior to the formal launch, don’t forget to identify key stakeholders and advocates—they will stand behind the project and provide valuable insight into the potential outcomes of decisions.

Create a transparent culture

Full transparency is key, even if it means dealing with resistance. Resistance, especially early on, often is unavoidable. However, it is better to deal with resistance, gain support and handle any issues early in the project. Being transparent gains you more support. Or, if the opposition is insurmountable, you can stop the project before wasting too much time or effort. Most important, you want to avoid criticism suggesting that the project is putting the company at risk. By looping in the right people from the start, you’ll have advocates in your corner who will raise red flags and anticipate problems on the horizon.

Remember that people will want to be involved

While cloud may still be an unfamiliar concept for many, almost always someone will come out and say, “Actually, I have experience, and already thought about how to do this. How can I get involved?” Bring those people on board, leverage additional insights and increase commitment to the project. This is an exciting change—people want to be on the team that introduces it!

Large, regulated companies historically have been slow to implement the cloud, due largely to perceived compliance concerns and security risks. However, in an environment marked with rapid innovation, companies are quickly realizing that to stay ahead, adopting a cloud-first business model is a must.

About the Author / Alexander Kalinovsky

Alexander Kalinovsky is Director of Engineering at Wipro Digital. He is a senior IT leader with extensive experience managing successful application development and operational teams, along with establishing global development practices and partnering with global organizations. Alex has spent over 10 years working in Financial Services in development and infrastructure, specializing in payment systems and FX solutions. Alex brings his strong FS background and enthusiasm for Agile development to his role at Wipro Digital, driving the digital transformation agenda for many of the world’s leading brands.

February 15, 2017
Communicating and Collaboration in a Digital Age

The way we communicate with the world has changed drastically in the last 20 years. Gone are the days of pen and paper, lengthy phone calls and, to a certain extent, even emails. Gone are the days of needing to have your entire team under one roof to accomplish anything. Gone are the days of communicating from one limited perspective.

In our digital day and age, we can move mountains with keyboards, perform miracles via Skype and enjoy a thousand different perspectives of any project. There is one key behind all of this progress: code. Without it, we would not be where we are and couldn’t accomplish any of the amazing interdependent things we do. In fact, STEM (science, technology, engineering and mathematics) is one of the fastest-growing areas of our economy, according to TechRocket. Between 2014 and 2024, the number of STEM jobs will grow 17 percent, as compared to 12 percent for non-STEM jobs.

Unfortunately, not everyone speaks this language, which can lead to a breakdown of communication in the office when the goal is to work as a diverse but cohesive team. It’s infinitely more difficult to collaborate with someone when you can’t translate each other’s visions. That’s why it’s important to have a basic understanding of this vital skill if you want to build successful teams.

Bridge Communication Gaps

I’m not saying everyone on your team needs to have the expertise of a developer, but it is helpful if everyone can learn how to code from a very basic level. This basic understanding helps everyone understand the time and effort involved from their tech peers, the boundaries of what can and cannot be accomplished, and the importance of clear, precise details in communicating their ideas.

It also makes troubleshooting and revising much easier to pinpoint and less time-consuming to correct. You can skip the IT-to-English translation that so often degrades communication on major projects because neither party feels they’ve been heard and understood.

Encourage Collaboration Throughout the Process

A recent poll by Gallup showed that 70 percent of U.S. employees are not engaged at work. Luckily, there are plenty of opportunities for companies to bolster engagement through effective communication channels that keep everyone in the loop and invested in the ultimate outcome.

Oftentimes, ideas flow from conception to initiation to implementation without ever having crossed the desk of the programmers charged with writing the code to bring these ideas to life. That’s the same as asking a carpenter to build a car. Although the carpenter may have a great design, understand the market and even be able to build some pieces of the car, at the end of the day the carpenter’s skills aren’t enough to do it alone and the carpenter’s vision will need to be communicated to someone with a different skill set.

When everyone on the team understands how code works and the resources needed to turn these fantastic brainstorms into reality, the entire process becomes more streamlined and can be executed more efficiently.

Be in the Know

At the end of the day, everyone on the team benefits from understanding how the machine is programmed to work. Whether you’re launching a new website or the latest iteration of your app, somewhere along the line you will need to write code to convey your message in this digital world—and you will want that message to come across loud and clear.

The good news is that it’s easy to harness the skill of coding these days. Employees can participate in a coding bootcamp, online courses or even mentorships with other employees who’ve already mastered the skill. The improved communication among teams and departments will help improve efficiency and help you realize visions.

About the Author / Michael Peggs

Michael Peggs is the founder of Marccx Media, a digital marketing agency specializing in Search Engine Optimization (SEO) and Search Engine Marketing (SEM). Before Marcxx, Peggs worked at Google in business development, forming digital media and advertising partnerships in the United States and Asia. He is also a contributor to The Huffington Post, FastCompany and Business Insider and a podcaster, hosting the iTunes Top 10 New & Noteworthy Podcast You University. Connect with him on Twitter.

February 8, 2017
4 Top Tips for Developing a New Feature

As a backend developer at CA BlazeMeter, I recently had the chance to work on a big feature for our web app. The feature required a massive change on our backend code, a big change on our frontend and multiple migrations. You know what I’m talking about—it was one of those projects that developers fear.

But, I gained a lot of valuable experience from the project. Not only did we deliver a kick-ass feature, I also learned the importance of teamwork and patience when developing something new. Armed with those two values, and the right tools and technologies, you can deliver a new feature quickly and efficiently.

Here are a few more things that I learned that should prove helpful whether you’re developing a major feature or one that only requires a minor code change.

Tip 1: Plan to understand the development requirements

The process of development starts way before you start coding, with planning. After the product manager decides what needs to be developed, R&D needs to understand the cost: how many days (or weeks) of development the feature will take, how many people will be needed and which APIs are required.

To do that, the product manager needs to conduct at least three “grooming” sessions with backend, frontend and QA leaders. The grooming sessions start with the product manager presenting the product flow and requirements.

Since no one wants to read a lot of text (except perhaps the awesome readers of this article), the product manager should prepare mockups. They don’t have to be designed or colorful, but they do need to show the team what the feature looks like in his or her vision. During our grooming sessions, we used Balsamiq, which lets you create screen sketches for how the product is supposed to look, feel and work.

Editing screen sketches using Balsamiq

After the initial presentation, the sessions should include a lot of open discussion—asking questions, checking for answers and examining compliance, etc. —until everyone is satisfied and understands what the feature requires from them and from others.

This is the time and place for frontend, backend and QA to give their opinion about the schedule and to alert and flag any potential problems in development or extreme use cases. It’s the role of the product team to answer questions and change the flow and requirements until there are no R&D rejections.

Afterward, the frontend and backend leaders need to create a feature development plan, sitting down to determine what data they need to transform the product vision into a real thing. Or in other words, which APIs are required for each flow or button, how long each step is supposed to take, including buffers, and the order of development. APIs are the agreed “contract” between the development teams. After determining the contract terms, each team can proceed individually.

I recommend you write your APIs in Swagger, which turns YAML files into detailed HTML pages with all the APIs.

Swagger Editor

This plan needs to be transparent to everyone working on the project. In fact, it’s even better if each project participant can add and describe scenarios. Tools such as TestLink and Atlassian do just that. Then, when the code is ready, developers can go over the list and see everything is covered.

We will need to stick to this schedule later on, especially when new demands unrelated to the feature come in, such as fixing bugs. Sticking to this plan is the best way to distinguish the wheat from the chaff of the development process.

Start writing automation ASAP

After planning, we want everyone to start working, and not only the backend. To do that, the backend can prepare mocks, either through outsourcing services such as JSON servers, which show inputs and outputs, or by writing dummy code.

I mentioned earlier that we used Swagger to write APIs. We also used it to generate a server in NodeJS/Python, like this:

Generating a JSON Server through Swagger

From this moment on, frontend can begin writing and building screens and QA can start writing API automated testing, by using tools suh as JMeter. At this point, frontend and QA have the response codes and response bodies; they know which features to put where; and which APIs they need and how to use them.

QA also know which data they’re missing, such as field names and field data. Sure, the backend code isn’t ready yet and all of the automated exams will return “red” instead of “green,” but the main issue is making sure everyone is as prepared as possible.

This includes load and performance testing. You can use open-source tools such as JMeter, which is enhanced by CA BlazeMeter. By uploading our test script into, configuring the number of virtual users and ramp-up time, determining geo-locations to test from and analyzing KPIs to determine trends and the health of our system, we can ensure our product is delivered to our customers without crashing.

This is also true about sending code to QA: Developers should test their code themselves and ensure that when they commit their code, it works 99.9 percent of the time.

By starting automation as soon as possible, you can minimize future bottlenecks in the development process.

Ensure the team communicates and works together

Like I said before, teamwork is the most basic requirement when creating a new feature. This is because new questions arise during development and because requirements change. Therefore, everyone in the team needs to be in sync all the time, from the product manager and the designer to the developers and QA.

Open communication lines are crucial because everyone is working on the same product at the same time, but each developer or QA should be working on a different part of the product and on different functionalities. Communication ensures each person’s work is compatible with the rest of the team’s work. This saves time, prevents frustration and redundancy and ensures maximum professionalism.

To make sure everyone knew what was happening, we used Slack and opened a dedicated channel for the creation of this feature. We also had daily (short!) standups where we discussed what we have and what is missing.

Opening a Slack channel

Communication also includes team and assignment management. JIRA, our primary issue tracking tool, offers Kanban-style, Scrum and fully customizable boards, which show the progress of each development task, including the developer assigned to it, and his or her exact status or phase.

This helps you understand what other people are working on, and just as importantly, where people are stuck. If you can, get everyone to work in an open space and put a large screen in the middle of it. This lets you stay in sync at all times.

We used JIRA to open and manage bugs as well. This was helpful when multiple QAs found the same bug. It’s important to carefully choose which tickets are opened and put them under one epic, so everyone stays focused.

JIRA Dashboard

Another important aspect of communication is ensuring everyone uses the same terminology, as sometimes people might use the same term to describe different things or different terms to describe the same things. To prevent mixups that could waste your time, make sure everyone is on the same page. For example, do you say “system admin” or “<name (ex. BlazeMeter)> admin”?

Work on the same development environments

It’s important that everyone works on identical development environment s and even on the same version of the tools and libraries. Otherwise, it could cause an (in)compatibility mess, including a lot of unnecessary back and forth and even might cause bugs to pop up. This environment needs to be as similar as possible to production, but only on a smaller scale.

Conclusion

Now look at the feature that came out of all that work here. I’m so grateful that I had the opportunity to work on it with an awesome team.

I hope these best practices help you the next time you develop a new feature. Don’t forget to keep going agile after development and delivery by running tests in a continuous environment such as Jenkins for every commit.

CA BlazeMeter works with Jenkins, ensuring you can load test in the cloud and analyze your data while shifting left. You can also use different APM tools, such as New Relic, AppDynamics and CA APM to alert you about errors before QA, which saves time.

— Dor Atias

February 7, 2017
Strategies to Build Trust in DevOps
I’ve been a member of the DevOps community a long time (literally since before it was ever even called “DevOps”). I was a pre-publication reviewer for both “The Phoenix Project” and the recently published “The DevOps Handbook,” and I’m always interested in seeing the outcomes successes software teams have and the failures they overcome, including trust.

While DevOps has proven to be effective in organizations of all sizes—startups and enterprises alike—there are still many who have not yet begun their DevOps journey, so I thought I might share some of my own experiences while introducing DevOps into an enterprise IT group.

When I was the VP of Operations at Big Fish Games, I had the responsibility for production operations at Big Fish’s three production data centers in Seattle, Virginia and Luxembourg. I also had responsibility for internal corporate IT and information security. Our production operations supported around a million game downloads per day. Games were localized into 11 different languages. We processed essentially every major currency, and right before I left Big Fish, we started taking Bitcoin as well. From an IT perspective, Big Fish Games was a global eCommerce company selling digital widgets on the internet. We had around 800 employees in our five offices in Seattle, Ireland, Oakland, Luxembourg and Vancouver, BC.

My first attempt to introduce DevOps at Big Fish failed miserably due to cultural misunderstandings and a fundamental lack of trust between the development and operations teams.

You absolutely need some minimum baseline level of trust in your organization if you are to have any chance at implementing DevOps. Trust is important because you are blurring traditional boundaries between development and operations responsibilities. Including development in production operations can be scary for operations teams. It just feels wrong from a historical ITIL-type perspective, and operations teams may know development as “the people who always break things.” At the same time, it can be scary for development to involve operations in the development process because they may know operations as “the team that always says ‘No.’”

Building Trust

Successfully merging the priorities and goals of development and operations teams to create one cohesive DevOps effort can fail due to cultural misunderstandings and, oftentimes, a fundamental lack of trust. I used three specific strategies to overcome these challenges.

Lose the Term ‘DevOps’

At Big Fish Games, application development teams equated DevOps with developers carrying pagers: an idea that they really did not like. They were highly resistant to this. Every time I brought up DevOps, they thought I was engaged in a secret plot to make the developers carry pagers. What I found interesting about this was that I had no intention of asking developers to carry pagers.

The term “DevOps” covers a large range of continuous improvement techniques. Some organizations have developers carry pagers, and it works for them, but this is certainly not a prerequisite for DevOps. What was worse, I had no idea they were thinking this because they were afraid to bring it up for discussion. We failed in our first attempts to implement DevOps because the trust level between development and operations teams was so low that we couldn’t even tell each other what we were thinking.

I realized two things from this experience:
- Getting rid of a term that scared both development and operations teams would eliminate an unnecessary distraction preventing these teams from coming together to focus on common goals.
- We needed to do something about the trust issues between development and operations before we could succeed with DevOps.
Once I realized this, I immediately stopped using the word “DevOps.” Instead, I talked about continuous improvement and specific continuous improvement methodologies that we wanted to implement, and how these new adoptions would result in shared, organizational-wide growth.

Focus on Methods That Benefit Other Teams

When deciding which of these DevOps, a.k.a. “continuous improvement,” techniques we wanted to implement, I specifically chose ones that would benefit development teams as opposed to ones benefiting operations or other teams. The development teams noticed this over time. They could see I was doing this and that I (the person who ran operations) was going out of my way to help them. This helped build trust. This wasn’t a one-time thing. We did this consistently.

Here’s another simple failure example: I noticed while managing release efforts that release schedules for the various development teams ranged from a couple weeks to a couple months—too long for a hyper-competitive industry like gaming. I asked the development teams if they would like our release engineering team to support daily code releases or even multiple code releases per day. I thought this was an innocuous question. In fact, I thought I was being a little clever by playing dumb here. I assumed I would get an enthusiastic high-five along with a, “Yes, we’d love that! How soon can you make that happen?!”

I was surprised when I received a violent “No! We don’t want that!” This confused me for a while until I realized that the development team thought I was in cahoots with the product management team to put the squeeze on them. In my mind, I was offering to make their release process more lightweight so they could break up their work into smaller, more frequent releases with less stress for everyone, but I didn’t understand their fears and problems. I wasn’t looking at it from their perspective and here was the trust issue rearing up again. They misinterpreted my intentions as nefarious.

I pictured them being able to release code whenever they want, which seemed like something they would like. In their minds, they pictured a product manager constantly randomizing them with demands for new features to be released right there while the product manager was leaning over their shoulder. This was not the kind of problem you could resolve with logic. The solutions and working procedures I was proposing were so far outside the development team’s experience that they couldn’t extrapolate through simple logic from their current state to the world I was describing.

Maximize Daily Interactions Between Development and Operations Teams

It’s easy to distrust someone you don’t know personally or you don’t have to see every day. It’s a lot more difficult to hate on someone you know and work with on a daily basis.

One of the DevOps techniques that I implemented early on was embedding a dedicated operations liaison into each development team. This liaison acted as an advocate for that development team back into operations and they helped smaller development teams navigate their work through a large, complex, centralized operations organization. The operations liaison attended daily development team standups, group meetings and project meetings. This program was a huge success. Development teams saw how their operations liaison helped expedite their work through the operations organization and advocated for their development team’s interests inside operations.

After the development teams got used to working with their operations liaison on a daily basis, we pushed a bit farther and got the development teams to agree to include operations in architectural discussions at the beginning of application design. At first, our development teams were skeptical that operations could provide useful input here, but the important thing is we had developed enough mutual trust that they were not fearful of including operations in the discussions. They were skeptical that we would add any value, but they weren’t afraid that we would cause them problems. This was a big win.

Although it felt a little weird to our development teams to include operations in design discussions, they quickly saw how obtaining input from operations early on, before a product was built, resulted in better quality and fewer headaches for everyone when we launched into production. I also made sure that we put our most senior and diplomatic operations team members into this role.

In conclusion, it is worth repeating that trust is a prerequisite for any organization to succeed with DevOps. This is due not only to problems arising from fuzzy definitions of what DevOps is and is not, but also the seemingly dangerous idea of blurring the boundaries between development and operations organizations. Whatever your definition of DevOps is, fostering an increased level of trust between development and operations teams as a prerequisite for your DevOps journey will ensure that even greater trust—and the positive business impact that comes with it—are your outcomes as well.

Good luck!

About the Author / Paul Farrall

Paul Farrall is the VP of Operations for Skytap. He was most recently vice president of Operations at Big Fish Games where he led the rollout of global infrastructure for the Big Fish cloud gaming platform and implementation of the Big Fish DevOps program. Prior to Big Fish, Paul was the Director of Operations and Chief Information Security Officer at Intelius, a Bellevue e-commerce company. Previous companies also include AT&T Wireless in Bothell, Washington, and GetThere.com in Palo Alto, California (now Sabre). Connect with him on LinkedIn and Twitter.
January 26, 2017
Relevant Continuous Testing: The Primary Key to DevOps
My recent blog, “7 Pillars of DevOps,” identified seven essential best practices areas that make up successful DevOps and explained that the pillars are not silos, but rather foundations that need to be kept in balance. One of the pillars is continuous testing (CT).

For well-balanced DevOps, CT performs tests that are fast enough to just fit within the time budget, yet thorough enough to cover the quality needed for deployment to production. Despite the need for balance, CT can be considered the true backbone of DevOps because testing is the one pillar that proves the validity of changes needed to operate and validate the correct operation of the other pillars from end to end across the pipeline.

Compared to older testing strategies in which large software changes were tested as a complete product after “release to QA,” DevOps CT strategically tests changes incrementally over a series of DevOps stages. CT begins the moment a change is made in development and continues through all DevOps pipeline stages. Release readiness is judged on analysis of test results accumulated prior to deployment. With Blue-Green, A/B and Canary test strategies, some tests may continue into the live environment after deployment, and the results may influence future CT activities.

Like continuous integration (CI), frequent, small, incremental testing has many merits. Quick feedback on problems enables a release consisting of many individual changes to be tested incrementally with confidence. The root cause of problems can be isolated much faster when the changes are tested incrementally. The problem is determining which tests are most relevant to run at each stage of a DevOps pipeline with content, with the risk of changes varying widely from one change to the next.

Given that testing is a stochastic sampling process, there is no such thing as 100 percent testing in any absolute sense. Left unchecked, the content of tests will grow exponentially beyond the content of the product that is being tested. No number of test resources, even if elastic on-demand cloud infrastructures are used, will keep up with the growing demands to satisfy test requirements. The key is to create and choose tests that are most relevant to the changes that are being processed in the pipeline rather than attempting to run all tests with no regard to actual matter what the changes you wish to inspect. A strategy that chooses the most relevant tests at each stage in the pipeline is optimal.

Several strategies for creating and selecting relevant tests are described below. These relevant test strategies may be used standalone for specific DevOps stages or a plurality of these relevant test strategies may be used end to end across the pipeline.
1. Choose tests that focus on source code deficiencies. Metrics based on static analysis scans can determine which areas of source code have the highest number of defects. Test creation and selection algorithms can provide higher weight to tests written for those areas. This seems like a good idea; however, there are several limitations. Scan tools are limited to specific languages, may miss critical defects and do not have intelligence regarding the systemwide relevance of a specific defect.
2. Create and choose tests for areas of the code with low test coverage. This also sounds like a good idea, but it is not. The most obvious reason is that tests probably are not available to select or the coverage would have been higher in the first place. Also, code coverage algorithms, especially at the system level, are limited to simple metrics such statement coverage, which is not a good indicator of system-level performance.
3. Create and choose tests that match the artifacts of build changes including build dependencies. In this approach, tests are mapped to build changes and tests are selected that match the changes. This sounds logical, but unfortunately, a static mapping between test and build artifacts may not reflect system impacts of changes, as critical failures may occur due to interactions with modules outside the build dependency chain.
4. Create and choose tests per risk analysis. This approach keeps a running record of code changes correlated to test results. Tests that tend to fail the most when a code module is changed are selected. This approach has several advantages over the other approaches. Test selection is not limited to implementation language or build dependency maps. Empirical experience correlates to risk better than code structures in the stochastic context of testing. The same algorithms can be applied to every stage in the pipeline. And the pipeline metrics can be adjusted dynamically for each pipeline stage run.
The above CT practices indicate that relevant testing strategies reduces the time for CT execution and reduces the infrastructure cost for CT, without sacrificing quality delivered to production.

Summary

What do you think? Do you agree with the above relevant continuous testing practices? Are there others you recommend?

While DevOps implemented with all seven pillars provides a strong foundation for long-term enterprise business improvements, it is important to understand DevOps is not an island. Enterprises implementing DevOps should be aware that DevOps interoperates with other IT systems and practices. Enterprises are well-advised to put in place governance policies that encourage the selection of tool-agnostic IT partners with solutions that best suit the needs of each unique enterprise and can integrate and evolve DevOps together with all their IT systems.

— Marc Hornbeek
January 19, 2017