devopsy.kinsta.cloud

Tag: QA

All Day DevOps: System Hardening with Ansible
The DevOps pipeline is constantly changing. Therefore, relevant security controls must be applied contextually.

We want to be secure, but I think all of us would rather spend our time developing and deploying software. Keeping up with server updates and all of the other security tasks is like cleaning your home: you know it has to be done, but you really just want to enjoy your clean home. The good news is you can hire a “service” to keep your application security up to date, giving you more time to develop.

At the recent All Day DevOps conference, Akash Mahajan (@makash), a founder/director at Appsecco, discussed hardening your system’s security with Ansible. In addition to his role at Appsecco, Akash is involved as a local leader with the Open Web Application Security Project (OWASP).

Misconfiguration

During his presentation, Akash mentioned the OWASP Top 10 Security Vulnerabilities list, zeroing in on No. 5: Security Misconfiguration. To determine if you comply with the guidelines, No. 5 on the list asks:
- Is any of your software out of date?
- Are there any unnecessary features enabled/installed including ports, services, accounts, pages or privileges?
- Are default accounts and their passwords enabled/unchanged?
- Are security settings and libraries not set to secure values?
I am sure no one reading this article still uses the default administrator password, but can we say the same of your peers? Have you gotten around to installing the latest software patches on your server?

Automation

If a task can be automated, developers automate it. So we should automate our security tasks too, where we can. OWASP provides guidance here, suggesting you should:
- Have a repeatable security hardening process.
- Ensure your development, QA and production servers are configured identically but with different passwords.
- Automate the process to minimize the effort required to setup a new secure environment.
- Implement a process for deploying all new software updates and patches in a timely manner to each deployed environment.
- Run scans and audits periodically to help detect future misconfigurations or missing patches.
This is all part of security hardening, which is, “the process where we identify default configuration present on a system and apply changes that will change the configuration to secure values.” This can be applied to your network, transport, application, and kernel networking parameters.

Ansible Playbooks

Ansible is one of the solutions Akash likes to work with, but there are others solutions on the market that provide similar value. Without trying to endorse or evaluate one solution over another, let me share perspectives from Akash’s experience with his toolset.

Why does he like it? It boils down to playbooks. An Ansible playbook is a codified security document, allowing you to describe the desired state of a system, rather than the specific steps of how to get to that state. As Akash points out, things change—it is better to have the end state described rather than have to change commands when the system changes.

Other advantages of playbooks include:
- Playbooks are written in YAML providing us with structure that we can learn and train on
- Playbooks are text files, so we can use Git for version control
- Managing playbooks is just like managing any software project
- Playbooks are infrastructure as code but for security
- Playbooks consist of roles, a key aspect of security
- Numerous playbooks are available as open source
The bottom line is you can—and you should—automate your security hardening process. Your users and other stakeholders will thank you and, most of all, you will thank yourself because you can spend more time on the things you love to do.

Ansible is just one example of a solution that can be used to automate your security tasks. If you want to know more, Akash goes into further detail on getting started with Ansible in his full All Day DevOps conference session (just 30 minutes). The other 56 presentations from the All Day DevOps Conference are also available online, free of charge here.

This blog series is reviewing sessions from the All Day DevOps conference from November which hosted over 13,500 registered attendees. Last week I discussed, “DevOps at Massive Scale.” Next week, look for “Operationalizing a Red Team for Fun and Profit,” delivered by Intuit’s own Ian Allison.

— Derek E. Weeks
February 27, 2017
Reality Check: Bringing Continuous Delivery Dreams to Life

Every developer shares the same dream: to write a piece of code or app that changes everything.

Whether it makes your company do things more efficiently, attracts hordes of new customers, simply makes existing clients happier or maybe even changes the world a little bit, that code is what we strive for. Some of us don’t merely harbor that dream; many of us actually achieve that moment of brilliance. For those who have, it’s remembered as a moment of pure joy.

So, where does this magical process begin? First, you talk to your manager and, if you’re lucky, she or he agrees that it is a fantastic idea and starts to sell it internally. Sometimes the idea is so good that you get the go-ahead on the same day and even get a small team to work on it!

Then, soon thereafter, reality kicks in.

You need data, inevitably from applications running somewhere in the cloud and/or some other data from on-premises data stores. You need to interface with APIs, some well-documented, some not. You need some user experience work done, and the standards your company has adopted don’t really match your vision.

You also need to interface with many of the standard services your company has already implemented because, after all, your app doesn’t run in isolation—you work with existing billing systems, a CRM system, a fulfillment system … and all of them run on servers that can be anywhere. Soon you find yourself fighting for test data, systems to test with and applications and APIs to interface with. Before you know it, you’re banging your head against the wall.

Fortunately, your teammates are flexible, creative people. And because your internal systems seem to work against you, you go around them. You download a trial version of some testing software; you install a copy of some internal systems on a virtualized test server (if you can find one) and start working with these.

Since your app needs to be snazzy and snappy, you download some open-source software to help you test the performance on mobile devices. After three sprints, you’re now ready to move your app to an environment where you can start performing real tests!

Most times this is the moment when the hardest reality kicks in. The copies of the systems you tested with were not up to date. Data has changed. APIs have changed. The real system responds three times slower than the copy that sat on the server right next to you. The data in the cloud has changed. The performance testing tool you used is not available in the official testing environment. … The list goes on.

This doesn’t even include the difficulties of moving all the components of your new app to different environments. Trust me, this is not a fictional nightmare scenario; it is reality and I see it happen all the time.

Now, what if you had the test-data generators necessary to create the data you need and a server that ran virtualized services, mimicking all the (up-to-date) external systems you need including the APIs to talk to them—services that would allow you to say, “What if this application responds three times slower when I request data?” or, “What happens if this system suddenly goes down?” What if, when you’re done building, you could simply migrate the app and all the components that go with it to another (test/QA) system with one click?

Everybody wants to support people with good ideas and turn these ideas into working software without the frustration that very often distracts and demotivates people. To do so, you need to create the infrastructure that supports this. Is it difficult? Not really, the tools to automate services, create test data (and test scenarios) and migrate complete applications from A to Z are available.

However, just like a great app, this too starts with a dream—the dream of allowing people with great ideas to work on these ideas without the overhead of an infrastructure that allows them to focus on the coding and not on the infrastructural issues; the dream to create a software factory where the creation of the app takes 90 percent of the time and the infrastructural burden only 10 percent, not the other way around.

If you’re serious about developing software and want to create an environment where the infrastructure allows people to develop and release software faster instead of working against them, find the tools that support it and jump in, because they exist today.

Enacting this change now and maybe the next time someone comes to you with a great idea you can be confident that they will have all the support needed to get there efficiently.

— Marcel den Hartog

February 17, 2017
4 Top Tips for Developing a New Feature

As a backend developer at CA BlazeMeter, I recently had the chance to work on a big feature for our web app. The feature required a massive change on our backend code, a big change on our frontend and multiple migrations. You know what I’m talking about—it was one of those projects that developers fear.

But, I gained a lot of valuable experience from the project. Not only did we deliver a kick-ass feature, I also learned the importance of teamwork and patience when developing something new. Armed with those two values, and the right tools and technologies, you can deliver a new feature quickly and efficiently.

Here are a few more things that I learned that should prove helpful whether you’re developing a major feature or one that only requires a minor code change.

Tip 1: Plan to understand the development requirements

The process of development starts way before you start coding, with planning. After the product manager decides what needs to be developed, R&D needs to understand the cost: how many days (or weeks) of development the feature will take, how many people will be needed and which APIs are required.

To do that, the product manager needs to conduct at least three “grooming” sessions with backend, frontend and QA leaders. The grooming sessions start with the product manager presenting the product flow and requirements.

Since no one wants to read a lot of text (except perhaps the awesome readers of this article), the product manager should prepare mockups. They don’t have to be designed or colorful, but they do need to show the team what the feature looks like in his or her vision. During our grooming sessions, we used Balsamiq, which lets you create screen sketches for how the product is supposed to look, feel and work.

Editing screen sketches using Balsamiq

After the initial presentation, the sessions should include a lot of open discussion—asking questions, checking for answers and examining compliance, etc. —until everyone is satisfied and understands what the feature requires from them and from others.

This is the time and place for frontend, backend and QA to give their opinion about the schedule and to alert and flag any potential problems in development or extreme use cases. It’s the role of the product team to answer questions and change the flow and requirements until there are no R&D rejections.

Afterward, the frontend and backend leaders need to create a feature development plan, sitting down to determine what data they need to transform the product vision into a real thing. Or in other words, which APIs are required for each flow or button, how long each step is supposed to take, including buffers, and the order of development. APIs are the agreed “contract” between the development teams. After determining the contract terms, each team can proceed individually.

I recommend you write your APIs in Swagger, which turns YAML files into detailed HTML pages with all the APIs.

Swagger Editor

This plan needs to be transparent to everyone working on the project. In fact, it’s even better if each project participant can add and describe scenarios. Tools such as TestLink and Atlassian do just that. Then, when the code is ready, developers can go over the list and see everything is covered.

We will need to stick to this schedule later on, especially when new demands unrelated to the feature come in, such as fixing bugs. Sticking to this plan is the best way to distinguish the wheat from the chaff of the development process.

Start writing automation ASAP

After planning, we want everyone to start working, and not only the backend. To do that, the backend can prepare mocks, either through outsourcing services such as JSON servers, which show inputs and outputs, or by writing dummy code.

I mentioned earlier that we used Swagger to write APIs. We also used it to generate a server in NodeJS/Python, like this:

Generating a JSON Server through Swagger

From this moment on, frontend can begin writing and building screens and QA can start writing API automated testing, by using tools suh as JMeter. At this point, frontend and QA have the response codes and response bodies; they know which features to put where; and which APIs they need and how to use them.

QA also know which data they’re missing, such as field names and field data. Sure, the backend code isn’t ready yet and all of the automated exams will return “red” instead of “green,” but the main issue is making sure everyone is as prepared as possible.

This includes load and performance testing. You can use open-source tools such as JMeter, which is enhanced by CA BlazeMeter. By uploading our test script into, configuring the number of virtual users and ramp-up time, determining geo-locations to test from and analyzing KPIs to determine trends and the health of our system, we can ensure our product is delivered to our customers without crashing.

This is also true about sending code to QA: Developers should test their code themselves and ensure that when they commit their code, it works 99.9 percent of the time.

By starting automation as soon as possible, you can minimize future bottlenecks in the development process.

Ensure the team communicates and works together

Like I said before, teamwork is the most basic requirement when creating a new feature. This is because new questions arise during development and because requirements change. Therefore, everyone in the team needs to be in sync all the time, from the product manager and the designer to the developers and QA.

Open communication lines are crucial because everyone is working on the same product at the same time, but each developer or QA should be working on a different part of the product and on different functionalities. Communication ensures each person’s work is compatible with the rest of the team’s work. This saves time, prevents frustration and redundancy and ensures maximum professionalism.

To make sure everyone knew what was happening, we used Slack and opened a dedicated channel for the creation of this feature. We also had daily (short!) standups where we discussed what we have and what is missing.

Opening a Slack channel

Communication also includes team and assignment management. JIRA, our primary issue tracking tool, offers Kanban-style, Scrum and fully customizable boards, which show the progress of each development task, including the developer assigned to it, and his or her exact status or phase.

This helps you understand what other people are working on, and just as importantly, where people are stuck. If you can, get everyone to work in an open space and put a large screen in the middle of it. This lets you stay in sync at all times.

We used JIRA to open and manage bugs as well. This was helpful when multiple QAs found the same bug. It’s important to carefully choose which tickets are opened and put them under one epic, so everyone stays focused.

JIRA Dashboard

Another important aspect of communication is ensuring everyone uses the same terminology, as sometimes people might use the same term to describe different things or different terms to describe the same things. To prevent mixups that could waste your time, make sure everyone is on the same page. For example, do you say “system admin” or “<name (ex. BlazeMeter)> admin”?

Work on the same development environments

It’s important that everyone works on identical development environment s and even on the same version of the tools and libraries. Otherwise, it could cause an (in)compatibility mess, including a lot of unnecessary back and forth and even might cause bugs to pop up. This environment needs to be as similar as possible to production, but only on a smaller scale.

Conclusion

Now look at the feature that came out of all that work here. I’m so grateful that I had the opportunity to work on it with an awesome team.

I hope these best practices help you the next time you develop a new feature. Don’t forget to keep going agile after development and delivery by running tests in a continuous environment such as Jenkins for every commit.

CA BlazeMeter works with Jenkins, ensuring you can load test in the cloud and analyze your data while shifting left. You can also use different APM tools, such as New Relic, AppDynamics and CA APM to alert you about errors before QA, which saves time.

— Dor Atias

February 7, 2017
Don’t Leave Software Testers Out of DevOps

The term DevOps implies that software delivery is all about integrating the development and IT operations teams. But the software testing team is an important, if oft-overlooked, part of the picture. Here’s why.

Could you deliver software continuously without involving the QA team in the continuous delivery pipeline? You could try, but it would probably not go well. Software testers play a vital role in continuous delivery, for the following reasons:

They are the glue connecting developers to IT Ops. If there is no one to test the code that developers write, IT Ops cannot deliver a stable product.

They help find problems before they become costly. If you skimp on integration, unit and performance tests, you won’t find problems in your software until after it reaches users. By that point, fixing those problems will be much more time-intensive than it would if they had been caught early on. Plus, your organization’s reputation will have taken a hit, because problems that could have been nipped in the bud prior to delivery ended up affecting users.

They know things that developers and IT Ops don’t. If DevOps is all about continuous visibility across the delivery chain, then it only works well when the delivery team’s collective expertise covers all parts of the delivery process. Since developers and IT Ops people are not experts in quality assurance or testing, you need to involve testing professionals in the continuous delivery chain so that they can help everyone else understand the parts of the process that QA professionals know in detail.

All of the above is to say that if the QA team is not plugged into every stage of the delivery process, your software either won’t be tested efficiently, or it won’t be tested at all. Not testing efficiently means that the QA team is not in sync with the rest of the organization. In this case, the QA team would delay software delivery because it would have to wait on developers and admins to complete its work, and vice versa.

Not testing at all, of course, is even worse. No one should deliver unvetted software to users.

Don’t Forget about QA

The takeaway for organizations seeking to adopt DevOps practices is this: Even though the DevOps conversation on the surface may seem to be about only developers and IT Ops professionals, QA teams play an equally crucial role in continuous delivery. They need to be as involved in the continuous delivery pipeline as everyone else if an organization wishes to reap the full benefits of DevOps.

— Chris Tozzi

November 8, 2016
DevOps and the Identity Conundrum

“Who am I?” It is perhaps the most basic question of consciousness. The answer could be subdivided by aspects of my life story … my career, my family, my beliefs, my education … the list could continue. But since I am unique, the challenge in this digital age is for me to somehow remain unique, and whatever descriptions are used to catalog who I am, I must remain at the center of them all. If my identity is stolen, how does anyone know, that I am who I say I am?

This question is fundamental to every interaction I have with the digital world. And unfortunately, every system I interact with looks for me to redefine who I am, over and over again for each of them. I create a Facebook profile, a Twitter profile, a LinkedIn profile … and the list goes on again. But yet another weakness of our digital age is that there is no cross-reference between these systems to ensure I am who I say I am. This phenomenon remains true within DevOps tooling as well.

The Career Identity Conundrum

Let’s focus on only one aspect of who I am, my career. Who my current employer is dictates what electronic business systems and networks I should have access to. Switching jobs to another employer changes—or should change—the access I have. But getting more specific, what my role is at my current employer should also have an impact on what tools I have access to, and what I can do within them. And as my role changes, my abilities within a tool should change, as well. If I move from practitioner, to manager, to director, my abilities within a given tool should probably diminish, holding practitioners accountable for actions rather than superseding their responsibility by doing things myself.

Then there is the area of focus I work on. Business segmentation differentiates my work from product or service “A” in contrast with product or service “B.” My role might be an application programmer, or database administrator, but then “who” I focus on within a company is yet another level of differentiation. The business may prefer that teams of people focused on product “A” are restricted from making substantive changes on product “B.” So two different database administrators need two different levels of access within the same tooling, segregated by line of business or product line to accomplish it.

The Career Consistency Conundrum

The problem with my corporate identity in the digital age is that it is fluid, like life. In the course of my career I will change employers at one time or another. But beyond this, I will change roles, and/or areas of focus within the same employer over time. I may be assigned to work a “special project” in addition to my normal duties during my time at a given employer. And where DevOps relies upon several types of tooling to facilitate the continuum, this fluidity in my career makes it difficult to keep up with, and can drive administrative costs quite high.

For example, if I am an application programmer (normally), I will then have certain abilities within the build, deploy and release tools—likely high ability in build, moderate in deploy and low in release. If I am temporarily assigned as a release manager for a different product/service (in addition to my normal duties), it should radically alter my abilities within the DevOps tools, at least for a short time. But when I return to normal duties, the only mechanism that gets my abilities reset to what they should be is a strong process control system. The tools do nothing to facilitate this (no reminders, no examination of HR systems for roles, no look at each other to see what I am supposed to be).

An Identity Architecture

It occurs to me that the constants in resolving these kinds of problems are time and who I am. If I could hold an identification construct that identifies myself in career terms—which employer, which role, etc. and for how long—that identity construct could be employed into every system that accepted it. In effect, it would be the ultimate abstraction of user management, transferring the burden from companies and HR departments to the individuals who know, and need to know, the end user. While companies would still need to interact with my personal identity construct for verification, assent and record keeping, having all electronic systems accept my ID construct takes all of the work out of programming those interfaces, makes permissions and roles common, and keeps up with life’s fluidity.

In addition, who I am becomes the same in every system that accepts this construct. I cannot define myself as a release manager in one tool and a test manager in another and an application programmer in yet another to give myself god-like abilities in the DevOps tooling at my company. Each tool would examine the same identity construct I hold and receive from it, my normal duties, roles, permissions, etc. I would have a consistent profile across corporate tooling that would reflect my latest roles and use dates (beginning and end) to validate me against their own records.

An Identity Lockbox

The trick is, how do I keep this identity construct from being stolen? I believe this is where biometrics begins to play a key role. Fingerprint technology is commonplace now, even in cell phones, which are usually chained to us like an additional finger or toe. We rarely go anywhere without them. And when carrying them, we use features such as Touch ID to replace the manual keying-in of passwords. If my identity construct was paired between information likely I alone would have and biometrics tuned only to me (retinal scans, fingerprints, etc.), the pairing could unlock the construct for either use or revision.

Stealing the construct does little good, as without the pairing biometrics, it could not be used or revised. The applications for an identity construct like this are limitless—everything from securing the right to vote, work and collect social security to validating credit card purchases and opening up new social media accounts. If Facebook employed (or accepted) this kind of construct, it would be more difficult to create spoof accounts of me, as the unbeatable pairing could decipher who really owns a photo, or document, etc. And looking forward, a standardized identity construct such as this could interact with the emerging world of IoT (internet of Things), where devices could know a lot about me, and know it was truly me.

Impacts to the Bottom Line

I am not suggesting some wild end-of-the-world scenario where computers control our every move. We already have that in some form or another (see banks – ha ha). But I am suggesting that in the interest of reclaiming digital privacy we should be moving the ownership of identity constructs back to the people who own them: us. We should each own our own singular construct, keep it up to date and as filled out or complete as we want to, sharing only what we want to. The chief benefit of this would be an increased difficulty in stealing them from us. We can all skip the 666 tattoos, and instead try to reclaim a moniker of privacy by holding our own digital identities ourselves. Using them like a master key that will open systems we “should” have access to and abilities we “need” to have to perform the work assigned to us.

The upside to corporations is increased security that requires less on their part to maintain. A single set of records with start and end dates validates IDs used or attempting access. The upside to the software industry as a whole is a common interface for logging into an application with discreet identification. It removes the doubt about “borrowing” an ID if unlocking access requires a biometric pair as well. It would allow every application to abstract user management and make it common. Roles would be abstracted. Permissions could remain unique to each application based on their interpretation of the role.

Moving to a self-maintained identity construct could be taken further for benefits in other areas of our lives. While we have discussed some of how the career portions of our identity might be subdivided and cataloged, imagine for a moment the benefits of having our heath characteristics attached to the same kind of identity construct. Family records, memberships and associations … pick your classification of how a person is described for who they are. Having a way to retain ownership of those things in the digital age is key to keeping our unique position in the world unique. Losing control of those things becomes catastrophic. Where DevOps benefits from a construct such as this is but the tip of an iceberg compared to the benefits society reaps—or at least, our digital society could reap if we pursued it.

To continue the conversation, feel free to contact me.

— Kristian Nelson

September 26, 2016
Putting Ops Back in DevOps

What Agile means to your typical operations staff member is, “More junk coming faster that I will get blamed for when it breaks.” There always is tension between development and operations when something goes south. Developers are sure the code worked on their machine; therefore, if it does not work in some other environment, operations must have changed something that made it break. Operations sees the same code perform differently on the same machine with the same config, which means if something broke, the most recent change must have caused it … i.e. the code did it. The finger-pointing squabbles are epic (no pun intended). So how do we get Ops folks interested in DevOps without promising them only a quantum order of magnitude more problems—and delivered faster?

The problem is exacerbated when the topic of DevOps comes up, as it is most often a developer doing the talking. And the topics under conversation are things every developer would have substantial interest in and pretty much nothing that would interest the operations part of the equation. Many Ops folks feel as though they have been dragged into this DevOps phenomenon by osmosis, without their consent, and now they are supposed to be reaping huge benefits from it (by magic, as near as anyone can tell). Developers working faster is traditionally not something Ops folks look forward to; rather, it may be their worst nightmare. So, if the advocates of DevOps all tend to be developers, how do we get operations folks to become our evangelists as well?

QA Impacts on Evangelism

It begins with quality—not the ethereal concept of quality, but rather the specific responsibility for who owns it. Finger-pointing be gone; developers now own QA. If the software development lifecycle (SDLC) environment is broken, developers own it. Ops can sit back, toast marshmallows and enjoy the show. Developers should be including whatever configuration file changes their application needs as part of the deployable package. So nothing is done manually anymore. This means no one could forget something, as machines do not forget; they do what we tell them. If by chance, some random operations staffer decides to doink with a config variable, the deployment software should simply overwrite the manual change with what is in the deployable package created by the developer. Even malicious changes can be overwritten, so now, all change is included in the development class environment in the one and only deployable package. That is the only package that moves forward.

If developers own the QA, there is now a shift in the force: The operations team has a new role, one of coach and adviser, instead of doer. The software now is the doer, the developers are the packager and the operations staff are the witnesses of how it works. If it breaks, we all know who must fix it (and who is to blame). But once blame has been permanently affixed, something else changes, namely the culture. Now that the operations team does not take the blame, it remains interested in seeing its operations run without error. This allows operations to take an active “helping” interest in getting things fixed faster. It does not really matter who is to blame, once that argument has been settled; now it only matters if they can make something work. Harmony between factions begins to emerge.

Scalability Impacts on Evangelism

It continues with scale. Self-service is the nirvana of your average operations staffer. If the responsibility for provisioning can be moved into a push-button self-serve automation, nirvana will have been achieved. Imagine the offload of repetitious work tasks that consume the life of your average operations staffer. Instead of setting up (and taking down) environment after environment of one class or another for one application or another, requesters can all do it themselves with the push of a button. Even the accounting or financial impact is something the accounting team can guide the user on; operations does not have to worry about the cost of its operations anymore (or, at least, worry less). If too many environments are being provisioned, the accounting team has the power to change the rules and keep it from happening automatically without operations having to be the bad guy.

At this point, Ops can resume a focus on something it tends to enjoy more: monitoring. Ops staffers enjoy being able to provide real-time reporting on where things stand at any point in time. It is even more rewarding to provide more meaningful information such as who is using this application, what users are doing with it and when are they performing that activity. To monitor that kind of information, operations must partner with developers at the outset of a project to determine what kind of application logs are needed later for monitoring. Without the daily barrage of mundane provisioning tasks, an operations engineer is free to focus on more rewarding (translate: higher skill-level) tasks. The days of changing tapes for the backup may have reached their end (or pick your similar set of mundane tasks which DevOps automation can fully eliminate).

And Speaking of Monitoring

It merges into the future. How do you create an operations staffer as a DevOps evangelist? Start by exploring potential that only DevOps can enable. All those nagging alerts that are created when an environment goes down while a deployment is made … all of that could be turned off by the DevOps automation, then turned back on when the deployment is completed. Again, it all would be machine-driven, with no human mistakes anymore, ever. This will begin to cut down “noise” in monitoring/alerting systems into only useful information. However, integrating monitoring systems into the normal DevOps continuum extends beyond just an on- and off-switch.

Forward-thinking organizations begin to consider testing and monitoring as two parts of the same evaluation discipline. When a test is needed and when a monitoring characteristic is needed now can be determined in total rather than just in a development capacity or operations capacity. A holistic view of what can and what should be evaluated about an application, and how that will be achieved, brings the operations team to the table with the development and testing teams to plot holistic solutions that will live on beyond the development project timelines. Examining monitoring relative to version-controlled artifacts allows for organizational constructs within the monitoring discipline that never were possible before DevOps enabled them.

The Masters of Hardware Abstraction

Containers, or similar types of technologies, abstract an application from the hardware details it runs on top of. Technology such as this is a trend in DevOps that greatly simplifies the world of hardware from your average application developer. So who will retain the knowledge about the jungle that lives and breathes underneath? Ops has an extended role in understanding what lives underneath the abstraction layer that lives on top. Over time, only Ops will understand these particulars. They will become the only in-house experts about which cloud provider to use, which sets of physical hardware performs best and under what conditions.

It is this kind on unique expertise that makes an operations engineer extremely valuable (translate: pricey). It is the kind of knowledge careers are made upon. It is ageism-neutral. Older employees do this today, and they will continue doing this in the future; the only difference will be fewer and fewer people will know how it is done. If the domain of application development is the province of our youth, then the domain of hardware abstraction is easily the province of our more experienced employees. These realities create personal value for our operations teams, and they are only possible in a post-DevOps embrace.

Impacts to the Bottom Line

The adoption of DevOps by the Ops portion of the equation is critical to overall success. Passive-aggressive resistance by operations staffers can delay or destroy altogether the value of DevOps. Overcoming this is a first step. But creating operations evangelists goes an order of magnitude further: With active operations staffers becoming internal DevOps evangelists, the entire operations team can begin to look forward at what DevOps will ultimately enable for them. The value they will carry back into development efforts becomes substantial and the entire company benefits. Careers can emerge that were not possible before it—and, just as importantly, so can profits.

To continue the conversation, feel free to contact me.

August 26, 2016
DevOpsQA NJ Meetup – Test Automation Tech Talk

DevOpsQA NJ just held our August Meetup on Test Automation Tools and Processes with Polina Rodov and Kamlesh from Medidata, Anil Jaising from JP Morgan Chase and myself, Tanya Kravtsov, from Audible.   The venue, food and refreshments were provided by Audible at a state-of-the-art, vast office space in
Jersey City with amazing views of the  NYC skyline. The main topic of the meetup was Testing and Test Automation tools and processes and how they are used in different companies.

Takeaways from QUEST and StarEast

After fun networking over pizza, wine and beer, I kicked off with an overview of my takeaways from Quest and StarEast QA Conferences. I gave a brief summary of the session that I presented at the conference titled “The Road to DevOps” followed by highlights from the most memorable sessions on Test Automation, Metrics and Mobile Testing. I focused on the importance of identifying and addressing the bottlenecks in the product delivery cycle which include manual processes such as test data management, environment configuration and test execution.

WOW (W-What we do O – on time W- with tools) at Medidata

Next, Polina and Kamlesh from Medidata presented Tools and Frameworks used at Medidata (which provides cutting edge clinical solutions).   The unique presentation style portrayed the overall solution as a cooking pot, with tools presented as ingredients and testing as degustation. Medidata solutions are built on top of Service Oriented Architecture and utilize tools such as Jasmine and Cucumber for automation, Github, TravisCI and HipChat for CI, Medistrana and GoCD for deployment, Mist, JIRA Capture and Bug Magnet for testing, and SumoLogic and NewRelic for monitoring. Polina and Kamlesh also spent some time describing VCR, an alternative way of mocking APIs.   VCR allows to run the tests against prerecorded response, but still faces challenges if cassette files become stale due to backend service changes.   The session was followed by intense Q &A covering Code Coverage tools, PACT and minimizing cycle time.

Acceptance Test Driven Development

Our last session of the day was ATDD (Acceptance Test Driven Develoment) with Anil Jaising, Executive Director at JP Morgan Chase. Anil set the background by talking about history of Lean which came from manufacturing, specifically Toyota factory. It is very important f
or Testers and to be able to pull the chord whenever they find a critical defect. Anil suggested reading the IEEE paper by Robert Martin and Grigori Melnik, the primary point of which is that we need to start thinking about testing while requirements are still developed and avoid assuming that testing is testing.   He then gave a great definition of ATDD – from user stories to business value, ATDD is a team sport as opposed to test driven development which is more individual.

At the end of the sessions, we had some great conversations with group members who stayed behind.   Looking forward to our next session. To learn more about exciting opportunities at Audible visit http://about.audible.com/

August 20, 2016
The ‘Continuous’ Illusion

Continuous delivery is an illusion. That doesn’t make it insubstantial—illusions can have a tremendous impact. Like the first motion picture: the locomotive felt so real to first audiences that they fell from their chairs, screaming. But of course, the locomotive was an illusion, merely rushing past on screen.

A good magician, like a good inventor, will tell you that the secret to a powerful illusion is hard work on top of a good idea. Indeed, good magicians and inspired technologists have a lot in common. Each is always on the hunt for the next big insight, the transformative angle, building on existing industry techniques to create something even greater. But each knows the secret to truly wowing their audience, whether for a one-night-only show, or an ongoing relationship. What really gets the job done is the vast, complicated, hard-won machinery ticking away behind the scenes.

Today’s No. 1 magic trick for making great applications, a directive now pervasive across every industry, is continuous delivery. Right now, continuous delivery provides that wow. It takes app updates from a customer stumbling block to a seamless benefit. It takes feature delivery from “maybe in a year” to “is tomorrow soon enough?” It allows Dev teams to wave their magic wands and transform the impossible into the effortless.

And, of course, continuous delivery is an illusion. Remember that movie locomotive, leaping out of the screen? Every instant of its motion was its own little frame of celluloid. And the true magic was the process that bound each and every frame together, delivering them onto the screen with perfect timing.

Continuous delivery is the very same sort of illusion. The steady stream of new features, new capabilities and bug fixes must be created one by one by a vast army of talented developers—not unlike the thousands of talented stencilers who first brought color to motion pictures. And the true heart of the process is the automated systems that support this army, and which ensure the coordinated delivery of its products to the end user.

The heart of continuous delivery is the machinery of automated testing and deployment. The approach behind continuous delivery encourages teams to produce software in short cycles, ensuring that the software can be reliably released at any time. For these releases to be reliable, continuous testing is integrated into the software delivery pipeline, for immediate feedback on any bugs an update would generate. Then, when a proposed change has passed the automated tests, it is deployed to production automatically. This software development life cycle (SDLC) allows companies to achieve the business goal of truly responsive development. The team sees an opportunity, confirms the solution and then delivers that solution to customers extremely rapidly.

Each stage of continuous delivery—develop, test, deploy—has its own technical challenges, but the testing step in particular poses a unique bottleneck.

In the development stage, developers primarily require code. Code is relatively lightweight, and can be managed through industry-standard tools such as Github. Developers need their laptops, but in general code management poses a limited operations support challenge.

The deployment stage can be more challenging for ops. The newest version of the code must be deployed to a wide population of final platforms, which may vary in hardware, software, location, permissions and so on. However, the target systems are at least those in ordinary use. The server the new page runs on or the smartphone that installs the application update are pre-existing systems, and they are managed by their users. So Ops teams are not required to support specialized hardware to deploy. Furthermore, in the past few years a number of innovative technologies, such as Ansible and Docker, have emerged to simplify this process further.

Testing, on the other hand, combines the challenges of both of these stages. Continuous testing requires dedicated resources for developers and QA. But these resources are full-scale application environments, including not just code but data, operating systems and even appropriate hardware. And because best practices encourage developers to write new tests for all new code, the amount of regression testing required for every code change can only grow. That means that testing must grow ever more parallel, or suffer increasing delays—and any delay in testing can undermine overall continuous delivery.

To preserve the Continuous Illusion, a development team needs on-demand access to scalable testing environments, including their data. Data in particular is often overlooked in test environment creation, but it must be delivered to every test instance. That means test data management (TDM) is an inescapable part of any successful continuous strategy.

TDM is relatively self-explanatory: it refers to the process of creating, managing and delivering production (or production-like) data to non-production data environments. Every organization does some TDM, though many do not consider it a separate practice. But to achieve the sort of seamless efficiency that can support the Continuous Illusion, teams must adopt a modern TDM practice.

Legacy TDM relies on multi-stage, manual delivery chains. Developers and testers file tickets, and then they wait. Storage admins, DBAs, backup admins, security personnel and other staff each have to spend a huge portion of their time servicing the ever-growing stream of tickets, handing off environment creation from one function to another. Meanwhile, developers and testers are left in the lurch, and continuous delivery grinds to a halt.

Once, this process might have been unfortunate but necessary, imposed by technological restrictions. But modern solutions can tackle these challenges and overcome them. Too often, teams with poor TDM have simply neglected to implement the solutions that they need.

The most effective TDM practices will use a data management solution on par with other continuous delivery pipeline tools. The TDM solution will be able to stand up data environments in minutes and at massively parallel scale. It will rely on production data and automatic security, rather than synthetic data generation or data subsets (both of which requires substantial upfront planning and cannot hit the full suite of edge cases). It will integrate readily with existing continuous delivery workflow tools, allowing data to be automatically provided whenever a test job runs.

There are some solutions in the market today that meet these criteria. Adopting one can represent your opportunity to tighten test cycles and deliver code faster—regardless of how fast your software timelines are today.

To deliver the Continuous Illusion, every component of the continuous delivery pipeline must be operating at peak efficiency. A powerful TDM solution is a key requirement for allowing your customers to sit back, relax, and enjoy the magic of your product.

About the Author / Louis Evans

Louis Evans is a DevOps evangelist with Delphix. He’s excited about the potential for superior TDM to enhance overall software quality. He is a huge Alan Turing fan, and partial to a well-placed semicolon.

August 15, 2016
DevOps: Commoditizing Infrastructure

Hardware vendors face a quandary. How do you distinguish yourself as superior from your competitors if you are unable to continue to grow a distinct architecture as your competitive advantage? But this dilemma does not begin and end with only hardware vendors; operating system vendors and virtual machine vendors are on the same horizon. In an effort to “uncomplicate” hardware idiosyncrasies, our industry evolves efforts from open standards to least-common-denominator functionality for a “common” language or set of features, and over time technologies such as containers emerge. If I can abstract my application from the “uniqueness” of hardware vendors, from the complexity of configuration settings, and make only assumptions about basics such as CPU, disk, memory, etc., then I am free to focus on the application and allow my “hardware mitigation technology” to perform its function.

The Relationship Between App and Storage

But to accomplish this feat, I must agree to a basic set of features or functions I intend to call, such as making “x” amount of machine memory available to my application. Whether disk or RAM, I am looking to define what my application needs and then expect the container (or hardware mitigation/abstraction manager of my choice) to manage the actual allocation. Then the quandary begins. Asking for 1TB of disk space as an example is generic enough. But does my request include the ability to specify I am looking only for disks that spin at more than 7,500 revolutions per minute, or will I be saddled with 5,000 rpm disks? There is a clear implication in speed for my app.

Ultimately, as I sit and tune my code because my app has become “slow,” it could be something as innocuous as having its actual execution occurring in a cloud provider who uses the slower disk revolutions as opposed to the faster ones. Worse, it simply could be that the same cloud provider has a batch of the slower ones in one of its data centers where I wind up executing (only on a temporary basis), and I am never fully able to figure out why my app got “slow” for a while. That kind of random bad ju-ju requires the execution of a chicken to fix (I prefer Popeye’s, myself).

Different storage vendors might have additional peculiarities that make one or the other superior in speed, reliability or performance over their respective competitors. But the container’s main functionality is to provide “disk” to my app. Specifications of specific disk vendors or providers are the problem of the cloud provider, not the app developer, in theory. But hardware refresh can be pricey, and to make a claim that “all our disks” have rpm specifications above 10,000 will be meaningful only for about 10 minutes when the 12,000 rpm models come out. And of course, there are many more architectural designs in the engineering that are intended to boost speed and reliability. If these are not to be ignored, a decision must be made as to “who” has the responsibility to manage their usage (usually the cloud provider), and “who” will complain when things get slow (usually you the customer).

And what happens when “breakthroughs” occur in a specific technology? Solid-state hard drives, for example, are not exactly new now, but they once were. When you move your app from operating on a high-speed hard drive that spins to a chip-based hard drive the operates at the speed of the RAM it is based on, the difference in speed is an order of magnitude better. The inevitable question emerges, Will my container have the ability to specify RAM-based disks instead of traditional spinning disks, as part of the disk commodity it is providing to my app?

Since the difference in speed is so great, you would think it would come up with a way. You would equally expect that providing offline storage such as tape (from the age of the dinosaur) would have its own designation in my container structure (not be equated with offline spinning drives due to the speed differential). When you go to make a backup or restore, the difference between tape and hard drive becomes as apparent as the difference between solid-state and spinning drives on app execution. So when a platform evolves, my container mitigation must evolve with it. But to offer the uniqueness of vendor superiority requires my programmers to learn who to request (or what level of hardware specs they must know), which is the whole point of wanting a commodity in the first place: I don’t have to know.

Impact of Commodity on Other Components

This same dilemma can be exhibited in CPU vendors. Intel touts its superiority, AMD would disagree, and there may always be an upstart who wants to rival the current leader in any space. Lest we think Intel-based computing is the undisputed champion, quantum computing may upset that assumption (unless, of course, Intel corners that market as well). The point here is simply to ask: how many features, how many specifications, how much do we the programming team need to know about our platform provider to understand why and where our application runs better and worse? In the days before container-style ideology, hardware vendors touted their differences and their uniqueness as the reason they were better. Now what?

Then comes our virtual machine providers, and eventually our operating systems. If I want to truly abstract from my hardware layer, I must not call an OS-unique command that only Linux or only Windows can do. I must stick to functionality that both OS instances can reasonably understand and accommodate. While this least common denominator is great for extending where my app can run (on multiple types of hardware, in multiple types of OS installations), it has the same effect on uniqueness and potentially on competitive advantage. If the intent is to come up with a one-world OS, I am generally in favor, but our journey there is sure to have certain casualties in innovation that a single OS is suited for and its competitors are not.

The inevitable question becomes, How far down, or how far up, do we want to form commodity provisioning of computing capabilities? One could argue that the simplicity of the iPhone platform is a user-centric expression of commodity provisioning at the top of the stack. Drag-and-drop programming languages are yet another expression of bucketing computing capabilities into more simplistic expressions (that more people can understand and, therefore, utilize). Commonality and simplicity do increase audience. In many segments of our industry, this is the nirvana of sales to the public.

Impacts to the Bottom Line

DevOps focuses on enabling innovation. At its core DevOps is focused on software innovation. Let’s face it—changes to hardware platforms are generally less frequent. If those hardware platforms are married to a particular app (not serving a large number of apps on the same box, same virtual instance, etc.) then at least provisioning can be tuned one to one with the app. But with the desire to turn infrastructure into a generic commodity that application programmers don’t have to think about comes the fact that they stop thinking about them. Over time they will understand less and less about why things become slow, or work better, as the knowledge of uniqueness goes away.

Depending on your market focus, ubiquity is either your friend (as it touches consumers) or your enemy (if you are the hardware vendor attempting to maintain relevance over your competitors in an increasingly commoditized market). The questions I raise in this article are designed to focus on how to keep our trend toward hardware mitigation from stifling innovation in the very hardware platforms we depend upon. In any case, I welcome your thoughts on the topic.

To continue the conversation, feel free to contact me.

August 10, 2016
Integrating ARA into traditional RM

Application Release Automation (ARA) is terminology that includes several DevOps constructs from a singular, application-centric point of view. In essence, ARA implies the automation of a single app from build through deployment (even if that deployment is to production). Of course, embedded or implied in that automation terminology is build automation or construction; deployment automation, which may include provisioning; and governance automation, which includes testing and approvals.

If your company’s portfolio of applications is small, or if the applications are fully independent from each other, using ARA to describe your DevOps continuum is more or less a complete vernacular. However, if your application portfolio contains interdependencies; you need a coordinated/scheduled release event of multiple applications into production in a particular sequence or timing;or you are required to meet blanket compliance governance requirements that exceed the scope of a single app; you will need to learn how to integrate an ARA-based DevOps strategy with the more traditional discipline of Release Management (RM).

Clarifying ARA

One of the nicer implications of ARA thinking is that it places the value—or, accentuates the value—where it should be; that is, on the application. Automating infrastructure is great, but if no application rides on top of it, the infrastructure is essentially nothing more than unexplored capacity. Most CIOs are not fond of having excess computing capacity sitting idle. So the idea of ARA sets the focus of the automation where it should be: moving an “application” from construction to production release.

ARA is also development methodology-neutral, which means the build, deployment, and releasing automation will look the same regardless of whether the code was prepared using Agile or Waterfall. Mind you, Agile tends to drive the need for speedier delivery systems, but even legacy development teams can find value in consistent delivery system automation once a build is ready to begin. It is, after all, consistency that reduces support costs by reducing errors and creating predictability, thereby improving quality. It also is consistency within the automation that permits scale, whereas variability in the automation deters scale.

ARA also allows for depth within the stack. If code reigns at the top of the stack, then the operating system of the server reigns at the bottom (perhaps networking in the wine cellar and facilities management in the root cellar). ARA incorporates the notion of automating dependent layers of technology under the code such as database services, middleware or messaging services, perhaps the provisioning of a virtual machine, even down as far as configuring the OS, if needed.

The practicality of moving down the stack however, is that the scope of change will continue to have a broader impact than a single app. I usually do not install a single instance of Oracle for each application I create; more often, I have several applications relying upon a single instance of my Oracle setup. So the actual changes I automate as I move down the stack tend to be more of a configuration adjustment. It is more likely my particular app needs a config change to accommodate it rather than a completely new install of a supporting technology component underneath my app. It is also worth noting that the frequency of change as you move down the stack slows down considerably compared with app code riding on it, and the cost or impact of change increases greatly, as you begin to impact a wider number of things riding on it.

Deconstructing ARA

ARA automation, then, includes all the traditional DevOps disciplines, but limiting them to the viewpoint of a single application as follows:

Build Automation: This includes version-controlling each particular build. It also includes packaging all the component technologies to create a deployable package used for deployments throughout the software development life cycle (SDLC). The golden rule here is create a “build once, deploy many” philosophy in your build automation. Changes to code, included configuration files or dependent technology components down the stack must occur in the DEV class environment. Once the deployable package is completed to satisfaction in DEV, it will move unaltered through any other layers of testing environments before it gets to production for ultimate release.

Deployment Automation: Since each deployable package is tightly version-controlled during its construction in build, there is now an ability to roll forward (normal deployment) or roll backward (revert to a prior version) in the event of failed testing in a given SDLC environment. Some of the common environment classes include integration testing, performance testing, user acceptance testing and pre-production quality assurance (QA) testing. Smart DevOps deployment tools utilize the ability to deploy (or overwrite) only the components that are out of sync with the versions listed within the deployable package. This smart deployment capability reduces time and ensures consistency with what “should” be there.

Provisioning Automation: This is an offshoot of deployment automation. For those using virtual machine or cloud technologies, the deployment automation can include the ability to provision or set up an environment of any class on the fly. When testing is completed, the same environment can be decommissioned to not waste capacity or budgets. Provisioning bare-metal servers is precluded by human interactions, although what should go on them also can be fully automated once engineering is done with the physical work.

Governance Automation: Even though substantial testing can and should occur in development, a significant amount of it must occur in other testing-class environments later in the SDLC progression. It is, therefore, a good idea to have the ability to isolate testing automation outside of both build and deployment templates or scripts, still making it callable or executable from an overall workflow manager (often included in the release orchestration or release automation tooling within DevOps). This gives the flexibility to run a series of tests without having to rebuild the application or redeploy the application just to kick it off. In addition to testing evaluation, there may be other process regulations driven by a governance team or compliance entity within the organization. All of these flags must be met and passed before final promotion to production.

Integrating ARA with RM

Dependency is the hobgoblin of simplicity. Applications that are fully independent of each other can utilize an ARA strategy and deploy into production as often as they want. No one else is impacted by this. But if a given application has dependencies upstream—I get my data from, or transact with, someone else—or downstream—my data is provided to someone else—then my path to production must include testing an application different from my own. Determining where the responsibility for failure lies now requires policy instead of just common sense. Synchronizing in-state changes where two dependent applications are now undergoing change at the same time requires policy to govern as well as more sophisticated SDLC testing environments to manage.

When a wider view than just a single application is required, the traditional DevOps discipline of RM or its synonym release orchestration (RO) becomes the solution. RM allows for the sequencing of applications going into production to occur as needed. It allows for testing to occur when a dependency spiderweb is kicked off. RM allows for corporate governance to be applied universally to all applications attempting to enter production, instead of having to build it over and over again in each deployment script or template. RM can produce sophisticated logs designed to satisfy outside regulators for compliance history automation. These kinds of tools become invaluable when there must be a wider view of deployments to production beyond a single app at a time or in a small portfolio.

Given this reality, the key to integrating ARA with RM is to ensure automation is constructed independently by discipline, and can be executed by discipline as needed. ARA encompasses or groups four disciplines together: build, deploy, provision and governance. RM uses the ARA package and extends it with other applications to create a needed sequence, or kicks off selective smaller units such as a testing package when needed to meet certain outside app change conditions based on dependencies. RM can selectively execute the automation ARA creates as it needs to to accommodate larger goals, while ARA continues to focus on keeping its own app as efficient as possible.

Impacts to the Bottom Line

There is no one right answer. There is only the perspective you hold within the DevOps continuum. Knowing that both ARA and RM are ultimately dependent on traditional DevOps disciplines offers flexibility on “how” you think about delivering innovation. In my opinion, ARA is simply the next evolution of automating build and deploy. It is a logical next step.

There are other disciplines within DevOps, however, that might benefit from a wider view of the workflow required to move innovation to production. Take, for example, the integration of monitoring automation in the process, capacity on demand provisioning, security oversight and configuration management database (CMDB) integration—a wider view of how and where these services are integrated with DevOps might reduce the labor required to create them for each app, reinventing the wheel—or worrying the wheel has not been updated properly in a while. At its heart, DevOps attempts to create singular automations to perform tasks, increasing speed, reducing risks and improving quality.

There is no immediate financial impact of this other than to pursue a strategy of single-app focus in an environment where a wider view would have been more ideal. Calculating the cost of that error is akin to picking six lotto numbers from the sky: It is always hard to know what might have been; at that point all one can do is wish.

To continue the conversation, feel free to contact me.

August 4, 2016