devopsy.kinsta.cloud

Category: Blogs

Murphy’s DevOps: 7 Habits of Rugged DevOps

Attendees of Rugged DevOps Connect at RSA packed Moscone West’s second floor Feb. 29 to immerse themselves in security. Among the presenters of the daylong event was Forrester analyst Amy DeMartine, who provided a tutorial on how security, developers and operations folks can work together better in the new DevOps world.

Through a slide show, DeMartine laid out her Seven Habits of Rugged DevOps, three of which are discussed in this column:

1: Increase Trust and Transparency Between Dev, Sec and Ops

We live in a world where technology’s goal increasingly is to automate everything, with less reliance on humans—at least, that’s what many marketers would have us believe. However, DeMartine emphasized that Security and Risk groups, if they are going to play a key role in DevOps, need to speak to the practice’s true owners—App Dev and Infrastructure and Ops—in a language they can understand. Here, it’s not automation that matters most. It’s communication—among actual human beings—that’s paramount.

She emphasized a real disconnect between these groups, with AppDev being perceived as the department of “Anything Goes,” Infrastructure and Ops thought of as the division of “No,” and Security and Risk seen as the “Persistent Nagging” sector.

Imagine for a minute, a family unit comprised of Mom, Dad and a highly motivated teenager. The teenager is fast, full of energy and always pulling the trigger; Mom is pragmatic, a disciplinarian always having to tug at the teenager’s reins; and Dad is constantly frustrated that no one listens to his determined warnings. The family’s living room is the scene of many “heated” debates, where each side scores its share of wins but there are really no common agreements that satisfy everyone’s needs. When the teenager wins, it’s because Mom simply doesn’t have the energy to resist any further. When Mom wins, the teenager turns on the iTunes, plugs in the headphones, withdraws and possibly even starts searching for a new family. Dad’s victories are often followed by a series of door slams and the silent treatment.

Now imagine if the three sides found a common language? One that not only got each to see the others’ viewpoints but also developed a system where the teenager was set free to operate within defined boundaries, and where he or she even wore enough safety gear to keep risk to a minimum. We’d have a much more harmonious place, one where the teenager continued to grow without risking the family ship.

Interestingly enough, there really isn’t any technology available that addresses this very real human communication challenge. No mater how counter intuitive it may seem, an effective Rugged DevOps environment is highly dependent on human-to-human relations.

3: Discard Detailed Security Road Maps in Favor of Incremental Improvements

The U.S. Office of Personnel Management (OPM), Sony, Hollywood Presbyterian, Anthem—each of these organizations experienced a security breach of massive scale. If these breaches all shared a word in common it would be PANIC—or at least headlines would have us believe this. The OPM breach was the ultimate example, wherein almost as soon as the alarm bell rang the government announced an elaborate plan followed by the big “Sprint.” Although, it might have been wiser just to beef up phishing defenses (smiley face).*

It turns out that a detailed long-term plan may not be the answer to data breaches, ransoms and system shut-downs.

DeMartine said outright that a comprehensive road map for addressing threats should be discarded in favor of a vision for real-time measurement and incremental improvements. Below is the “circle of life in security and DevOps,” as envisioned by DeMartine:

7: Test Preparedness with Security Games

Who doesn’t like competition? Not many. I think security folks and hackers are probably some of the most competitive people around; visit a Black Hat conference and watch as attendees intently capture flags, break out of handcuffs and hack signs.

With that in mind, it makes sense that this is one of the seven habits. There’s little doubt that simulations will do more to prepare people for real-life situations—after all, practice makes perfect.

Communication is a Rugged DevOps Essential Tool

Increasing trust between Dev, Sec and Ops mandates effective and, likely, improved communication. Perhaps one of the biggest takeaways is that, for all the technological bells and whistles the industry throws at combating cybercrime, in the end an age-old skill—communication—is at the foundation of the most-effective security.

*Phishing is assumed the most likely cause of the OPM breach; it is important to disclose that one of my clients, PhishMe, is a phishing defense solutions provider.

March 8, 2016
A Technical Journalist’s Lament

— ROELBOB

March 8, 2016
Software Complexity Driving DevOps Automation

The drivers behind increasing software complexity also are the drivers behind expansive DevOps automation. As the saying goes, “Software is eating the world.” DevOps is gobbling up software, too, engulfing it in practices that are necessary to make it thrive under the pressure of the current environment.

That environment and those drivers are of keen interest to many. DevOps authority Andy Vitus, partner at Scale Venture Partners, has identified four drivers in particular that are creating the software complexity pushing DevOps automation.

Multiple Distributed Software Locations

According to Vitus, by moving software from one server to multiserver to multiple data center deployments, the industry has made software more elaborate. Applications now execute in a distributed manner and in parallel using multithreading inside multiple processor cores; among many servers running multicore processors; and in and between geographically dispersed data centers. This makes software more powerful and reliable, though it demands more DevOps automation to produce it.

Simply incorporating the ability for software to run in parallel using multithreading adds intricacies to the development process. To keep both the workload and the number of man hours needed during development manageable, automate as much of the additional coding duties and testing as possible.

“Running applications across servers and data centers requires that you ensure consistency in the data layer,” which will then need new databases to handle the load, says Vitus. Depending on the database, the developer must rewrite the software to be able to use it.

The entanglements don’t end there.

With mobile applications, mobile devices and back-end servers both store data. “You have to be a lot more careful about knowing what version of the data is the true one, unless you’ve got a synchronization layer built in,” says Vitus. Either way, programming annoyances can be introduced.

Getting Lost in Translation

Expanding development from a single programming language to multiple languages, even for the same project, has made software more convoluted. In legacy development shops, programmers typically used one language for everything regardless of whether it was the best language for their creations. These days teams are applying choice languages such as C++, Python and Go to the tasks to which they are best-suited. “This increases the complexity of security and patch management as you use six, seven or eight different languages all with their own vulnerabilities,” says Vitus.

While adopting several languages certainly adds to patch management and security headaches, understanding how each language can and must be compatible with the others for them to work together is an entirely new challenge. “The ways that they handle variables and time stamps changes over time, and you have to ensure that different services can still talk to each other,” says Vitus.

Stacks of Stacks

Moving from a simple three-tier stack to a distributed microservices architecture catches software development in another snag, Vitus says. The database layer has grown to include in-memory and other database types. As development breaks the application layer into microservices, each potentially with its own data store, the two layers combine to create this new stack.

Millions of Endpoints

Finally, moving from deploying servers to deploying servers, mobile and Internet of Things (IoT) takes software down new paths of mostly uncharted territory, says Vitus. “Today you deploy software changes to a back-end server that must talk to a mobile client using iOS or Android or to IoT devices such as dishwashers and light bulbs in smart homes,” he says. “You could be deploying code to millions of devices.” This certainly affects the construction of the code for use on so many different platforms.

Conclusion

These drivers help create a magnified need for DevOps automation. Considering the costs of hiring engineers and technicians to do the same development work manually, the world will be watching companies that are pushing for increased automation to displace those labor costs. “To the extent that you can offload some of that to regression tests, you end up saving, and the cost of building a new software product goes down,” says Vitus.

March 7, 2016
Remote Deposit Capture: Mobile Testing for Your Convenience

“How much of human life is lost in waiting!” penned Ralph Waldo Emerson.

Any chore that can be digitized is surely welcome. How about cashing a check?

Cashing a check takes time. You have to go to the bank, wait on line for the next available teller or ATM. If you go to an ATM at night, you have to worry about who is lurking in the shadows outside.

Is this really necessary?

More than 7,000 banks have resolved the issue, saving us all time and hassle. You can now take a check, put it on a table, click a picture with your smartphone, and it will appear in your account as a deposit.

Using remote deposit capture, mobile applications now save you a trip to the bank. But accomplishing this feat was no easy task.

Mobile Testing for Mobile Banking is Serious Business

We are dealing with your hard-earned money. All it takes is one small error, and someone has to spend three hours arguing with a bank manager why their savings is a third of what it was last week.

Then they go on social media to tell their friends how Bank A took all their money. Then the clients of Bank A decide they would be better off moving their money to Bank B. For the banking industry, mobile testing can mean the difference between offering customers a quality-of-life upgrade and losing business.

For both the customers and the financial institutions the mobile testing process has to be perfect. To do that, we must hurdle three significant challenges:

Challenge #1: Achieve mobile test automation

The test coverage for a bank is a lot larger than for most applications. It can’t be achieved without an automated process coming from a cloud-based mobile lab consisting of hundreds of devices with their own versions, and mobile operating systems running tests all at the same time.

The challenge is that testing for an item external to the device—the check itself—eliminates the chance for automation. How do we install a robot to place a check on a table, pull out a mobile device from where it sits in the lab, and place it just the right distance from the check before taking a picture?

The solution was to create a repository of more than 100 of the most commonly used checks. Each check has its own size, color and format. A tester selects a particular check from the repository, and uploads the image file of the check to a phone’s camera app, “fooling” it into thinking it is looking right at the check. Using the zoom function on the phone, the tester has the option to simulate the distance of the check from the phone, determining how far the phone can be from the check to record accurate data.

Challenge #2: Perform mobile testing on different devices and versions to make sure each picture is of sufficient quality to read the data correctly.

Some phone cameras work better than others. Some versions of smartphones work better than others. It is essential that the mobile testing tool the banks use is able to work on all operating systems, devices and versions; otherwise, the banks would have to buy multiple testing tools and hire additional staff to operate each tool. And anything that increases complexity also increases the possibility for error.

Challenge #3: Verify that the data being read is what appears on the check.

Once the picture is taken, the mobile app retrieves the following information: the date, who is writing the check, who the check is being written out to, the check number, the bank and bank account of the payer and, most importantly, the amount. There is no room for error. The best way to test is to take hundreds of checks, all with their information stored in a database, and through numerous combinations of smartphones, versions and mobile operating systems take pictures of the checks, recording what it determines to be the information each picture contains.

The phone’s recorded data would be checked with the database of the “real answers.” The mobile testing process continues until the apps are perfect in every detail from every execution.

Once these challenges were met, the arduous chore of depositing a check became a convenience that you can take to the bank–or not.

March 7, 2016
Microsoft Buys Xamarin

— ROELBOB

March 7, 2016
DevOps and the Near-Decapitation of Variability

Not all automation scales. One of the chief impediments of delivering an enterprise-class DevOps service is the decentralization of tooling choices—or worse, the methodologies within the tool for delivering CI/CD. If the methods used to deliver automation are as varied as the last engineer who created them, the spaghetti that ensues will not scale much farther than previous methods did.

The trick to scalability is not so much tooling-dependent as it is approach-dependent. Templates or frameworks (or any word you would use) to describe a relatively preformatted approach to automating a particular kind of technology component are the key to achieving scale.

Ideally, your highest-skilled engineers create the basic template or framework that other, lesser-skilled engineers will use to create the actual automation for each system as they move forward. Using a single way to automate a particular kind of technology enables you to manage large numbers of systems utilizing that technology from a single point of administration. For example, a standardized template used to create the automation to deploy a MySQL-type database could be altered over time to include new testing requirements detected from results found later in the QA process.

Once the new testing is incorporated into the standardized template, the resulting automation is fixed once for every instance of that kind of database in every system that uses it. Otherwise, you’ll need to remember to update every script for every instance of a system that uses this kind of database. Those updates likely will be implemented differently across the applications by different engineers, which may negatively impact testing results (thus defeating the purpose of testing consistency, and actually generating more unnecessary work to locate the variations).

Templates or frameworks (or pick the synonym that suits you) can reduce variability in “how” the automation continuously delivers and integrates. Once variability is killed (or nearly killed), predictability can emerge. Predictability throughout the entire SDLC lowers overall costs, as finding and fixing bugs becomes more systemic. What’s more, there are fewer bugs, thanks to battle-tested processes that produce consistent behavior. If the tooling is sophisticated enough to offer visibility, the costs of quality are reduced even further, and the speed to deliver is radically increased.

DevOps promises that combination of reduced failure rates with increased speed of delivery. However, embracing unstructured variability in creating the automation can kill it. The Ops side of the DevOps equation is all about doing things systemically and predictably, as that is the only way to maintain stability while changes whiz through their world. Getting Ops to buy into DevOps is much easier when they have complete visibility into a predictable, repeatable, process for delivering change. Avoiding that 3 a.m. phone call is something everyone has in common, and reducing variability by focusing automation through the lens of templating or frameworks is an excellent way to achieve it.

March 4, 2016
Building a Foundation for Big Room Planning
When people think of agile, the first thing that comes to mind is a software development team. But agile is much more than that now, as every DevOps practitioner knows. Nearly a decade and a half of practice has shown us that high-performing agile software development teams are critical, but not sufficient on their own. When they operate in a silo, software development teams can easily become disconnected from the business and from other teams. To truly achieve the higher business goals, those development teams need alignment, or else they risk delivering the wrong things or the right things but at the wrong time. That is where Big Room Planning comes in.

Big Room Planning is the secret sauce for effective agile planning and execution because it brings together all the people in an organization responsible for delivering software value to market—from development and test to program management and the business—to plan collaboratively. When I say bring them together, I don’t mean that figuratively: The key is to bring them together quite literally, assembling all of the stakeholders together in the same room in real time. That can feel quite shocking because it represents such a significant commitment by the company, but that is the point. Making agile work requires a commitment across every level of a company.

Before the Big Room

So do you just have everyone roll into a large space somewhere in a hotel’s conference area and hit a home run? Clearly, no. Setting up an effective agile delivery group takes time, and is far more complicated than spinning up agile development teams because of all the roles involved and how each team and individual will approach their part of the work. Most importantly, establishing the leadership changes required to truly change the way an organization works must also take place. Bottom line: Establishing an agile mindset throughout the organization is critical to have a foundation for success.

There are three distinct phases of preparation required when standing up an agile delivery group (Agile Release Train, in SAFe parlance), which we call:
- The Ready phase
- The Sync phase
- Rhe Go phase
Each are vitally important, and without success in the earlier phases, your delivery will get bogged down in the latter stages.

Get Ready

The Ready phase is probably the most critical, and it establishes the right leadership, organizational mindset, and right group of people required for agile value delivery. Three key steps include:
- Enterprise value stream mapping
- Agile organization design
- Leadership development
Without these key steps, supported by leadership at every level, you simply cannot instill the cultural and mindset changes required to stick.

Get Synched

In the Sync phase, agile practices are instilled first. Agile practices specific to each role requires training and coaching to set in, and we advocate for a four-to-12-week period to ensure everyone understands their role, the ceremonies they will use to ensure collaboration and what artifacts they will use to plan and execute their part of the overall work.

Once everyone in the delivery group gets it, it’s time to set up a first Big Room Planning event. This typically takes one month of preparation—without which your event is likely to fail.

Preparing for Big Room Planning should focus on the inputs for the event. First among these is the business vision and context—the domain of executives, line of business leaders, etc. This is essentially the “why” behind what the teams will work on. Key initiatives for the delivery group, with clear outcomes defined (often supported by a lightweight business case), must be communicated effectively to inspire the group. It takes time to put this together.

Next, develop a feature backlog that represents what the delivery group will be planning out during Big Room Planning. Doing this well is a more rigorous process than meets the eye. Features are the crucial ingredient because they represent the value being built to both the business and the customer, and they represent the key deliverables teams will produce.

Product managers typically own this step, and if they are not well-prepared, they run the risk of either asking for too much or asking for the wrong things that won’t have the desired impact on business results. At CA, our product managers go through several iterations of backlog preparation involving stakeholders throughout to company to make sure the priorities are clear.

Even with a well-formed and prioritized backlog, effective Big Room Planning requires an understanding of the capacity of the organization to build what’s needed. This requires good estimation and a good understanding of the each team’s velocity. Estimation and a team’s velocity are big parts of what we learn during the Sync phase: Over a few iterations, the amount of work (a.k.a. story points) a team can deliver outcomes and provides an understanding of roughly the right capacity needed for effective planning. Doing this well ensures teams will be asked to plan not only the right features during Big Room Planning, but the right amount of features so that detailed planning can take place with enough time to design stories, resolve dependencies and address risks that emerge.

Then Go

The Go phase actually kicks off with the Big Room Planning event. The event itself occurs over a two-day period, which marks the starting point for agile execution. If formed well, the right people will be in the room to build value fast. With upfront preparation, the group members will understand what they will be building and why it’s important, and will be set up to create a plan that best delivers the value needed.

Planning collaboratively ensures that the plan is well-designed across the group. Most importantly, because everyone has a part in building the plan—with the opportunity to work together across teams and across roles, with the ability to challenge and even veto decisions as they form—everyone ultimately feels a strong sense of ownership and commitment to the final plan. A successful outcome almost always ensues, and the group only gets better from there are they learn to prepare, plan and execute together as a cohesive “team of teams.”

About the Author/Steve Wolfe

Steve Wolfe is Senior Principal, Product Marketing for CA Agile Solutions. He has worked for more than 20 years on software products in both hardware and software companies. His experience spans 10+ years in both software engineering management and product management, including 7+ years on agile teams and delivery groups. Steve’s passions include helping organizations learn how to build the right things fast and reliably by applying agile and DevOps practices at scale. He is a certified SAFe Program Consultant.
March 4, 2016
Designing User Experience

— ROELBOB

March 4, 2016
DevOps Connect at RSA: In Security, Choose Increments
Earlier this week the worlds of DevOps and information security collided at the DevOps Connect Rugged DevOps event, held in conjunction with RSA Conference 2016. The security-centric audience got to hear from a number of thought leaders across the DevOps and infosec worlds about ways security pros and DevOps teams can work together and establish incremental improvements in application security, as IT picks up the pace of software delivery.

Among the talks, Forrester Analyst Amy DeMartine discussed her team’s findings after studying the “best of the best” in implementing Rugged DevOps patterns. There are seven habits universal across organizations that are succeeding with this new way of developing and deploying secure applications, she notes.

“Think of it as a fitness program, and the muscles that you are building up are those that strengthen application security,” she says.

These habits included a number of obvious practices, along with a few surprises, including:
- Increase trust and transparency between Dev, Sec and Ops.
- Understand the probability and impact of specific risks.
- Use the continuous delivery pipeline to incrementally improve security practices.
- Standardize third-party software and then keep current.
- Govern with automated audit trails.
- Test preparedness with security games.
All are good tips for getting security, dev and ops to work better at delivering code both faster and more securely. But the seventh habit probably raised the most CISO eyebrows and could have the most potential to effect meaningful change and collaboration:
- Discard detailed security road maps in favor of incremental improvements.
According to her, the “best of the best” organizations in Rugged DevOps have security leaders that make it a habit to ditch detailed security road maps in favor of a establishing broad vision and achieving incremental security improvements over time.

As she related in her talk, she’s seen too many organizations develop extensive and extremely detailed multiyear road maps only to see them grow obsolete not only because of rapid changes in the IT environment, but also the constant pivots attackers make in their attack techniques.

Instead, she said, security leaders have to take the same continuous improvement approach that their peers in development and operations have made in their shift to DevOps.

“This is a mind shift change that has to happen,” she warns.

To achieve good results, security leaders should be focused on “real-time, actionable measurements” to focus on improvements that matter, such as decreasing remediation time for vulnerabilities.

“In DevOps, we call this Plan, Do, Check, Act, where you’re taking very small changes. Say you want to insert static analysis right after a developer checks in new code. So you’re going to go off and plan. The plan is to insert static analysis,” she says. “Then you’ll do—we’re going to look at tools, we’re going to create our rules. We’re going to check to make sure it works, if it improves things. And then we’re going to move on to the next thing—maybe it’s tightening our rules or maybe it’s adding another tool.”
March 3, 2016
Testing Mobile Apps? Consider a Mobile Device Cloud

Market demand for mobile app development services is expected to grow at least five times faster than internal IT organizations’ capacity to deliver by 2017, according to Gartner. As a result, enterprises must use the most efficient, effective and secure methods possible for their mobile app development processes, including mobile app testing.

Today’s users expect the highest quality user experience from their mobile applications, so testing should be a top priority for enterprise IT departments that want to keep up with these expectations. Mobile testing is more complex than legacy application testing, however, because of the myriad platforms, networks, OSes and devices on which the applications run. To ensure their mobile apps perform reliably for every user, businesses must test on key variations of this growing universe of mobile devices.

Sharing a single large pool of mobile devices across large or geographically distributed testing teams creates a number of challenges that risk bleeding IT and QA testing budgets dry. Mobile device access management has been plagued with inefficiency and risk, requiring IT teams to purchase an unreasonable number of devices to cover their enterprise’s growing needs. Inevitably, many of these devices remain unused or become lost, which can stall testing and result in idle mobile engineers.

In addition to the cost and inefficiencies of having multiple mobile devices, security has become a top concern for today’s enterprises, thanks to the rise in data breaches. To avoid complications, enterprises must hold mobile testing practices to the highest security standards. A data breach during app testing can create significant cost for an enterprise and harm its reputation. Plus, for industries that must adhere to Sarbanes-Oxley, HIPAA or other federal privacy regulations, a data breach or unauthorized user access can create a regulatory compliance issue.

The Mobile Device Cloud Option

A mobile device cloud may be the answer to the cost, security and compliance issues enterprises face in their mobile app testing environment. It can serve developers looking to test and debug applications and QA teams performing manual or automated testing, and facilitates collaboration among these groups and other stakeholders such as marketing and customer support.

A mobile device cloud offers organizations four significant benefits: security, accessibility, speed and cost. Let’s explore each in a bit more detail:

Security

On-premises, private mobile device clouds access a centrally managed pool of devices and mobile apps and enable testing within an enterprise’s corporate infrastructure and behind the firewall. This creates a highly secure environment for proprietary apps and sensitive data throughout the testing process, which is managed in-house. In addition, private, cloud-based mobile testing can improve compliance with regulatory standards by reducing the chances of a data breach or unauthorized user access.

Accessibility

Private mobile device clouds can still meet the enterprise need for global access—developers, testers and mobile engineers can securely access centrally managed mobile devices from locations anywhere across the globe without compromising corporate data security.

Speed

Technology advancements in mobile application testing platforms now allow automated, continuous testing simultaneously on multiple devices. This allows private mobile device clouds to be even more effective in increasing an enterprise’s app testing speed and quality without compromising security.

Cost

With IT budgets for QA and testing increasing 9 percent over just the last year, reaching 35 percent of enterprise IT team’s total budget in 2015 according to Capgemini’s “World Quality Report 2015,” IT teams need affordable options that still maintain efficiency, effectiveness and security. Private mobile device clouds often are much more cost-effective than off-premises, public device clouds, while also providing an agile, efficient and effective solution.

BI Intelligence predicts that mobile phone sales will surpass 2 billion in 2016, which means demand for enterprise applications will continue to increase rapidly. More enterprises already have begun using on-premises, private mobile device clouds and are seeing the benefits of this secure, agile, affordable option for their mobile application QA needs.

For those enterprises that haven’t yet considered private mobile device clouds, 2016 is a great opportunity to upgrade your mobile testing with an improved solution and to meet both employees’ and customers’ high expectations for enterprise mobile user experience.

March 3, 2016