devopsy.kinsta.cloud

Tag: devops

Dismantling Inefficiency: The DevOps Point of View

I recently caught up with Paula Thrasher, application delivery lead at CSC and DevOps pioneer in government circles. In our conversation, we discussed a number of important topics including cross-functional teams, software supply chains, paying down technical debt and her search for passionate people to hire. To succeed in managing large projects, she’s taking a different approach—one you might want to apply in your own organization.

Thrasher: I’m Paula Thrasher with CSRA and I work for a division company we call Delivery Services Organization.

Derek Weeks: What was your presentation at the Rugged DevOps conference about today?

Thrasher: We had a panel about the people, process and cultural aspect of DevOps, and the security context, as well. When you’re moving to the Ops transformations, what kinds of behaviors are both challenges and opportunities.

There was a comment at the beginning session about how any sort of transformation that relies on human nature has failed. So you can’t change how we work, but you can leverage how people are motivated to try and encourage the right behavior. I think that’s the thing that’ll resonate with the security professionals.

Link to Panel: https://www.youtube.com/watch?v=IbrY8mm9aI4&index=7&list=PLotLY1RC8HovW-XnZbItSBupjbTzFkfJw

Weeks: You’re setting up a new organization and have a lot of projects that you’re going to take on. Is it possible to achieve a lot with smaller teams? Do you need a big team to do rugged DevOps?

Thrasher: I think the main thing that we’ve been focused on is creating the team that is sort of the center of the universe. I think the most important thing is to have a team that is cross-functional. A lot of times when you think of teams … this is the development team, this is the Ops team, this is the security team … the question is, do you have all the players on the same team, or are they on different teams? I think if you find yourself in a situation where any one team has to deliver to another team, instead of actually being responsible for delivery of the actual business submission function, you’ve got some horrible web of dependency you’re going to have to untangle.

The big focus of what we’re trying to do in the new organization is build that completely cross-functional team and to enable them as a team to be placed on different projects, different missions. So instead of doing single-band rotations, where a single employee moves onto a new project, we’re actually trying to move full teams. As a team, they learn how they work best and how they can deliver. It makes a really powerful solution for actually solving problems. I’m really excited about that.

Weeks: A lot of people at the Rugged DevOps Conference are interested in the rugged DevOps phase because they’re concerned about being left behind by DevOps. Is that a real concern for security teams: getting left behind?

Thrasher: Yes, we definitely see a dichotomy in the world of how security interacts with development teams—between the traditional showing up at the end and inspecting versus really embedded security. It’s so clearly going to this idea that you can’t just inspect security. Security can’t be this checkbox at the end. You really have to build it in from a computer. You have to actually make design decisions about security. You have to make sure the developers know secure coding practices. You have to have tools that scan, check all the time, and get feedback. You can’t just add that at the end.

I think it’s a real challenge to security folks, just as it was for everyone—developers and Ops people—to change how you think to that systemwide view. You can’t just be a phase at the end. We actually have to be part of a whole equation, that whole systems thinking. I think that is so instrumental to what DevOps really is as a group.

Weeks: One of the things that you recently talked to Mark Miller about was an idea of using pictures to help with collaboration and to get everyone on the same page. Can you tell me about that practice?

Thrasher: Back to this human nature piece, I think humans are inherently really visual thinkers. I also think you’re asking people to bring cross-costuming across social teams. Well, if you’re a security person and you’re talking to an Ops person, you probably don’t speak the same language. Or if you’re a developer and you talk to a security person, again, you don’t necessarily speak the same language. But when you speak visually, you can understand each other better. You don’t bring your acronyms to the table. We’re talking about a diagram.

I also think it takes the personal element out of it a little bit. We’re talking about the system. We’re not talking about each other. The security team is not accusing the developers of being a bunch of renegades. They’re talking to them about a process and how they process it. It keeps the conversation where it needs to be, which is on making the process better and delivering better software.

Weeks: How is the growing interest in software supply chains changing the way you’re thinking about DevOps?

Thrasher: I think simplicity is an underused tool in security. Recently, one of our DevOps teams that was going in their rapid fashion had a deployment that had a vulnerability in it. They had to respond quickly, and they actually did respond quickly. But when we unpacked it, in retrospect, we realized: “How did we get here? How did we let this vulnerability get in?” We had all sorts of scanning. We had, supposedly, all the right things to have prevented that security flaw. Yet, how we got there was this Venn diagram of all these things overlapping. It was one of those security bugs that slips through the cracks.

We realized what happened is we had gone to over-engineering; we had over-engineered for performance. In doing that, we had overlooked the interaction of three components in a very particular circumstance. It was like a “every third Tuesday if you spun around three times” kind of bug. But I think that’s exactly what happens a lot. That’s super common in terms of those last defects, and there’s quite a bit that comes out of that overlap of complexity.

Fewer, more secured suppliers reduces the lift of having to upgrade updated components. It also means if you find an issue, your unpacking of that issue is a lot easier because you have a lot less things going on. All engineers need the reminder of “keep it simple.” But I think that’s a real powerful thing. Fewer, better suppliers, from a supply chain standpoint, is one of the mantras that I really appreciate because I just see it match. It really makes a difference.

Weeks: You’re working with a lot of government agencies across the board. What’s their interest in rugged DevOps practices, or just DevOps practices that you’re bringing into play?

Thrasher: I think in the government space, and it’s true for any enterprise, everything has to have security. There is not an agency that we can work with that doesn’t have privacy data, or heavily secured data, in the defense and intel agencies, or just straight-up vulnerabilities from being a target. The scale of the enterprise means there’s a lot of potential systems and a potential surface area for attack. You can think to all the high-profile attacks that have occurred in the federal government recently, and it’s made people really alarmed!

One of the things that’s very interesting is that the whole federal government did a “cyber sprint” last summer, where they did nothing but pay down technical debt, and get rid of systems and their vulnerabilities. It was wonderful! And I think every organization should do just that: spend one month a year doing nothing but eliminating old frameworks and applications. That really simple, easy thing often gets overlooked because businesses are always saying, “More, more, more, more, more.” However, we actually measured the metrics of the teams after the “cyber sprint,” and all of our teams’ velocity increased. After we let teams upgrade legacy components, all of them had an improvement, which is sort of like a “Well, duh!” But I think it’s a win-win really, that if we can allow ourselves to invest in paying down our technical debt we can get better results.

I think that in a government space, it just has to be part of the equation. So much of what we do is so instrumental to everything.

Link to interview: https://www.youtube.com/watch?v=5veGeDAaJSI

Weeks: As we started off the conversation, you have a new group. You have a lot of projects that are going to get underway. Are you hiring?

Thrasher: Absolutely.

Weeks: Who are the type of people that you’re looking for, if someone wants to get in touch with your company?

Thrasher: We are at CSRA.com, and if you go to our jobs and our careers website, you can see some of the openings we have. Actually, I’m really excited. One of the projects we did internally was building a whole new hiring model. We used big data. We built in some DevOps, and we’re taking advantage of our own technology to do that.

Like I said, what we are really trying to do is build good strong teams, and so that’s something we’re very focused on. So one of the pilots we’ll probably be doing shortly is a team-based hiring model where a team actually hires for themselves.

We’re looking for folks who are just really passionate about solving digital government. And I think that’s the main focus of us is really just being passionate about the mission we’re trying to do. We’re trying to help the government be that digital agency. Looking at the interactions we have with some of these Silicon Valley apps, how come we don’t have that interaction with the government? That’s what we want to create. So we’re looking for people who are passionate about our mission and willing and interested in solving problems. Hopefully we can build a great team.

Weeks: Excellent. Thank you.

If you loved this interview and are looking for more great stuff on Rugged DevOps, I invite you to download this awesome research paper from Amy DeMartine at Forrester, “The Seven Habits of Rugged DevOps.”

As Amy notes, “DevOps practices can only increase speed and quality up to a point without security and risk (S&R) pros’ expertise. Old application security practices hinder speedy releases, and security vulnerabilities represent defects that can leave a company open to cyberattacks. But DevOps practitioners can leap forward with both increased speed and quality by including S&R pros in DevOps feedback loops and including security practices in the automated life cycle. These new practices are called Rugged DevOps.”

August 30, 2016
DevOps: Achieving Process Without Too Much ‘Process’

While there’s no fundamental conflict between IT service management (ITSM) and DevOps, DevOps does bring new and different techniques for getting work done and interacting with the rest of the organization.

ITSM people who are used to frameworks such as ITIL and COBIT tend to regard the IT department as a “service provider” and the rest of the organization as “the customer.” But DevOps has come from web-scale organizations “where IT is the business,” and that means the prevailing view is more like the Toyota Production System mentality, wherein the aim is to move things along a production line with maximum speed and maximum quality, says Matt Hooper, product evangelist at LANDESK.

DevOps thus has already broken down some of the barriers between “IT” and “business”—practitioners realize they have to determine what the business needs, and then provide it. But Agile or Agile-like practices also are part of this approach, so there’s lots of experimentation, learning and redeployment in “lots of tiny little steps,” and, consequently, there’s not much risk of going very far wrong with any of them.

An issue is that this conflicts with ITSM’s view of life cycles, partly because the whole idea is to merge development and operations—for instance, by automatically spinning up software-defined infrastructure as needed.

What’s needed, Hooper suggests, is “service management with a digital mindset” rather than ITIL’s “human mindset.” It’s all about feedback, he says, but for ITSM and DevOps to sit comfortably together there must be a realization that feedback can come from humans or systems.

And even though the command and control mindset tried to avoid failures, systems still failed. So a better approach is to accept that “stuff breaks” and make small changes at a time, respond to issues as they arise and keep learning. In the longer term, this will lead to more durable systems, he says.

It’s a myth that you’ll be able to stop bad things happening, because people and systems aren’t perfect. DevOps, in contrast, accepts that things will go wrong but aims to respond quickly—for example, by keeping developers close to operations so they can see problems occurring, realize what mistakes they made, and put them right.

“That’s challenging to a traditional ITSM person,” Hooper says. It is important to accept that failure is an option, even though massive failure isn’t.

(Before you ask, he accepts that certain systems need to be extremely reliable. But others provide room for a somewhat experimental approach, in which case DevOps helps avoid overengineering.)

While some ITIL practitioners see value in DevOps’ continuous deployment approach, their premise is still that IT is a “service provider” rather than a “partner,” but the business side of the organization wants the latter. You don’t get home from a business trip and tell your spouse you were 99.98 percent faithful while you were away, he observes, and that sort of SLA-based thinking works against becoming a learning organization, achieving good demand management and so on.

ITIL v2 served a good purpose, but ITIL v3 arrived too late, Hooper suggests. The growing influence of the Lean movement means systems are going through rapid change cycles, and too much “process” leads to waste and lower performance. DevOps and software-defined technologies provide ways to better handle rapid change—infrastructure can be spun up and back down in seconds, providing opportunities for self-healing systems with better scalability and reliability.

Where ITIL focuses on meetings and human processes, DevOps prefers to use analytics to determine the right time to release and then whether then release was successful. “The feedback loop is an essential part of DevOps strategy,” just as it is with ITIL, but ITIL has a more human-centric idea of feedback.

“ITSMers tend to hide in the back closet” waiting for the business to tell them what to do, but DevOps takes a more active approach, he suggests.

August 29, 2016
5 DevOps Techniques to Improve Software Hygiene
Thinking about frequent food contamination outbreaks, perhaps the old adage, “You are what you eat,” should be modified to, “You are what you eat—as assembled across an unknown supply chain.” Even foods advertised as fresh and healthy can be quite the opposite. And it could be because of just a single unhygienic practice hidden within the food production life cycle.

It’s not unlike the issues facing modern software development. Now with the advent of open-source, cloud computing and component-based designs, the succulent software morsels we deliver can be fouled by a range of contaminates. And while not as a physically debilitating as a case of gastric flu, the impact on business can be extremely severe, such as major security incidents and compliance breaches—the E. coli of digital software production.

The good news is that IT organizations can take effective steps to mitigate the risks. Much like food processing, software contaminates can be introduced anywhere code and data is handled and stored—from requirements and coding to testing, release and operations. This means all stakeholders must take responsibility and be held accountable for software quality. To that end, here are five good practices DevOps practitioners can use to improve software hygiene:

Thoroughly wash test data before handling – Creating, maintaining and provisioning accurate test data is one of the most time-consuming and resource-intensive challenges in continuous delivery. It’s also fraught with business health hazards if teams fail to protect personally identifiable information or continually rely on data that’s well past its use-by date. With public cloud services providing convenient platforms for testing, ignoring these issues will only increase the risk of major compliance breaches. It’s important, therefore, to supplement data migration with fully automated methods to mask and subset sensitive data. Additionally, by continually enriching virtualized or emulated services with realistic test data, teams have a repeatable method for improving the quality and efficiency of testing.

Pay special attention to open source freshness – Open-source software has become a large contributor to the portfolios of many organizations. But just because something is “free” doesn’t negate an organization’s responsibility for due diligence, including understanding total cost of ownership and mitigating security risks. Even though open-source projects have active communities, that hasn’t stopped code pathogens lying dormant for years (for example, Heartbleed went unnoticed for 2.5 years). With open source, the onus is on development to retain code “freshness”—something that might be difficult if teams are constantly being pulled onto different projects, or if thorough risk control management isn’t a core competency.

Don’t leave software lying around for too long – Just as many foods spoil quickly, software release bottlenecks can quickly leave a sour taste for business. Like a truffle imbues a marvelous aroma, modern digital business must be flavored with a continuous stream of application updates. And, since windows of business opportunity close quickly and application lifespans are as short as days, it’s critical any deployment delays are removed. Letting software linger in the release queue also introduces other hygiene issues—for example, teams putting off documentation or code-refactoring because they know they can always get back to it later. Inevitably they never do, and when the code finally gets deployed it’s unclean or laden with defects and bugs.

Set up a “health inspection” SWAT team – In the traditional “software factory,” application performance management, security auditing and compliance checking often comes late in the software development cycle—leading to delays and conflict. DevOps, with its focus on team collaboration and continuous feedback, aims to rectify this by embedding these practices much earlier. But when resources are limited, it’s not practical to assign a security or operations professional to every agile team; it just won’t scale. What’s key, therefore, is to take a more pragmatic approach; developing teams of specialists and repeatable automated practices others can leverage to remove “bad smells” from code. This could involve:
- Regular education on web application cross-contamination issues and modern API security techniques
- Moving application performance management into pre-production to improve quality
- Auditing services to regularly check composite applications assembled from third parties and open-source software
- Automatically invoking application security testing before and after release and deployment
- Methods to automate testing as requirements are modeled and changed—immediately linked to correct data and expected results
- Scoring mechanisms to highlight security and risk problem areas—all bubbling up to a team and divisional level
Work with your suppliers – In the rush to embrace modern cloud architectures for application scale and resilience, teams often forget that even the most robust platforms can succumb to problems. Savvy teams understand this, working with their cloud partners using modern monitoring and analytical methods to quickly detect and respond to many complex issues well beyond the scope of traditional rules-based methods and slioed tools. By using a design-for-failure approach, DevOps teams understand that no platform is infallible, preferring instead techniques that quickly anticipate and contain inevitable failures to keep the digital business running.

Even the tastiest of food can be toxic, and it’s the same with software. However functionally rich our software creations appear to be, contaminates lurk everywhere within the software pipeline. Use DevOps techniques to pinpoint and eradicate “digital spoilage” quickly, and avoid making the business and customers ill.
August 25, 2016
Can Culture Detract from Success with DevOps?
Collaborative culture is considered to be an essential best practice for enterprises. DevOps culture correlates strongly with enterprises that achieve the highest level of business performance. However, is it possible that an organization can get overly fixated on organizational culture improvements? Can culture actually detract from achieving success with an enterprise DevOps transformation?

An article by Curtis Franklin Jr.,”8 ways to fail at DevOps,” indicated that there are “plenty of places in the DevOps process where an organization can run aground on rocky shoals of woe.” My blog, “7 Pillars of DevOps,” identified seven essential best practices areas that make up successful DevOps. One of the seven pillars is collaborative culture. These pillars are not silos, but rather foundations that need to be kept in balance.

I bet most of us have experienced situations in which cultural improvement missteps hindered an enterprise DevOps transformation.

Culture Missteps

I’ve listed some examples l below which I have personally seen. Have you experienced these? I would love to hear examples of your experiences and solutions you have observed, too.
1. Unrealistic expectations from leadership: Business expectations from leadership get too far ahead of the DevOps implementation reality—then teams reporting to leadership end up feeling unappreciated and experience frustration, resulting in organizational friction.
2. Ops-side transformations fall behind Dev-side transformations: The Dev team accelerates ahead of the Ops team and implements “its part” of a DevOps transformation before the Ops team has a chance to implement complementary changes. The Dev-side is disappointed that Ops has not kept up with the complementary Ops-side planned changes. Dev changes cannot be put into effect, which deepens the mistrust between Dev and Ops.
3. Dev-side transformations fall behind Ops-side transformations: This is the reverse of the second example, with similar effects.
4. QA tester roles are confusing: When new continuous test strategies are implemented, the lines of test responsibilities between Dev, QA and Ops team members became blurred. Communication between cross-functional team members become confused, resulting in misunderstandings, dissatisfaction and organizational friction, which obstructs progress.
5. Infrastructure and tools teams are siloed: Dev, QA and Ops teams maintain separate, siloed tools and infrastructure staff and do not cooperate to support end-to-end infrastructure and tools improvements. The transformation needed for a smooth DevOps pipeline stalls.
6. Rewards and measures are not consistent with DevOps goals: The organization makes changes toward a cooperative culture but fails to change the way people are rewarded and measured. Conflict results between the teams.
Solutions

Below are solutions that were applied to the situations listed above. Do you agree with these solutions, or have others to suggest?
1. Leaders were coached to be more aware that DevOps transformation is a journey of continuous improvement. Leaders then understood that there is really no “end state” to DevOps—there are specific business improvements that can be expected along the way. Each DevOps improvement initiative was refocused on realistic specific use cases and goals. With improved clarity, leaders were more realistic in their expectations for each improvement. This drove alignment between leaders and their teams, improved satisfaction and productivity.
2. Project goals and team meetings were realigned and organized. Dev and Ops sides could monitor end-to-end progress together. When situations occurred in which one team got ahead of the other, the end-to-end progress visibility helped overall group communication. Empathy across the Dev, Ops and QA teams improved. With improved communication, team members celebrated the success of each other’s improvements. Teams shared a common end-to-end goal.
3. Same as 2.
4. QA testing responsibilities changed significantly during the DevOps transformation. Separate independent test team member responsibilities changed toward QA testers being more integrated with Dev teams. New skills requirements for the QA engineers emphasized test automation more than in the past. Training for the cross-functional team helped mitigate concerns related to the new QA responsibilities. QA testers were uplifted with new skills and became more valuable and respected by their peers. Cross-functional productivity improved.
5. The tools and infrastructure teams from the separate Dev, QA and Ops teams were combined into a common tools and infrastructure team. Project goals were realigned. The improved clarity around knowing they are working toward tools and infrastructures for a common end-to-end pipeline accelerated implementation of DevOps.
6. Rewards and measures that had been in place for separate Dev, QA and Ops roles were changed to rewards and measures that encouraged end-to-end cross-functional team activities. This improved overall morale and job satisfaction and reduced organizational friction that was previously impacting progress.
Summary

From the above real-life examples, it is clear to me that culture and organizational practices must evolve in concert with progress on other DevOps pillars. And as the pace of change increases with each transformation, the cross-functional teams must be even more cohesive. Continuous acceleration of interactions between team members drives the need for tighter relationships, more end-to-end responsibilities, improved customer awareness, continuous skills training and end-goal focused leadership.

The is no need to create the “perfect culture” before starting a DevOps transformation journey. It can be counterproductive for sections of the culture to get too far ahead of itself, as indicated in the examples above. Perhaps it is best to evolve the culture carefully. Changes most important for each transformation project should be the focus. Continuously communicate goals and progress, and evolve the culture together with evolution of other pillars. What do you think? Do you agree is it possible that an organization can get overly fixated on overall organizational culture improvements?

While DevOps implemented with all seven pillars provides a strong foundation for long-term enterprise business improvements, it is important to understand DevOps is not an island. Enterprises implementing DevOps should be aware that DevOps interoperates with other IT systems and practices. Enterprises are well-advised to choose tool-agnostic IT partners that can provide solutions that best suit the needs of each unique enterprise and can integrate and evolve DevOps together with all of their IT systems.
August 23, 2016
REP SPOTLIGHT: Daysha DevOps Solutions

DevOps Institute: Tell us a little about your organization.

Daysha DevOps Solutions: Daysha DevOps Solutions is a leading IT services and solutions company focused on helping companies to deliver their IT projects, transform their capabilities and ultimately to provide value to their customers and stakeholders. Our goal is to help organizations benefit from disruptive technologies, such as cloud computing, Lean/Agile software development and DevOps. We work across a range of industries including financial services, government, aerospace and high-tech.

Daysha DevOps Solutions has partnered with the best-in-class product vendors to help organizations improve their ability to deliver software quicker, more frequently, with higher quality and less risk. Our DevOps solutions address all stages from project planning right through to deployment.

DOI: What other topics do you teach or consult on?

Daysha DevOps Solutions: Daysha bring proven expertise and consults across the wide domain of DevOps. Partnering with vendors such as Atlassian, App Dynamics, Chef and Zephyr, we offer expertise, training and tooling to support all aspects of the DevOps journey.

Our IT Project Services team consists of experienced IT consultants including project managers, technical experts and business analysts. These can be deployed to help augment your existing IT teams or to provide specific project help, such as requirements gathering, gap analysis and process improvement.

DOI: What type of interest in DevOps are you seeing in your region?

Daysha DevOps Solutions: The interest in DevOps is growing, though to be honest, a lot of organizations are wary of it. That said, the same organizations are heavily rooted in traditional waterfall environments. There is a recognition that DevOps is the way forward but also a realization that cultures need to change first.

DOI: How do you think that DevOps contributes to the business’ value stream?

Daysha DevOps Solutions: Primarily by delivering value earlier and increasing the visibility of the IT contribution to the business stream.

DOI: What topic area in DevOps excites you the most?

Daysha DevOps Solutions: How business can get more value from IT. In other words, turning an IT department into a revenue-generating tool rather than a cost.

DOI: Why should enterprises and individuals consider formal DevOps Training?

Daysha DevOps Solutions: It is the first step, and a certified first step, on the road to improving the value of IT functionality.

August 23, 2016
First-Time Quality for Software on a Blockchain Network
There is probably more trust in the world today than at any other point in history—not necessarily by choice, but because of our use of software to carry out almost every type of transaction. The use of blockchain networks brings visibility to those networks of transactions: Information about an asset that was once held by separate owners is changing to the lifetime history of an asset or transaction and is shared between all stakeholders.

For software developers, this means ensuring the integrity of a blockchain network is paramount. To that end, three key interesting aspects regarding continuous testing are emerging:
- New approaches to business networks require the ability to test supply-chain business processes and transactions.
- Reliance on shared technology ecosystems requires the ability to test APIs from multiple perspectives and to simulate the behavior of consensus.
- Increased dependency of APIs requires the ability to test integrations at scale without affecting distributed ledgers in which any interaction is immutable (unchangeable) and final.
Project teams face an unprecedented demand for first-time quality while minimizing the impact of testing on the team’s delivery. To respond to this challenge, test teams are enhancing their capabilities so they can test:
- Apps that integrate with a blockchain network;
- Chaincode that manages the execution of transactions on a blockchain network; and
- Provision and scalability of a blockchain network.
In addition to these new capabilities, proven continuous testing techniques and tools can be adopted to help the team deliver quality software into their blockchain ecosystem.

Test Entire Ecosystems

Using a blockchain network means an organization is redefining the structure of its business networks and processes, connecting distributed supply chains with a single transaction backbone. Projects need testers to minimize the risk of business disruption by validating every business process works across all stakeholders on the blockchain network.

Test State Early—Shift Left

In a blockchain ecosystem, the new paradigm is state-based communication, not message-based communication. The behavior of the state of an asset is influenced constantly by market, internal and regulatory changes, so projects increasingly are adopting Agile and DevOps practices to respond as quickly as possible. Shifting testing to the left on an Agile project means the team can test early, and those tests can then be reused continuously. Providing early, iterative feedback on quality at the time of development means fewer problems are found late in the life cycle, where their business impact can be severe.

Establish Technical Trust

One of the great things about blockchain technology is the potential to create greater transparency throughout an ecosystem. Trust is embodied in the transaction itself, which gives an important new role for digital systems of trust. Regulators and owners of a blockchain network must ask testers to validate the codification of contracts, compliance and certifications, which are then deployed into the blockchain network and embodied within the business transactions.

Use of blockchain technology will fundamentally change business processes and enable new opportunities. Success depends on the level of trust stakeholders have in the blockchain network and the blockchain ecosystem.

However, a blockchain ecosystem is an environment shared among different organizations, each of which has a stake in—and dependence on—the overall system. This means each organization must trust the other stakeholders to fully test any software that interacts with the blockchain ecosystem in a manner that will not impact the reliability of the overall system. As such, a blockchain ecosystem that has a culture and reputation of technical trust between the dependent stakeholders is more likely to be successful. Continuous testing brings trust throughout the blockchain ecosystem.

To establish technical trust in a blockchain ecosystem, ensure these four key things:
1. Your team can test when needed, not just when possible.
2. Required technical environment works as required to support the business flows.
3. Business processes and transactions run as expected.
4. Solution scales to the expected usage.
Benefits of Continuous Testing and ‘Shift Left’

Enabling teams to shift testing to the left means projects can test when needed, not just when possible. Continuous testing means an organization can deliver capabilities faster, respond quickly to ongoing changes and realize the value of transforming applications to utilize the power of a blockchain network.

Watch the webinar replay on how to implement quality checks as early as possible, run tests as often as possible and integrate automated testing with release management to minimize the complexity of deployment.

About the Author / James Hunter

James has spent a large part of his career architecting and implementing large-scale, distributed systems in the finance, public and defense sectors. He now leads a global business segment responsible for defining technology strategy then developing and implementing solutions for both IBM and clients. Mentoring and encouraging future technical leaders is important to James. He works with students of all ages and speaks in primary schools to universities to evangelize on the benefits of technology and the positive impact software can have on our society and the environment. He has a degree in Management Systems and he is a Chartered Fellow of the British Computer Society. Connect with James on LinkedIn/Twitter
August 22, 2016
DevOpsQA NJ Meetup – Test Automation Tech Talk

DevOpsQA NJ just held our August Meetup on Test Automation Tools and Processes with Polina Rodov and Kamlesh from Medidata, Anil Jaising from JP Morgan Chase and myself, Tanya Kravtsov, from Audible.   The venue, food and refreshments were provided by Audible at a state-of-the-art, vast office space in
Jersey City with amazing views of the  NYC skyline. The main topic of the meetup was Testing and Test Automation tools and processes and how they are used in different companies.

Takeaways from QUEST and StarEast

After fun networking over pizza, wine and beer, I kicked off with an overview of my takeaways from Quest and StarEast QA Conferences. I gave a brief summary of the session that I presented at the conference titled “The Road to DevOps” followed by highlights from the most memorable sessions on Test Automation, Metrics and Mobile Testing. I focused on the importance of identifying and addressing the bottlenecks in the product delivery cycle which include manual processes such as test data management, environment configuration and test execution.

WOW (W-What we do O – on time W- with tools) at Medidata

Next, Polina and Kamlesh from Medidata presented Tools and Frameworks used at Medidata (which provides cutting edge clinical solutions).   The unique presentation style portrayed the overall solution as a cooking pot, with tools presented as ingredients and testing as degustation. Medidata solutions are built on top of Service Oriented Architecture and utilize tools such as Jasmine and Cucumber for automation, Github, TravisCI and HipChat for CI, Medistrana and GoCD for deployment, Mist, JIRA Capture and Bug Magnet for testing, and SumoLogic and NewRelic for monitoring. Polina and Kamlesh also spent some time describing VCR, an alternative way of mocking APIs.   VCR allows to run the tests against prerecorded response, but still faces challenges if cassette files become stale due to backend service changes.   The session was followed by intense Q &A covering Code Coverage tools, PACT and minimizing cycle time.

Acceptance Test Driven Development

Our last session of the day was ATDD (Acceptance Test Driven Develoment) with Anil Jaising, Executive Director at JP Morgan Chase. Anil set the background by talking about history of Lean which came from manufacturing, specifically Toyota factory. It is very important f
or Testers and to be able to pull the chord whenever they find a critical defect. Anil suggested reading the IEEE paper by Robert Martin and Grigori Melnik, the primary point of which is that we need to start thinking about testing while requirements are still developed and avoid assuming that testing is testing.   He then gave a great definition of ATDD – from user stories to business value, ATDD is a team sport as opposed to test driven development which is more individual.

At the end of the sessions, we had some great conversations with group members who stayed behind.   Looking forward to our next session. To learn more about exciting opportunities at Audible visit http://about.audible.com/

August 20, 2016
Let’s Talk About Ops: An ITSM Crowd Special

The ITSM Crowd is a regular hangout hosted by DevOps training provider ITSM Zone that covers many different IT and management topics. On Aug. 11, host Claire Agutter was joined by IT Skeptic Rob England and James Gander from Gander Service Management to take a deeper dive into the impact that DevOps is having on operational roles.

Gander comes from an operational background and wanted to discuss how the role of ops is changing, and England works with many clients who are evolving to work in more agile ways, embracing DevOps principles.

DevOps Needs Ops

All agreed that DevOps doesn’t mean No-Ops. The traditional “build” and “run” phases are no longer clearly demarcated. Dev will work with Ops to improve and fix live services, and Ops will provide Dev with feedback that will help them to improve.

Ops also needs to embrace the role of “toolsmith” or “toolmaker.” It can support Dev by creating environments that make it easy for Dev to do its job. This could include providing fast, automated access to virtual servers or providing input on monitoring and event management.

Elements of Ops such as the service desk are essential and will never disappear. Customers always will need to be able to contact IT, so this part of the operational role will be present in some form, perhaps moving toward a greater focus on self-service.

Ops Processes Won’t Disappear

Traditional operations processes also still have a place. Change management, for example, is still essential, but it needs to “get out of the way” and enable change to happen, rather than acting as a barrier.

Ops staff need to treat DevOps as an opportunity to learn and improve, not as something to be resisted. There are many great blogs to get started with, and this is an exciting time to be working in IT.

You can view the video here, and I’d love to get your feedback and comments on how you think DevOps is changing the role of operations.

August 19, 2016
The Cost of Rework when Developing for OpenStack Clouds
According to the 2016 State of DevOps Report by Puppet and DORA, the mean time to recover when a service incident occurs ranges from less than one hour for high IT performers to less than one day for medium and low IT performers. Put that in a real-world scenario: your OpenStack application and the data associated with it that spans multiple virtual machines failed (for one reason or another). Now you need to rebuild something you’ve already spent time developing. Are you—and your boss—OK with that rework taking hours to days? Or do you think you’d be wishing you fell in the range of high IT performers, and you could quickly recreate what you lost in less than one hour?

Now, take it a step further. What could you be doing with that time you now need to dedicate to rebuilding something? What will that rework take you away from? The next phase of your development? Lunch? Happy hour? Sleep? What price would you put on the personal cost to you?

The 2016 State of DevOps Report authors wanted to calculate the cost of excess rework per year for organizations. They created a formula to calculate the number:

Cost of excess rework = Technical staff size x average salary x benefits multiplier x percentage of technical staff time spent on excess rework

Here’s what they found:
- High performers have 11 percent excess rework
- Medium performers have 22 percent excess rework
- Low performers have 17 percent excess rework
At first glance, those numbers seem odd. Why would the low performers have less excess rework than the medium performers? The report clarified that by explaining “low performers spend less time on rework … because they report spending a greater proportion of time on new work. We believe this greater amount of new work could be occurring at the expense of rework, thus racking up technical debt.”

It’s clear all organizations have some amount of rework as a result of a failure of some kind. The question is, What is the cost of that rework? Let’s take a look at the results of the study.

For the most part, the simple answer is that rework will likely cost your organization millions (unless it’s costing you billions). It’s just a matter of how many millions.

We often get asked why it’s necessary for OpenStack environments to have a data protection strategy and a backup solution. The conventional (but wrong) thinking is that virtualization can take the place of data recovery. Frankly, the answer to why is contained within those numbers above. If you don’t have a backup and recovery solution for your OpenStack development efforts, are you willing to spend money and resources on reworking the lost data? And if you are, how many millions are you willing to spend on it? Let me remind you that you are trying to recover to a point in -time … persistent data included. If you say, “If you built your cloud appropriately, this shouldn’t be an issue ….” Madame or Sir, if your virtualized cloud has that kind of resiliency, then I would like to work for you.
August 18, 2016
Vivint Meets Smart Home IoT Challenges Using DevOps

Smart home technology vendor Vivint serves more than 1 million U.S. and Canadian consumers with services including broadband internet, cloud storage, home automation products, home security services and energy automation controls.

The company, which got its start in home security, leverages door and window sensors, motion detectors and glass-break sensors, as well as those that detect flood, heat, freezing, smoke and carbon monoxide in the home, said Nick Brown, director of Platform Architecture at Vivint. “We have a bunch of indoor and outdoor cameras such as doorbell cameras and the ping camera that is an interior camera supporting two-way audio.” For the smart home, Vivint extends automation devices, intelligent thermostats and Amazon Echo into the household.

Vivint used DevOps to get the software platform behind these sensors and smart home devices up and running within a year. Vivint continues to use DevOps to meet the needs of its growing platform.

Building a Software Platform for the Smart Home

Vivint started its expansion from home security company to the largest smart home provider with a challenge of growing a platform that could serve a million customers. This was necessary to fold in the 700,000 home security customers it already had in with the 200,000 new smart home customers its leadership knew were coming in a matter of months.

“We were building this new infrastructure from absolutely nothing and we needed to support 200,000 new installations in the first few months of operation,” Brown said. Vivint had its existing home security customers on a legacy platform supported by a third-party company; that company had a full 10 years to build out its infrastructure. Vivint had one year to bring those 700,000 existing customers into a new system and less time than that to prepare for the influx of new customers.

Vivint accomplished its goal, serving more than 900,000 homes in the first 12 months.

What DevOps Now Accomplishes for Vivint

DevOps enables Vivint to manage stateful and historical data at scale, Brown said, which allows the company to power the biggest smart home platform in the world. Customers see everything that is going on in their house, including current and 45-day history of what happened, through a 7-inch control display.

The real value the Vivint platform provides is in the integrated in-home experiences, which the company virtualizes and makes available anytime, anywhere around the world, giving customers a real-time view into their home data, he noted.

To build, use and evolve this platform, Vivint uses MongoDB, RabbitMQ, Nginx, ELK (once the ELK stack, now the Elastic Stack), Salt and homegrown tools. Vivint applies its homegrown code to the task of guiding a communication protocol that enables the smart homes and the Vivint cloud to talk to each other.

The company uses RabbitMQ, an Erlang language-based open-source message brokering tool that supports Advanced Message Queuing Protocol (AMQP) to disperse those messages in and out of its software platforms, driving them to and from Vivint’s services in its internal infrastructure. Vivint uses Nginx, a multifunction web server/reverse proxy server to serve web applications and services.

Vivint runs an ELK stack, which consists of Elasticsearch, Logstash and Kibana for monitoring. Its ops team uses Salt, a Python-based configuration manager to manage everything. “We’re looking forward to adding Kubernetes as a new infrastructure solution to make this a little more modular,” Brown said.

The DevOps Path to Modularity

Vivint needs its software platforms to be as modular as possible. “Each piece of the infrastructure must support a modular paradigm. Because we are following a high-risk, urgent schedule, we need to make sure that any mistakes we make along the way are easy to tear out and replace,” Brown noted. Vivint uses virtually countless services to make this possible while pushing out weekly software releases.

“From a DevOps perspective, we have a lot of things to manage here,” he said. Vivint uses AWS to support its infrastructure in the cloud and the company has its own private cloud as well. Vivint needs to move safely and quickly while retaining its agility and orchestrating all these different elements.

To bring all the smart home technologies into the new system, Vivint chose MongoDB because its data models and code tend to naturally mesh with the data models in that database. “We took advantage of the flexible schema, and the document-oriented architecture of Mongo really allowed us to write very little data access code,” Brown said. This significantly cut back on implementation time.

August 18, 2016