devopsy.kinsta.cloud

Author: contributor

DDoS Defense: Can You Tell Friend from Foe?
In many organizations, networks are at the core of the business, enabling not only internal functions such as human resources, supply chain and finance, but also the services and transactions the business depends on for revenue. That makes network availability critical. Any interruption of access from the outside world turns off the revenue spigot, impacting profit and creating a poor user experience that can damage customer satisfaction and result in customer loss. The worse the outage, the worse the damage. That’s why speed is so important in detecting, diagnosing and responding to denial of service (DoS) and distributed denial of service (DDoS) attacks.

One of the chief challenges in responding to an attack is to distinguish friend from foe. Without a way to drill down into traffic details and examine host-level traffic behavior, it can be difficult to tell the difference. Traditional network analysis technologies based on pre-cloud architectures have been too limited in their compute and storage capacity to do more than perform predefined alerting and summary reports. That’s just not enough information to really get to the heart of what’s happening in a complex networking scenario.

Fortunately, new big data techniques allow us to dig deep into huge volumes of network traffic details so that it’s possible to understand what is really going on. With a properly implemented big data platform, you can pivot your views of data to gain insight rapidly, in operational time frames, so you can act to mitigate an attack or remediate a more innocent but still painful network issue. We’ll examine data that is readily available through common network traffic flow telemetry exports such as those provided by routers and switches enabled by NetFlow, sFlow or IPFIX.

Starting at the Top

Let’s say we’re seeing symptoms of an attack in our infrastructure. We’ll use traffic flow summary data to quickly scan total traffic in bits per second just to see if anything stands out.

There’s no obvious traffic spike from this view, but then again the network we’re looking at is running at an average of 60 Gbps, so that doesn’t mean there aren’t worrying things going on at deeper levels of the network.

Analyzing Source Geography

One of the things that big data is good at is fusing many data sources together. By combining NetFlow data with GeoIP, we can look at traffic by source geography. In this case, the network doesn’t get a lot of traffic from China, so what happens when we filter total traffic by China as source?

A ha! That analytical pivot produces a graph above, showing two obvious spikes that are well above average. Below, we zoom in on the time of the spikes.

Analyzing Unique Source IPs

The spikes themselves are suspicious, but is this just a large data file transfer? We can find out by looking at how many different source IPs are sending traffic. To look at host-level details, note that we’ve gone beyond the point where you can use summary information. At this point, we are analyzing raw NetFlow record details.

Those raw NetFlow details sure are useful, because there is, in fact, a huge increase in the number of unique source IP addresses sending traffic to particular destination IPs. This tells us that we’re not looking at a large file transfer from a single machine, but a highly distributed set of senders. Botnet much?

Who’s Getting Hammered?

The next step is to to determine which IP or IPs are getting all this (probably unwanted) traffic from 14,000 or so individual host IPs.

The ability to dig into high volumes of host-level NetFlow details again proves its utility. We can see that the main target is a solitary destination IP address. There’s really only one likely explanation for traffic from thousands of hosts in a country which you have no business dealings with to a single IP that suddenly spikes from nearly nothing to more than 1 Gbps: This is a DDoS attack. Note that this isn’t a mega attack, but it still can cause real problems for whatever is running on that individual host and anything else that depends on it. If it’s your DNS server, the attack might make it impossible for lots of other servers and applications to function.

Going Deeper

Now that we know it’s a DDoS attack, we shouldn’t stop because we don’t know whether that attack is coming from other countries besides China. We pivot our analysis again to widen our lens.

Lo and behold, there is indeed DDoS traffic coming from multiple countries, including the United States, Japan, Russia, Sweden, China, Taiwan, Brazil and Estonia. Good to know.

Characterizing the Attack

Next, we want to know specifically what type of traffic we’re seeing and what that tells us. We’ll group the traffic coming from all those source host IPs by protocol.

Clearly, this is UDP-based attack. Now we’ll look at the destination port(s).

We can see that the UDP traffic is being sent to multiple ports, and it’s obvious that we’re experiencing a DNS redirection/amplification attack occurring on port 53, with a lot of port 0 UDP packet fragments being generated as collateral traffic.

Is Something Underneath This Attack?

So far we’ve gotten a lot of insight into the details of the DDoS attack from full NetFlow details. But is this volumetric DDoS the main event, or are we being distracted from looking for other, less obvious threats? We can see a lot of packets being sent to port 4444 (green line in graph).

Port 4444 is the UDP port for the Kerberos service, and is—at least for Windows machines—a well-known target for buffer overflow attacks, often used to insert trojans such as Hlinic and Crackdown.

So, potentially there are two types of attacks going on in parallel: a DDoS attack and a buffer overflow trojan insertion. Many security blogs and publications note that DDoS attacks often are used to obfuscate other exploits. This very well may be an example of that technique.

Getting a Handle on Attack Mitigation

Characterizing the attacks leads us to mitigation. One way is to take the attack traffic and group it by x/24 source network addresses:

We now can take two mitigation steps:
- Ask our upstream ISP to drop this traffic or send it to a scrubbing service or device, if we have one.
- As an added precaution, drop all traffic from these countries going to port 4444 on our own routers.
Conclusion

One of the benefits of being able to dig into full-resolution NetFlow data is that you can get operationally useful insights without needing in-line devices. You also get more freedom to employ a portfolio of mitigation methods. Big data is the way forward if you want to have full details at your disposal to deal with DDoS attacks.

About the Author / Alex Henthorn-Iwane

Alex Henthorn-Iwane is the Vice President of Marketing at Kentik. He has more than 20 years of experience bringing new technologies in networking, security and software to global markets. Henthorn-Iwane leads the global marketing strategy for Kentik and help the company bring its innovative story and solutions to network-dependent organizations around the world. Connect with him on LinkedIn, Twitter or SlideShare.
August 5, 2016
Good News: DevOps is not One-Size-Fits-All

Want to try something fun? Ask different groups around your organization what “DevOps” means. It’s fun because the odds are good you’re going to get very different answers, many of which will be vague, if not outright contradictory. This is perhaps unsurprising when even industry-recognized DevOps luminaries tout varying definitions, approaches, tools and methods.

Here’s what’s not so fun: trying to figure out how DevOps might look at your organization, and what resources and amount of time it would take to do it right. The one thing upon which everyone seems to agree is that “DevOps is a good thing” and something organizations ignore at their peril, in light of the significant competitive edge it provides.

Here’s the good news: DevOps is not one-size-fits-all. In fact, there are many different facets in defining DevOps relative to your purpose(s). This article will identify some of the axes along which the answers to those questions fall, and sketch how they contribute to the overall picture. By the end, you should be better equipped to navigate your organization through the resulting, thoroughly non-Cartesian witch-space that is DevOps today.

Axes and Allies

My use of “axes” in the previous paragraph was neither random nor merely picturesque. It was chosen because it’s a bit of a chameleon, the only English word that can be the plural of three separate noun forms. In short, you don’t know what the word means apart from context. The same is true of DevOps.

Linguistic pedantry aside, DevOps grew out of the organizational friction between developers and operations, hence the name “DevOps.” But the DevOps movement pearl unquestionably has been layered and shaped by the oyster of today’s development trends: the increased (if not nearly exclusive) focus on web and mobile applications. This is simply because it’s an easy fit, those products often being much simpler to build and deploy than legacy applications.

This gives us the first axis: the industry (or industries) in which you compete. If you’re building the next great social networking site—please don’t, I beg you!—you’re going to have a tough row to hoe. And you’re going to have to do at at the speed of Web whatever-dot-oh. Because continuous deployment isn’t an option for you; it’s your personal Obi DevOps Kenobi (your only hope). Yet, at the other end of the spectrum, even the most legacy-heavy manufacturing concern can benefit from DevOps principles, though the requisite pace of innovation should be rather lower.

Industry also can determine the degree to which you accumulate necessary but hard-to-satisfy stakeholders. In the medical industry, for example, keeping your compliance department happy is both essential and more than a full-time job in itself. The DevOps drive to deploy quickly will run straight into the sort of bureaucratic inertia one expects in a heavily regulated industry. You’ll likely need to temper your DevOps expectations and shape your processes to deliver and archive additional content if your industry is more legacy than modern.

A second axis around which DevOps revolves is the type and level of maturity of products you deliver. The more you lean toward legacy products, languages and processes, the more you’ll likely need to scope and direct your DevOps efforts differently, depending on the answers to a variety of questions. For example, it’s rather a different thing to build, test and deploy a monolithic C++ application than an app intended for the web or mobile devices.

Ask yourself: How complicated are the true essentials of your build process? Which parts can you tighten? Can you isolate and prebuild components and leverage an artifact management system? What build artifacts must be retained to satisfy support, compliance and other stakeholders relevant for your industry? You’re going to need tools that can handle the load in every sense.

Similarly, how much testing can you realistically automate for the best payoff? And what sorts of testing will give you the strongest guarantee of success for the effort? Unit tests are a good start, but have you automated your all-up functional tests? What about UI testing? Stress testing? Again, the type of industries you serve and the type of products you create will generate different test demands and degrees of time and effort to automate your test suites.

A third axis along which to evaluate DevOps is your organizational culture. I’ve heard “legacy” defined as code lacking unit tests, but this seems incomplete. I’d suggest that, in fact, there’s a legacy state of mind as well, by which I refer to the sort of inertia that accretes in products and organizations over time. You’ll find it expressed in “the way we do things.” If that phrase is a discussion-ender at your organization, then you’re going to have a much more difficult time embracing DevOps.

What allies can you bring to the table to push for necessary change? Attending a DevOps conference or event can connect you with experts with experience in your industry who might be able to recommend materials. The more your organizational attitude “leans toward legacy,” the more you’re going to need a thick bench of allies to make DevOps a reality.

Leaving a Good Legacy

If it seems as though I’ve been hammering on the notion of legacy in all its forms, I have. But it’s important to recognize that legacy isn’t always a bad thing; in fact, legacy applications are essential to a non-trivial fraction of our world. Not every need can be met with a web or mobile app. It can be very difficult to tame the legacy beast in all its forms, but it’s nearly always possible to at least trap it in a well-defined cage, then shrink that cage over time.

DevOps principles can raise your game no matter how legacy your industry, products and personnel are. But it’s important to understand where you fall on all the relevant axes to determine the right scope, methods and tools to deliver the most value. DevOps isn’t one-size-fits-all, but it can be tailored to fit in at your organization.

About the Author / John Williston

John Williston is a veteran software developer for Windows, .Net, and the web. He is currently a product marketing manager at Perforce Software.

August 5, 2016
Delivering DevOps’ Business Value

DevOps is an interesting space to be operating in right now. More than just culture or a set of tools and processes, DevOps delivers speed and agility that can be truly meaningful for the business. Unfortunately, for most large enterprises, the path to the benefits isn’t always well-known and isn’t assured.

Over the last five to 10 years, a number of DevOps tools and solutions have emerged: Puppet, Chef, Jenkins, Git, SaltStack, Elasticsearch … and the list keeps growing. Originally, these tools were sought by developers and system engineers for their ease of use and low cost of ownership: You could download it, use it the same day and solve your specific problem. The issue for enterprises, however, is that nature abhors a vacuum; the problem one developer addressed with Chef is being fixed with Puppet by another developer in a different business unit. Group A is using Jenkins, Group B is using Chef Delivery. In short, after five to 10 years of technology adoption by developers and system engineers, the tooling landscape in the DevOps space is packed to the rafters with competing technologies, all of which are solving similar problems with overlapping features.

So why should this matter to you at all? Without careful planning, consideration and execution, your business will derive little value from this DevOps exercise.

Here are a few common pitfalls we see far too often:

Too Many DevOps Tools Doing Too Few Things

Building Rube Goldberg devices looks neat on paper, but those mousetraps fail very often in real life. The more tooling and solution handoffs you have, the greater your likelihood of failure: the less likely you are to be able to scale horizontally or vertically, the more toolsets you need to maintain, and the more skills you need on your team. Convoluted messes do not yield speed and agility. If it feels wrong, it probably is. If it is overwrought, look to remove complexity from the landscape.

Reintroducing Risk and OpEx Spent on Menial Tasks

As an organization you’re onboarding risk, hidden costs and a lot of the things you just spent the last 10 years automating away: tribal knowledge, lack of governance, lack of compliance, lack of third-party auditability and reporting. Many enterprises spend millions of dollars making it so their admins and developers could spend time on more valuable workstreams, just to throw them back into the salt mines of making continuous integration (CI), continuous delivery (CD) and infrastructure-as-code “work” over umpteen toolsets. The goal isn’t for DevOps to “work”; the goals need to tie to some tangible (i.e. measurable) business benefits. Anything can be made to work, but far fewer IT programs deliver value. Be the people who deliver real value.

Taxation Without Representation

All of the stakeholders need a seat at the table. Whether that means forming a SWAT team from operations (all infrastructure domains and security), development, QA or a permanent DevOps matrix organization, all vested interests need a say in how the process and tooling develops for all impacted domains (which is just about everything) across all environments (i.e Dev, Test, QA, Production). Far too often we see one set of stakeholders acting as the service provider for another group of stakeholders playing the role of customer. This is a solution for and by Development and Operations.

Congratulations, you’re having a baby. Let’s make sure it’s not ugly, okay?

About the Author / Jesse Langhoff

Jesse Langhoff is co-founder and CEO of Raindance, a cloud and automation consultancy serving enterprises nationwide. With over 15 years of experience in leveraging technology to address business challenges, Jesse works with customers to determine the optimal path to meeting their goals through the use of automation and solutions provided by the leading vendors in the space. Areas of expertise include strategy and assessments, private/hybrid cloud services, process automation and orchestration, DevOps and operations as a service. Jesse holds an MBA from University of St. Thomas – Opus College of Business.

August 3, 2016
The DevOps Devil is in the RFP Details
The request for proposal (RFP) process in IT outsourcing is, at its best, a challenging, time-consuming process rife with pitfalls for both customers and outsourcing providers. The process is made even more complicated if customers are unable to provide clear and detailed requirements because of haste or inexperience in software development outsourcing.

In my work matching companies that need to outsource their software development to international partners, I’ve seen this particular challenge, where less mature customers do not offer potential partners enough of the needed detail in the RFP process for determining an accurate estimate. The question remains, What is the best way to navigate a situation like this?

There are two key considerations at the very start. First, because IT outsourcing can mean several types of services, including help desk, cloud enablement, network monitoring, etc., outsourcing of custom software development is a type of IT outsourcing that has specific best practices for engagement, methodology and process. Second, I recommend dropping the word “vendor” from your lexicon to refer to a company you hire for custom software development; instead, use the term “partner.” In my experience, treating your software outsourcing partner like a vendor inevitably leads to an adversarial relationship that shuts down creativity and innovation. For software development outsourcing to work, it is absolutely critical to maintain good communication between equal partners. A good working relationship cannot be underappreciated as a key element of success.

From the provider perspective, there is no ethical way to respond to an RFP that lacks details with an accurate time estimate and cost quote. Any proposal offered should be for a discovery phase, where the service provider uncovers the details the customer is unable to provide on its own. The customer can use the deliverable from a discovery phase to have a more realistic discussion with multiple service providers and receive credible proposals. If the prospective customer is unwilling to pay for the discovery phase or provide the required details, then the service provider should turn down the opportunity.

From the customer perspective, the answer to the question of how to resolve unclear requirements, then, is to require that the contract governing the relationship be for a dedicated agile development team that can start developing software as the highest priority requirements are defined. The team may grow or shrink over time to accommodate changing priorities, and the release plan can be updated every three or four sprints as you learn more about the requirements and what users want.

The customer should be prepared to commit contractually to providing a dedicated, easily accessible and available product owner to resolve questions and issues quickly. A “local” product owner can also be on the outsourced team to facilitate communication when developing apps with complex software requirements, but he or she should not be considered a replacement for the customer’s product owner.

The provider, then, should be prepared to provide at a minimum:
- Team ramp-up time
- Monthly rates per developer at various levels of seniority
- Historical retention or attrition rates of team members
- References to other customers using similar software technologies and/or domain experiences for what is needed for a particular project
- For customer experienced with agile development, an estimated velocity of story points per sprint
An additional challenge from the client perspective is to be able to define the IT outsourcing speed that is needed for the task at hand. The recent concept of two-speed IT defines two kinds of IT software development—slow and fast. Each kind can be outsourced, but with different types of service providers.

Slow IT outsourcing is most typically used for the careful update of legacy software systems that are changed slowly over months and years to maintain the stability of the business. Fast IT outsourcing is most frequently used for product development and digital transformation and requires an agile approach that includes UX design, user testing, development and DevOps. By recognizing the distinction between fast and slow IT outsourcing speed approaches, you’ll be able to select a partner company with the right experience.

By abiding by a few key parameters, customers and providers alike can make the RFP process a constructive and productive experience. While it’s true that customers always seek the best possible deal in an outsourcing bid, they don’t always offer the information required during the RFP process to help providers meet their needs. An RFP should be viewed as an opportunity to build a relationship, rather than executing a transaction, and a commitment by both customer and provider to trust and transparency will go a long way toward long-term success.

About the Author / Steve Mezak

Steve Mezak is CEO of Accelerance, a Silicon Valley firm that, since 2001, has matched hundreds of U.S. companies needing software development with international, outsourcing partners. Steve has more than 30 years of experience in the IT industry, from writing software code to facilitating and managing software development teams. He’s authored two books, including most recently, “Outsource or Else! How a VP of Software Saved His Company” (June 1, 2016). Contact Steve by email: steve@accelerance.com; or Twitter: @SteveMezak.
August 2, 2016
Multi-speed IT: The Reality of Enterprise Agility
Today’s business demands the ability to change systems at an ever-increasing speed, but for many companies, including their systems of record have created a mismatch in development and operations practices between the traditionally dynamic systems of engagement and the traditionally slower systems of record.

The problem isn’t in the ability of the systems of record to move as fast; they can change as fast as any other system. The problem lies with the tools and processes used for the systems of record. The traditional view—that to decrease risk you must reduce change—has been shown to be not necessarily true. Manual change or large changes can cause increased risk, but through a completely automated deployment process that includes configuration changes and by delivering smaller changes that have gone through automated testing together, risk actually is reduced.

The concept of multi-speed IT is not that some systems are slower and some are inherently faster, it is that the changes need to be made at the speed business requires. If a business change is required in the back end, it should be changeable as fast as any front-end change. The reality is, many back-end systems or systems of record provide a set of APIs that are used by the systems of engagement. These APIs can be more stable than the mobile application you are providing to the end user. Imagine a calendaring service—when was the last time the calendar changed? When did we add or remove days from a month? So this one is very stable; the only changes have to do with new interfaces required, such as providing a new REST-based interface. Other APIs will be much more dynamic based on business need.

So how does a large, storied enterprise bring agility to systems both old and new, mainframe to mobile, while reducing risk and fear of bringing down legacy or mission-critical applications?
1. Test across systems—Test everything together, where applications old and new cross paths, ensure thorough testing earlier and more often.
2. Release what you tested together—if you are testing earlier to ensure applications across systems are working together properly, it only makes sense to release those applications together.
3. Create a singular deployment process for that release—your deployment process should reflect whatever applications or components need to be released in the proper order to ensure functionality across systems.
Of course, to enable these actions across teams and systems, collaboration and a shift in culture are key.

Recently we hosted a webinar on how organizations can incorporate continuous testing and deployment into releases across systems and how multi-speed IT can be used as a stepping stone for enterprise agility.

About the Author / Rosalind Radcliffe

Rosalind is a Distinguished Engineer within the IBM Rational organization. She is Chief Architect for CLM and DevOps. She is responsible for driving the DevOps for multi-platform architecture. This includes System z and Power system. In addition she is responsible for the architecture for the Collaborative Management capability for Enterprise solutions. This includes UrbanCode Deploy and Rational Team Concert’s support for standard Mainframe development activities. She is a member of the IBM Academy of Technology and a Master Inventor. Prior to Rational, she was in Tivoli responsible for the SOA Management Strategy for IBM. Connect with Rosalind on LinkedIn / Twitter.
August 2, 2016
Separate but Equal: Not Equal and Not the Right Way

The DevOps Enterprise Summit in London provided another opportunity to speak to many clients about their organizations’ DevOps transformations. The key message is, “It’s all about people and culture,” but how do you make those changes? In session after session it was clear the source code manager (SCM) is central to this transformation. All clients included the SCM as a key part of the solution. SCM’s key value is as a common place to version all the artifacts from the source code to the system configuration. As such, it must provide visibility to all to allow for sharing and collaboration.

So how can z/OS participate in this transformation if its source code is locked away in some host-based SCM? The answer is, It doesn’t. In my interview with devopsy.kinsta.cloud editor Alan Shimel, we discussed this issue and he related it to the idea of separate but equal, which we all know does not work. What’s worse is separate but equal was really never equal, and the same is true for the host-based SCM to which the z/OS source code is relegated. I have asked many companies if they would move their distributed source to those host-based SCMs, and the answer is always, “Of course not; it does not provide what the distributed teams need.” How can it provide what the host teams need? It doesn’t.

Separate but equal does not work; it’s never equal and it isolates people. That’s not the way we should be looking at our z/OS source code. It’s critical to the business and should have the best support and capabilities, not the least. Now is the time to move to modern tools and processes and stop isolating the z/OS environment from the rest of the development environment.

The key is to focus on collaborative management solutions that do not separate, but bring teams together by providing one environment that all teams can use together for cross-platform collaboration. Various solutions include all the capabilities companies need to provide the modern agile development environment including z/OS. No separation—the z/OS code is in the same repository as all the rest. In these tools, the modern dependency-based build provided for z artifacts is based on Ant technology, so it can be extended just like existing build tools. With this tooling you can use your existing continuous integration pipeline and plug z into it.

It’s time we move z/OS development into the 21 century.

Read the thought leadership paper on DevOps for Enterprise System POV to know more on the essential tools and process that can enable disruptive technologies and an innovation at the speed the business requires.

About the Author / Rosalind Radcliffe

Rosalind is a Distinguished Engineer within the IBM Rational organization. She is Chief Architect for CLM and DevOps. She is responsible for driving the DevOps for multi-platform architecture. This includes System z and Power system. In addition, she is responsible for the architecture for the Collaborative Management capability for Enterprise solutions. This includes UrbanCode Deploy and Rational Team Concert’s support for standard Mainframe development activities. She is a member of the IBM Academy of Technology and a Master Inventor. Prior to Rational, she was in Tivoli responsible for the SOA Management Strategy for IBM. Connect with Rosalind on LinkedIn / Twitter.

July 28, 2016
Reinventing Monitoring for Private, Hybrid Clouds
Monitoring of systems, networks and applications has been around since the inception of computing. However, in the last five years there have been dramatic changes and innovations in the computing, networking and applications development landscapes.

For instance, infrastructure is shared and allocated dynamically—often by continuous automation. Server virtualization by VMware and dynamic resource scheduling (DRS) allowed virtual servers to move around automatically between hosts based upon CPU and memory constraints. Automation vendors including VMTurbo and Cirba have taken this one step further by incorporating constraints such as networking and storage congestion, as well as allowing for the prioritization of workloads.

These days, virtualization extends to the network and storage. In some cases, virtual networks are established in software between every two things that have to talk to each other using products such as VMware NSX. Storage is being virtualized so that there is only a faint resemblance between the storage layer accessed by the operating system and the applications and the actual physical disks or flash arrays. With that said, there also are ever more layers of abstraction between the actual application and the physical hardware. These might include Docker, the Java virtual machine (JVM), a platform-as-a-service (PaaS) stack such as Cloud Foundry and the layers of virtualization listed above.

Now, the application layer is characterized by both diversity and rapid change. It is not just a Java world anymore; other languages are just as popular and new ones arrive every day. The pressure on business to put more functionality into production more quickly has led to both agile development and DevOps—both of which result in the applications being modified in production more frequently.

Private and hybrid clouds lead to layers of infrastructure, layers of management and teams and layers of tools, creating the same problem that silos of management created in the past. Each of the infrastructure-as-a-service (IaaS), PaaS and software-as-a-service (SaaS) layers in the cloud architecture have their own teams to manage them and their own tools to manage that layer. The tools to monitor these layers are:
1. For the IaaS layer – storage tools, storage area networking (SAN) tools, networking tools, server tools, virtualization (hypervisor tools) and tools to monitor the wide area network.
2. For the PaaS layer – tools to monitor all of the software services that support the applications, which include database servers, web servers, Java application servers, PaaS frameworks and, of course, now Docker.
3. For the SaaS layer – tools to monitor the applications themselves, which include application performance management products, as well as tools to monitor actual end user experience.
The Layers of the Private or Hybrid Cloud

The Problem: Monitoring

There are several core problems with these layers of tools that must be addressed in order for private clouds to be successful:
1. Lack of visibility – The data from each of these tools at each of these layers is not shared with the owners or operators of the other layers of the private cloud stack. A consolidated set of data or metrics is not available to business constituents or users of the private cloud, giving them no end-to-end visibility into the behavior of their applications and the workloads that support them.
2. Communication issues – When there are performance problems (as there always are), the layering of the toolset and lack of collaboration across the teams that own the three layers make problem resolution a time-consuming and frustrating process.
3. Improper data analysis – Despite the vast amount of monitoring being collected by all of these tools, in many cases the data necessary to ensure the private cloud is working correctly for its hosted applications and business constituents is not collected at all—or in the required manner.
What’s Needed: New Types of Monitoring Data

Highly dynamic and interactive systems have a new approach to collecting and processing monitoring data. The following concepts must be incorporated in a modern monitoring data pipeline to stay in the game:
1. Streams of metrics – A modern online application system produces streams of metrics starting with the response time (user experience) data and including data about how all layers of the software and hardware that support that interaction are behaving.
2. Streams of relationships – In a modern online application system, a transaction of interest is related to the JVM in which it runs, the operating system upon which the JVM runs, the virtualized hardware where the operating system runs and the entire virtualized and physical infrastructure (down to the spindle on the hard disk) that support that transaction.
3. Streams of state – Most enterprises have configuration management databases (CMDBs). These are supposed to store the current state of the entire software and hardware environment of the enterprise. But now online systems change too rapidly for CMDBs to be kept up to date—relegating them to the junkyard of worse-than-useless legacy technologies. Private clouds and their modern applications are simply too dynamic for the CMDB and the CMDB must be replaced with something that is kept up to date automatically and continuously.
The Crucial Metrics and Their Sources

The single most broken thing about monitoring is the mistaken notion that monitoring for resource utilization across the set of devices and stack of software that comprises an application or a private cloud can ensure acceptable service quality.

To fix this misconception, we have to redefine performance as not having available resources, but rather in terms of how long it is taking to get work done (response time and latency) and how much work is getting done per unit of time (throughput). Response time, throughput and, of course, errors need to become the crucial metrics by which service quality is judged and delivered.

If we accept that response time, throughput and errors are the crucial categories of metrics, the next steps are to identify what these metrics are at each layer of the private cloud stack and then identify the sources of those metrics. The most important thing to realize is the standard interfaces that exist at each layer of the stack (SMIS, SNMP, Netflow, JMX, etc.) do not measure response time, throughput and errors, and we have to rely on vendors that have invested heavily in the custom instrumentation required to collect these metrics.

We also have to recognize that the investment in collecting these metrics at a particular layer is significant, as is the level of technical expertise required to collect these metrics more than what can be assembled by a single vendor or tool.

Finally, we have to recognize that the industry is changing too quickly for any single vendor to be able to cover the waterfront or the entire cloud stack. Network virtualization, storage virtualization and containers are just the most recent examples of innovations that require advances and innovations in instrumentation, and these advances are going to continue to occur at an accelerating pace.

For the reasons listed above, the only monitoring strategy that can work for the private and hybrid cloud is to focus on best-of-breed vendors that collect the required data at a layer in the cloud stack, then feed these metrics into a common big data back end for processing or analysis.

A Monitoring Architecture for the Private, Hybrid Cloud

Both private and hybrid clouds require a completely different approach to collecting data, processing data and making this data useful to users and analysts. Enterprise IT operations organizations must adopt these modern approaches to managing their private clouds or else those clouds will not be cost-competitive and functionally competitive with public cloud offerings, leading to ever more outsourcing of workloads to public cloud providers.

About the Author/Bernd Harzog

Bernd Harzog is the CEO and Founder of OpsDataStore, the real-time big data back end for all IT operations management data and vendors. OpsDataStore’s open big data back end consumes and relates data from multiple sources and immediately makes that data useful to decision-makers using market-leading BI and visualization tools. Learn more at www.opsdatastore.com, and by following OpsDataStore on Twitter, @OpsDataStore, and LinkedIn.
July 27, 2016
Managing Your DevOps Releases Efficiently

The Top 5 Things You Need to Know When Selecting a DevOps Release Management Solution

DevOps is a modern software delivery movement that encompasses agile principles and encourages automation in application delivery. Why adopt it? It allows teams to reduce errors due to manual steps and break down silos between teams to improve communication and collaboration between all stakeholders within the business.

Whether you’re knee deep in a DevOps transformation or just getting started, implementing a DevOps release management solution is an important step along the journey. Why? Because in today’s on-demand economy, organizations must capitalize on any opportunity to improve application delivery throughout the organization, irrespective of the development processes in place.

Based on a recent discussion between Michael Azoff, principal analyst at Ovum, and Anders Wallgren, CTO of Electric Cloud, I was able to glean why DevOps release management is a must for enterprises to modernize the software delivery process and compete at a higher level. More specifically, Michael and Anders discussed the top five requirements for selecting a release management solution, which I’ve laid out for you below.

No. 1: Application Manifest: Model and version applications, infrastructure and middleware.

This is what you’re looking for in an enterprise grade solution—a solution with ease of use in defining an application deployment blueprint, and being able to reuse it and version it and have variations to meet the complex and ever-changing needs of your organization. If you are doing continuous delivery and DevOps correctly, you should never have to make a change to a deployed application by hand—ideally, changes are made to the application model and then re-released into a production environment. This allows you to make and test changes in one place, and easily keep track of everything in production. Having an application manifest is a key facet for a great release management solution.

No. 2: Pipeline Design: Model, version and refactor pipelines, workflows and processes.

This is not only an important consideration for selecting a release management solution, but also is at the very heart of continuous delivery—the ability to define and manage delivery pipelines for complex legacy applications or newer microservices. For large organizations with lots of technical and process debt, being able to flexibly model and manage that sort of complexity is essential.

Ultimately, the whole concept behind DevOps comes down to the idea that there is the “what,” the “how” and the “where.” The “what” is the application manifest, mentioned above. The “how” is the pipeline design. What are all the stages this particular application or component or service goes through? Understanding your end-to-end pipeline is very important. You need to have the end to end of what your software goes through—not just conceptually, but even physically written down somewhere. If you don’t have that that process completely nailed down, you don’t really have a full understanding of what it takes to ship your software. Without this complete understanding, it can be much more difficult to have success. By modeling these pipelines, you allow teams to visualize the flow of value through the life cycle, and make iterative improvements on it in a collaborative and safe manner.

No. 3: Pipeline Execution: Predictably and efficiently deliver legacy and CD-style pipelines and releases at scale.

One of the most important things when practicing DevOps is enabling fast feedback, so getting visibility into pipeline execution as they’re occurring is very important. This provides teams an easy way to troubleshoot problems quickly, and then refactor and optimize the pipeline over time. Enterprise pipelines can be very complex and touch a lot of tools—your Dev team might be using Jenkins and Chef and Selenium, and your Ops team might be using ServiceNow and Azure cloud and Puppet. We have customers that have more than 60 tools as part of their ultimate DevOps toolchain. For each one of these tools or services you need to make sure you’re getting the best performance possible, and operating them as efficiently as you possibly can. As you execute them, the ability to see those executions live as they happen, then collect information about them and optimize your pipeline helps to make the entire process more efficient.

No. 4: Governance, Security: Role-based access control, traceability and auditability.

This is one of the key functions to consider for an enterprise-grade solution. If you just tie together your release capability with tools that still require some manual steps, you’re not able to scale that. This is where a large-scale organization, especially one that is regulated and has auditors, will want a solution that can demonstrate clearly that the whole process is governed such that no one can make a change to the process, environments or binaries without it being logged, and that any change that is made has gone through a QA and testing process. Those are the sorts of things that are really crucial for the Ops people, as they are handed responsibility for ensuring a low-risk environment.

When DevOps was starting, the overwhelming belief around security and governance was, “Gee, this is going to really bother the auditors.” However, it turns out it is quite the opposite—to get good at DevOps and continuous delivery, you really need to automate everything. What do auditors want? They want companies to do two things:

1) Document their process and

2) Demonstrate they are using that documented process.

In the process of sitting down and automating your software delivery pipeline, you are essentially documenting that process. Executing that pipeline gives you the record of what you did: How did we deploy? When? Who did it? What was the pipeline that was used? What was the testing? All of those things are now available to be audited.

No. 5: Integrations: Works with the tools you already use

You want a solution that fits in with the tools and process that you have. It would be a non-starter to show up and say, “Throw away all of your current tools and let’s use this new thing.” It would also be inefficient to have to change your process to fit into some tool’s preconceived notion of how your enterprise should run. Make sure you don’t have to change the way you do business, and that can help the DevOps adoption curve quite a bit.

By keeping these top five considerations in mind when selecting a release management solution, you can help ensure your journey into DevOps and continuous delivery is a much smoother, more efficient and more fruitful one. Regardless of the tools your organization decides to use, consider first how you can make your pathway to production a competitive advantage and reap the benefits of an efficient delivery pipeline!

If you have the chance, take a listen to the full conversation between Michael and Anders.

About the Author/Sam Fell

Sam Fell is the head of Marketing at Electric Cloud, responsible for driving awareness and demand for the company’s DevOps release automation and acceleration solutions. For 20 years, Sam has helped companies achieve success by marketing, selling and implementing DevOps release automation, IT change management and application performance management solutions. Prior to joining Electric Cloud, Sam held leadership roles at BMC, CA, Marimba and Nimsoft.

Sam holds a Bachelor of Arts degree from Illinois Wesleyan University.

July 20, 2016
7 Reasons to Move to Parallel Testing

While development operations are more scalable now than ever, there is one area that continues to constrain the ability of organizations to scale: functional testing.

Most organizations still rely on sequential tests to deliver quality assurance. No matter how rapidly a business can produce great products, its ability to bring them to consumers quickly is limited by its dependency on a slow, tedious testing process that does not adapt well to changing development needs or ever-faster business processes.

That, at least, is the way things traditionally have been. But today there’s a replacement for sequential testing: parallel testing, which lets you leverage the cloud and virtualization to perform more tests at faster speeds than your physical infrastructure or homegrown testing grid could support on its own.

What is Parallel Testing?

First, a definition: Parallel testing is the process of running multiple test cases on multiple combinations of operating systems and browsers at the same time. The process is automated, and it often runs on virtual machines. For the QA professional, no additional effort is required once the appropriate test scripts have been developed. All the testing solution needs is the test suite and a pool of resources.

Granted, new does not always equal better. But it is hard to make an argument against parallel testing, especially if you leverage a robust testing cloud. Here are seven key advantages of parallel testing to consider over sequential testing:

You Can Test for Wider Compatibility

Because parallel testing is faster, it allows you to test for wider compatibility in the same amount of time. If you’re releasing a new app and want to test the user interface (UI) functionality, you don’t have to limit yourself to testing against only the most popular OS and browser combinations or mobile devices—you can test all the combinations that are important to your users.

You Can Radically Reduce Testing Time

Good software testing must be thorough and comprehensive. In a sequential testing environment, it can be (and usually is) a very time-consuming process. Parallel testing can divide invested time (roughly) by the number of test machines being used in parallel, so that overall testing time is a fraction of that required for sequential testing.

For example, if you are running 10 concurrent tests, you can reduce your testing time by a factor of about 10. This can cut days or even weeks from delivery time. This reduction in testing time also allows you to make more efficient use of your engineering team. They are freed from a set of necessary but time-eating and repetitive tasks, allowing them to concentrate on more forward-looking and innovative projects.

Parallel Testing Optimizes Continuous Integration and Delivery

Modern development best practices such as continuous integration and delivery require frequent and rapid functional tests. The execution of pre-scripted tests on a web or mobile app saves considerable time, plus having test data accessible in detailed reports is valuable to development teams who can use this information to quickly identify issues. Parallel testing in the cloud allows you to execute test suites continually as developers submit and integrate code changes throughout the day.

The Cost per Each of Your Tests is Less

Because parallel testing is automated and and can run at high concurrency in the cloud, the cost per test is considerably less. Leasing test time on a cloud testing service generally will be much less expensive than building and maintaining your own test infrastructure.

Even the common practice of using development hand-me-down computers as test machines has its costs and practical limits. For example, as used systems age, they require more maintenance and they become less capable of duplicating real-world deployment conditions. Cloud-based testing grids mean environments have high availability, and they are up-to-date, making parallel tests the better option.

It Supports Testing Science

Parallel testing enables teams to focus on making tests repeatable, measurable and accurate. By simply testing faster, you can test more, and testing more produces more actionable data to find bugs and also improve test cases. Automated parallel testing delivers strong data on test results with key indicators of where tests can be improved. The net result is a test process that is more scientific.

You Can Still Perform Sequential Tests, If and When Needed

Parallel testing simply offers a different framework for testing, which can easily accommodate most or all of your existing test regime. This means that transitioning from sequential to parallel testing does not force you to sacrifice test targets, strategic testing goals or test quality.

If anything, moving to parallel testing increases the range and depth of your test program, allowing you to better meet your test goals and it improves the quality of your testing. When you make the move to parallel testing, you are not abandoning traditional software testing. You are simply applying modern development practices and technology to it.

You Do Not Need to Make the Switch to Parallel Testing All at Once

The great thing about parallel testing is that you can transition to it in steps, starting with the tests that are most easily converted to parallel, then adapting some or all of the remaining tests after your team has had time to adjust to the parallel testing environment. Once you have started to use parallel testing and have integrated it into your development process, it is easy to expand the quantity and scope of parallel tests as the need arises.

Conclusion: A Faster Test Run is a Penny Earned

The faster you run a test, and the more tests you run, the more frequent the releases and the more defects you can find. The math is simple. Parallel testing in the cloud decreases the cost of maintaining test infrastructure, the cost of lengthy testing cycles, the cost of delayed releases and, finally, the branding cost of unhappy customers due to poor application quality.
Even without changing the way you test, a shift to a parallel testing environment will provide an immediate speed enhancement—which means every test that still runs serially is a cost that can be avoided. And every test run in parallel is an opportunity to test your application better and faster at the same time.

About the Author / Lubos Parobek

At Sauce Labs, Lubos Parobek leads strategy and development of Sauce Labs’ web and mobile application testing platform as VP of product. His previous experience includes leadership positions at organizations including Dell KACE, Sybase iAnywhere, AvantGo and 3Com. Parobek holds a Masters in Business Administration from the University of California, Berkeley.

July 18, 2016
Using Case Management to Tackle the Unknown

One of the biggest challenges for an organization is producing IT response models that accurately reflect real-world scenarios. It’s pretty straightforward to establish protocols for predictable situations. We call these “standard” scenarios.

It gets harder when you have to tackle the unknown.

“Case management,” as it’s called, involves establishing strategies and methods to approach unpredictable situations. In my recent whitepaper, titled, “How IT Response Models Drive Modern Operations,” I explain the benefits of combining these two approaches to have a complete model for all forms of responses that arise.

Standard response models establish and standardize processes for certain situations. Companies that I have worked with use standardization to increase performance and improve outcomes through to faster response times, more efficient resource usage and proven methodologies. Agile methods have accelerated the pace of change; to keep up, your organization should pinpoint opportunities for automation and enhanced collaboration. Many companies are using DevOps to do so, boosting workflows by integrating new communication and collaboration tools.

It’s all great until a situation arises that has no response template to follow. Then organizations fill the gap with case management. Unfortunately, most IT organizations are not even aware that case management discipline exists. Sometimes their business has case managers, but it does not occur to apply the same methods to IT. These organizations perform ad hoc case management without any formal guidance, rules, resources or methods. As a result, they often perpetuate unnecessary inefficiencies and fail to address business-critical problems.

By applying a level of formality to case management, organizations can confront the unfamiliar head on and maximize their chances of handling incidents successfully. By acquiring and analyzing useful data, and devising ways to improve mechanisms over time, they eventually can create new standard models.

Embracing the flexibility and agility afforded by DevOps is key to a solid case management strategy. Case workers can choose procedures from any number of sources to assemble a case management process. They can create toolchains to connect the procedures, making team hand-offs and collaboration more efficient and effective. This allows organizations to structure and control the response process as much as possible, even in unpredictable situations. If you have the right tools to help your teams work together, you can better adapt and respond to new information, external events and actions taken. And, as I explain in my white paper, the benefits of case management can extend beyond expectations and lead to greater flexibility in responding to user needs as well as higher customer and user satisfaction.

Standardization is the goal: Although we can never completely standardize the world, you should do your best to transform the uncharted into the defined. Having a robust case management process allows you to identify new unfamiliar scenarios and ultimately develop mechanisms for standard models of response. By treating cases with the same level of rigor as standardized procedure, your IT department can truly maximize the benefits of DevOps.

The standard+case approach introduces case management to our thinking for the same reason that we introduced DevOps: to bring in defined practices, mature management, relevant reporting and structured improvement so we can be more effective and efficient at responding to the situations the world presents to us.

Check out my white paper for more on info on how to implement standard+case.

About the Author/Rob England

Rob England B.Sc., MIITP, CITP is an independent IT management consultant, trainer and commentator based in Wellington, New Zealand. Rob is an internationally recognized thought leader in IT Service Management (ITSM) and a published author of seven books and many articles, best known for his controversial blog and alter-ego, the IT Skeptic. He speaks regularly at international conferences. In 2005 Rob founded his company, Two Hills, with the motto “Sensible Business Practices.”

Books: http://www.twohills.co.nz/books

Blog: http://www.itskeptic.org/

July 15, 2016