devopsy.kinsta.cloud

Author: contributor

How to “hear” OpenStack silent failures earlier and remediate faster
Troubleshooting “silent failures” in an OpenStack cloud environment has been a tough problem to solve for many production users. eBay’s chief engineer of cloud, Subbu Allamaraju, whose team has been operating PayPal on an OpenStack-based private cloud since March 2015, has publicly defined one of his ongoing challenges as: “Too much noise, too many alerts, and still working on how to detect troubles before customers find out.”

For enterprises and service providers in various stages of rolling out OpenStack, tackling silent failures invariably starts with two questions:
- Why and where do silent failures occur?
- Can my Ops team receive push notifications of warnings earlier and proactively remediate them before silent failures materialize?
Silent failures happen because OpenStack Ops teams often don’t know which service domains to tune into for early-stage warnings. Even if they do, the warnings get drowned out by noise. For instance, your metrics and logs for the compute domain show a successful VM creation. Yet, a tenant is still not able to use her requested VM instance – a silent failure – while the warnings actually show up in metrics and logs outside compute domain, such as DHCP agent failure and identity database query timeout.

To catch a situation like this, OpenStack Ops teams need a real-time, data-driven approach that automatically detects early warnings in DHCP agent and identity service, then correlates them with VM creation, and finally push-notifies only the relevant domain experts to collaboratively remediate “tenant can’t use requested VM.” All noise has been removed and related warning signals have been clustered and contextualized together – automatically.

And Operators shouldn’t have to rely on writing rules to model silent failures. Writing rules will be too time consuming and are unable to cover all failure scenarios. There are too many service domains – at least 13 of them (e.g. compute, identity, dashboard and storage) – plus their service daemons and virtual switch tunnels that stitch them all together.

Using real-time, data driven management to resolve OpenStack silent failures has been a recent success for OpenStack. For example, a large OpenStack production shop was able to proactively detect OpenStack cell failures by receiving early notifications from outside the cell domain – CEPH and JBOD I/O failures.

By becoming data-driven and ingesting all events and status messages across the entire OpenStack into real-time, machine learning algorithms, OpenStack Ops team automatically receive cleaned and contextualized alerts, then use modern social collaboration technologies to rapidly troubleshoot issues, capturing tribal knowledge for the future. The new mantra for these teams should be “clean, contextualize and collaborate,” the 3 pillars of real-time situation management.

But real-time situation management doesn’t just stop here. It scales across the entire ecosystem that lives in and around OpenStack. Data sources include the ever-pervasive RabbitMQ message bus across all services, metrics and logs across the diverse tenant operating systems, and metrics and logs across all technology and vendors. In essence, monitoring the entire technology and vendor independent reference architecture can be rolled up and simplified as “situation management” vs. reacting to many seemingly unrelated “domain silo alerts.”

Real-time situation management can also sit above multiple OpenStack instances for load sharing and DR purposes, and the interactions with public cloud. For instance, Amazon CloudWatch can be quickly incorporated as a data feed, giving OpenStack Ops teams a “manager of managers” view across OpenStack hybrid clouds.

Real-time situation management essentially provides Ops team with a single pane of glass to see and resolve failures earlier, with amplified signals and suppressed noise. Doing so without relying on extensive rules offers much needed flexibility for Ops team to keep up with so many “unknown unknowns,” as the OpenStack community continues to evolve its entire reference architecture and ecosystem.

About the Author/Feng Meng

Feng Meng is a Senior Director, Product Marketing at Moogsoft. His expertise lies in IT service assurance, with 15-year technical marketing background in Application Performance Management (Bytemobile, acquired by Citrix), Network Performance Management (Cisco), Infrastructure Monitoring (VSS Monitoring), and Server Management (HP). Twitter: @fengmengmoog @moogsoft Linkedin: https://www.linkedin.com/in/fengmeng1970
July 10, 2015
The Next Wave of Software Delivery – Integrating Agile and DevOps for the Digital Enterprise
In the past five years business cycles have greatly accelerated, forcing companies to respond faster to customer needs and competitive demands. This trend has been driven by business adoption of consumer-oriented technologies like social media, smart phones, b2c internet services. We all expect to have a constant stream of new services and capabilities delivered to us quickly and with little disruption.

There is currently intense competition across all industries, creating a digital “survival of the fittest” business environment. If a business function can be enhanced by a digital solution, it will be. Most enterprises are now dependent on the rapid implementation of new digital technology and the software solutions to remain competitive.

This extraordinary change has been achieved in part through adoption of Agile development, which promotes close collaboration between business users and developers. In Agile, environments, software development is broken into well-defined intervals with the goal of releasing software quickly, generally within less than a month. Updates are ongoing and nearly continuous, allowing product development teams to pare delivery time by 50 percent or more.

Adobe is a good example of this new way of doing things. After deploying CloudMunch DevOps platform Adobe has increased the pace of application development by as much as 60 percent.

Already, companies like Twitter, GameStop and Etsy use Agile processes as part of DevOps for continuous innovation. In fact, in a recent IPO prospectus, Etsy noted it updates code every 20 minutes, or 70 times per day. In 2014, the company executed 10,000 code “deploys”—a number that was inconceivable just a few years ago.

However, there is a disconnect between traditional IT processes and new Agilde and DevOps delivery processes. Here are three barriers to integrating Agile and DevOps in the enterprise:
- Traditional operational IT processes are much longer than intervals used in Agile. While IT is often planned and measured in quarters on an annual budget cycle, Agile development may happen on a two-week sprint. Once built, a business group is eager to use these new capabilities, but most IT organizations aren’t prepared to deploy new capabilities that quickly.
- The supporting software ecosystems are very complex, since applications are built by standing on the shoulders of giants. The comprehensive features supporting software allow development teams to focus on the desired business capabilities rather than low-level utility components. Since all of these components are updated frequently, the process of tracking and deploying the right supporting ecosystem presents a complex problem.
- Agile teams have autonomy in the use of development tooling, which results in a number of different and disconnected development tools being used. This also makes it difficult to implement these processes and provide the metrics that provide a consistent view of the enterprise portfolio of development projects.
Using modern tools for success in today’s hybrid cloud

To support rapid implementation of infrastructure needed to support the Agile delivery process, enterprises are increasingly moving more of their essential business operations to the cloud. For example, Adobe has embraced CloudMunch, a DevOps platform to automate delivery of software updates to dozens of its products, pushing multiple updates more quickly and easily to its customers.

Today, most companies have a hybrid IT infrastructure, comprised of both on-premise and cloud systems. While they have moved many functions to the cloud to take advantage of the benefits of cloud computing, they still keep many systems in-house. In the past, they’ve used development tools like Chef and Jenkins to manage application development and infrastructure automation, but these aren’t enough in today’s hybrid environment. Increasingly, businesses need to plug in and integrate their developers’ chosen tools into a single workspace to apply Agile development to a DevOps environment with a single dashboard. This will be critical if enterprises need to do Agile and DevOps at scale

This ability to manage Agile development in a hybrid cloud environment will become even more important in the next decade. Already, new platforms have emerged, allowing development teams to better orchestrate multiple toolsets & processes in a single “pane of glass” to streamline collaboration and application delivery. To stay competitive, businesses should assess their development infrastructure to ensure they don’t get left behind as technology progresses.

The impact of these accelerating changes in virtually every industry is astounding, affecting every facet of an organization, from company culture to management styles and business infrastructure. This is driven by changing customer expectations — breakneck speed and complexity have become the new normal. Companies are being forced to seek new development methods and leverage the cloud to create software that has the capacity to continuously adapt to ever-changing user needs.

These new approaches mean companies need to intensify collaboration and address the disconnect that exists between the old and new worlds. Successful integration of DevOps and Agile development will play a key role in the success of companies in today’s digital economy. In the new landscape, it’s about rapid, continuous high quality software delivery for the digital world, and companies that can do this at enterprise scale will outpace those that struggle to adapt.

About the Author/Pradeep Prabhu

Pradeep Prabhu, Co-founder and CEO CloudMunch, Previously Pradeep was Vice President and Head of SaaS for Infosys, where in his 18-year career he was instrumental in growing the tech company to a multi-billion dollar business. Before pioneering Infosys’ SaaS business, Pradeep incubated the Infosys-Microsoft solutions and alliance program and the Microsoft Technology Center. His previous leadership roles include establishing the Infosys Seattle office and delivering many successful software programs for marquee clients like Amazon, Boeing and Microsoft and conceptualizing and launching the internet banking product eFinacle, which today is installed in banks around the globe. Follow on twitter @cloudmunch
July 9, 2015
Security at the Speed of Innovation
With the advent of the cloud and platform as a service (PaaS) offerings, developers are innovating faster than ever. But with this rapid pace comes risk. Companies quickly implement new apps to meet their business demands, but the increasing complexity of the infrastructure being used – both in the cloud, on premise, or a hybrid combination – makes it difficult to ensure vulnerabilities are not introduced or vital corporate information is not exposed. In the quest to ensure that policy and governance are in place and security risks are negated, IT often brings innovation to a screeching halt.

Is there a happy medium that enables developers to continue accelerating innovation, but makes IT confident there’s adequate security? In this article, we’ll look at the importance of governance and policy, and outline the five fundamental security requirements that will help enterprises bridge that chasm that currently separates development and IT.

Policy Debt – The Downside of Faster Innovation

Security flaws are not new. Enterprises have faced serious security problems for decades. As today’s computing landscape becomes increasingly diverse and complex with increasing use of the cloud, new vulnerabilities are introduced, which exacerbate problems with existing security methods, such as simple usernames and passwords and an over-reliance on network perimeter security models.

The rapid introduction of new developer technologies far outpaces an enterprise’s ability to vet them. As a result, enterprise IT faces a tradeoff: rapidly adopt new technologies to satiate business demands thereby accepting more risk, or slow down new technology adoption with the hope of mitigating risk. Tension between enterprise IT and the rest of the business is the inevitable outcome. More importantly, because new apps have so many dependencies, as soon as one is put into production, an enterprise almost immediately incurs policy debt.

Policy should enable enterprises to track these dependencies, but in today’s IT environments, each piece of infrastructure or cloud resource has its own unique policy model, operational mode and lifecycle. As a result, enterprises end up with a layered cake of dependencies, which is difficult to keep track of and ensure no security holes are left unpatched.

Most of today’s security technologies – like gateways and firewalls – are either point-based or palliative solutions. They treat symptoms rather than root causes. And, they only work for traffic that’s routed through them, which precludes most of the server-to-server interactions that happen within the datacenter. Reliance on gateways is akin to locking the deadbolt on the front door of your house but leaving the back door open.

These simply are not enough to address the proliferation of attacks that have paralyzed companies and exposed customer and consumer data. Consider the three major security Linux exploits – Ghost, Shellshock and Heartbleed – that came to light in the past year. To patch their systems against these problems, companies needed to find the vulnerabilities and reset servers to activate the fix, which resulted in a cascading failure of servers, downtime and a host of other performance-related issues – simply because there’s no centralized way to apply policy and determine whether a resource in use is vulnerable.

The Fundamental Five Security Requirements in the Brave New IT World

To fully protect their increasingly complex IT environments, there are five fundamental security requirements no enterprise should be without:
1. Resource Naming and Identity – To secure access of a workload or process, it needs to be named so that it can referenced as logic is applied. There is no notion of identity for the applications running today. They’re simply workloads running on a CPU. If enterprises want to apply policy, they will need an overlay operating system that gives identity to that workload, so it can be organized into a manageable, policy-driven hierarchy. This type of naming scheme is being done for Internet domains, but there is nothing comparable for the data center that enables IT to find, maintain and operate workloads residing there, regardless of where they are physically run.
2. Chain of Trust – In today’s data center topologies, no actual proof is needed to ensure that a holder of a token is exactly who it claims to be. To guarantee a high level of proof of identity and actual trust of components, enterprises require a central signing authority that issues tokens to all clients and internal components. This ensures that the management backplane is resilient to token and credential theft, and the system offers a high degree of trust.
3. Context Aware Policy – Most systems rely solely on role membership to make authorization decisions, such as “who” can do “what” to “what.” But enterprises need to go further than that, with the use of a dynamic scripting language for authorization decisions. Every authorization decision will require information over and above simple role membership (date and time, authentication strength, targeted resource and more). The result is a system that adapts more easily to enterprise authorization requirements.
4. Control of the Network – The network is a vital part of any enterprise system, and controlling access to network nodes is key for enterprise IT governance. Today, network access control is a hodge-podge combination of network perimeters, firewalls, VLANs, VXLans, proxies and security appliances. Workloads exist in network segments and workload connectivity is a function of network segment configuration. Over time, these configurations become rigid, brittle and difficult to maintain. To prevent this, enterprises need to control the network within the cluster so that any workload running must be given explicit access to the network and other workloads (via policy) in real time and regardless of physical properties of the infrastructure.
5. Ephemeral Service Credentials – Applications and services require connectivity and sometimes credentials to talk to one another. Once an application gets a credential for a service, the application becomes responsible for safeguarding that credential. However, credentials sometimes leak — they can appear in log files or configuration files, and can even be scraped from memory. What enterprises need is a new approach to service access that binds a credential to an individual network path. This ensures that even if a credential were to leak, it couldn’t be used from another location on the network. Moreover, with this method credentials live only as long as an individual workload in the system, rotating every time the application stops and starts again. By taking this approach, the burden on applications for handling credentials is reduced, strains of NetOps and network perimeter security models at scale decreases, and weak links in the trust chain are removed. The result is a more secure system that you can trust without changing your existing investments.
About the Author/Derek Collison

Derek Collison is founder and CEO of Apcera. An industry veteran and pioneer in large-scale distributed systems and enterprise computing, Derek has held executive positions at VMware, Google and TIBCO Software. As CTO and chief architect of cloud application platforms at VMware, he co-founded the cloud computing group and designed and architected Cloud Foundry,the industry’s first open PaaS. During his time at Google, he co-founded the AJAX APIs group.
July 9, 2015
Security Breaks DevOps – Here’s How to Fix It
The concepts of communication, collaboration, abstraction, automation and orchestration are cornerstones of the rapidly growing DevOps movement. At the same time reliance on virtualized infrastructure and Infrastructure-as-a-Service has exploded, making manual provisioning and management simply not feasible anymore; it takes too long and locks up too many resources. Modern DevOps methods and tools have emerged, allowing IT organizations to move faster and with higher quality, thus giving them the ability to respond to the business with more agility.

Now security teams have an opportunity to learn from the DevOps experience. Manual policy provisioning and security operations in highly dynamic IaaS environments simply doesn’t work, for the same reason it doesn’t work for DevOps teams – the pace of change is simply too to fast to handle manually.

Applying security policies based on static parameters and making manual rule changes just before production leaves little time for provisioning the policies. This impacts release quality, increases risk of errors and slows down the DevOps cycle.

Trying to use DevOps orchestration tools to provision security can leave companies exposed since these tools lack critical controls and don’t integrate with the rest of the security infrastructure.

To solve these challenges, IT and security teams need to adopt platforms and processes that match the speed and agility of their DevOps brethren.

Here are the key ingredients you should look for in security solutions that can move at the speed of DevOps:
- Built-in Automation – Security automation means that any control (e.g. firewall policies, configuration vulnerability scans, intrusion detection, multi-factor authentication) can be deployed and managed without human intervention. Most desirable is full-lifecycle automation, in which policies are set once and tied to some context, after which underlying controls are 100% automated at each stage of the control’s lifecycle, from deployment to de-provisioning. Automated collection of audit and operational data is also critical, especially in environments where infrastructure components are only operational for short periods of time. Even though short-lived, these ephemeral resources are still in scope for auditor inspection, even if not running at audit time. Well-implemented automation enables security organizations to keep up with the scale and rate of change associated with dynamic infrastructure models. Security accuracy and effectiveness are both improved by automation, and potential for human error is removed—especially if API instrumentation enables cooperation of otherwise disparate technologies.
- Security Orchestration – Platforms that enable security orchestration centrally manage the composition, deployment, and management of individual control components into more complex, service-oriented security systems. By composing many individual controls into a larger system, security orchestration is considered to be a higher order function than simple control automation. In many implementations, orchestration also addresses licensing, metering, chargeback, and other security resource consumption issues that are important in service-oriented cloud computing and software-defined infrastructure environments.
- Instant Visibility & Continuous Enforcement at the Workload – Public clouds have no natural perimeter and network segmentation, which leaves individual servers exposed. In private clouds, malicious East-West traffic inside the network is undetected by perimeter tools and can become a serious threat. So choose a security platform that extends your investments in network security directly to the workload itself. The solution should be on-demand and easy to deploy. Many of these platforms have an agent-based model, so make sure the agent is ultra lightweight to eliminate drag on the virtual server, is non-intrusive to the workload and is easy to integrate into DevOps continuous deployment model. The agents should be deployable through orchestration tools, with scripts or manually, even on live systems without reboot, to speed up the process even further.
- Flexible Policy Definition – A modern security platform should allow security policies to be defined by logical application groupings instead of static network parameters, which protects new workloads automatically and overcomes natural limitations of traditional network security tools.
- Security at Every Stage – The DevOps model has multiple stages, many of which are conducted on various cloud services and on other virtualized architectures. This leaves assets vulnerable to attackers, so baking in security at each stage prevents this issue. Just as importantly, development teams need to know how security will impact the application being developed, so incorporating security early in the process makes a ton of sense.
- Layered Approach – Having a platform that provides layered security (not just a firewall) in the DevOps model, is key. Integrating multiple functions from different vendors would prove enormously daunting from an orchestration perspective. So make sure layered security functions like file integrity monitoring, security configuration monitoring, strong access control and vulnerability management are baked into a single platform and included on every system throughout the lifecycle.
- Seamless Integration with Orchestration Tools – Make sure your security platform integrates seamlessly with the orchestration tools you’re already using. Jumping back and forth between tools can slow things down, introduce errors and lower your overall security posture.
By implementing a security solution that incorporates all of these attributes, IT can bring security into the high speed, high quality DevOps model that is now required to provision and manage modern infrastructure.

About the Author/Amrit Williams

Amrit Williams is the CTO for CloudPassage. Previously Amrit was the Director of Emerging Security Technologies and CTO for mobile computing at IBM. Prior to IBM, Amrit was a research director in the Information Security and Risk Research Practice at Gartner, Inc. where he covered vulnerability and threat management, network security, security information and event management, risk management, and secure application development. Previously, Amrit was a director of engineering for nCircle Network Security, and undertook leadership positions at Consilient Inc., Network Associates, and McAfee Associates.
July 8, 2015
What CIOs need to know about building deployment quality

Application Release Automation, or ARA, is the process of packaging and deploying applications (or updates to applications) in an automated, repeatable and consistent way.

In today’s market landscape software has become a major differentiator for many businesses, across all industries and verticals. In a world of apps, Internet of Things and customers’ expectations to be always connected, – just about every company is now becoming a software company. Software development is now critical to the success of your product, and its delivery process needs to be fine-tuned and streamlined to ensure that you get quality software in the hands of your end users as quickly as possible.

To understand how ARA can help you achieve higher quality software releases (which contribute to your bottom line), you must first understand the challenges you might face if your business is still using manual deployments, or other archaic methods. If that’s your case, though, you’re not alone: a survey we ran last year found that over 75 percent of deployments are still done manually, despite the many problems with this model, which is extremely brittle and error prone. This creates not only many failed deployments, but loss of hundreds of men hours trying to troubleshoot those failures. This is certainly painful in the mission critical production environment, but carries great cost long before we reach production. Even in the lower environments (such as testing or pre-production), 66 percent of respondents report its very time consuming just to troubleshoot why the failure occurred. As we promote code to be deployed in production- the stakes are higher. Costs of failed deployments, delayed releases and loss of men hours are exponentially aggravated when you add possible fines on top of that, cost of SLAs violations, customer churn, and loss of business. There’s no question the implications of failed deployments put a major dent in your bottom line, and with some companies – failed deployments have literally put them out of business.

A prime example of how a failed release can have huge impacts on a business is the case of Knight Capital. Knight Capital WAS a financial firm. In August of 2012, they manually pushed out a new release to their software when the markets were down over the weekend. Monday morning came and when the market opened, their new software kicked in. The results were disastrous. They accumulated (extremely quickly and erroneously) a $9 billion [H1] position in the market – represented by the movement of 397 million stocks in just 45 minutes. Two hours later, when Knight was still chasing the culprit in their release process and trying to rollback to a previous version of the software, the New York Stock Exchange said, “We’re done waiting for you,” cut their hardline to the market and Knight couldn’t make any trades.

The effects were disastrous and swift. Knight Capital endured a pre-tax loss of $440 million. Within weeks, Knight Capital was sold for pennies on the dollar – a failed release literally killed the company. While this is certainly a worst-case scenario, it is a great example of how it is necessary to understand that once you start talking about production environments, there is a lot of regulatory and best practices that have to be followed or big problems can occur.

Why are manual deployments so problematic?

First of all, manual deployments are error prone because each deployment has to be done by an actual human. Even the most veteran of developers or ops engineers make mistakes, and when you start talking in the hundreds (or thousands) of deployments, across complex stacks and configurations, an error is almost inevitable. While no system can fully-eliminate the potential for an issue, you can imagine that in a deployment process where many of the steps are automated – and therefore consistent across releases – your chances of an error are reduced dramatically.

With manual deployments, it’s also difficult to achieve true, fine-grained, visibility and auditability into the entire deployment process of all the application’s components and their corresponding infrastructure or configuration. Once a deployment is pushed live, the only person who really knows how, exactly, the update was promoted from Testing to Production is the individual (or individuals) who performed the release. And as good as they might be in keeping notes, they’ll never be as comprehensive, or bulletproof, as an automated log that track every single component in the multi-stages deployment process.

Manual deployments also are not adept at negotiating what is called the “Delivery Gap.” Essentially the delivery gap is the chasm that exists between three different factions of the software team. On one side there is the development team, which has a bunch of tools and practices they like to use. On the other side is Operations, which has their own set of tools and methods/requirements. Then, finally, there is the infrastructure, which spans across both Development and Operations and places requirements on both parties. So you have essentially three different players, using different tools, with difference practices and requirements, and negotiating between them is a tricky and difficult process. Ensuring there’s no configuration drift between infrastructure or app components across the different team, and ensuring consistency in deployment processes has proven difficult for humans to perform, particularly as each process includes hundreds – if not more – of artifacts, configuration files, dependencies and deployment requirements. For an automated process, however, consistency and repeatability are the default state.

So now that we’ve established the pitfalls of manual deployment practices, let’s talk about how Application Release Automation can solve some of these issues and support higher quality software releases.

We’ll start with the aforementioned “delivery gap.” Application Release Automation is a DevOps solution that bridges the gap between the Development teams and Operations teams. It does so by creating a solution that encompasses Continuous Integration, Configuration Management and Deployment practices that respect infrastructure, as well as operation’s needs and regulatory requirements. It achieves this because you can build these considerations into the automated process, which is recorded, logged and tracked by default. Rather than each deployment being a unique ‘snoflake’, software deployments become routine and predictable. It’s not on one person to mentally “flip the switch” and jump back and forth between groups to make sure everyone are satisfied. Rather, it is an automated process that takes everyone’s interests and requirements into account, and then makes sure everything fits within the infrastructure and configuration requirements.

Furthermore, ARA is a “model-driven” approach that allows you to reuse existing templates, configurations and processes over and over to make your deployments consistent, and require minimal overhead for the initial set up of the process. This means you can deploy any version to any environment quickly, with push-button deployments. This not only increases the speed of deploying across different environments throughout your process, up until the final release to Production, but also allows for repeatability in a way manual deployments simply can’t. This is especially important when it comes to testing. Imagine if the framework of every update you deployed was identical – how much easier would that make testing? And how much easier would your tests be if they were all ran continuously and automatically? That is what ARA can offer.

Additionally, with consistency, your deployments are very predictable. You know what the run times will be, allowing you to plan your development processes more accurately and efficiently. This means you can allocate time and resources in a way that maximizes your output – which allows your team members to spend more time on their core competencies and improving your product, versus having to manage the deployment process or the infrastructure.

Along the same line, one of the key tenets of ARA is packaging. When you manually deploy updates, bits and pieces that comprise your application are being deployed (“moving”) at different times, creating a “noisy” process, with lots of dependencies, that’s difficult to manage. With ARA, everything is wrapped up in a tidy “package.” This effectively creates a “bill of materials” for your release and all dependencies and related configurations, which allows you to easily indicate what components of the application need to be deployed where, and in what order.

Application Release Automation also solves the issue of having such a critical process to the company be dependent on what is often just one person – a “star student” in your team who “holds the keys” to the project and only he has the tribal knowledge and expertise to perform or debug a deployment. With ARA, there is an implicit transparency to the project that gives any team member the ability to access the system and deploy the software (pending the right security permissions, of course).

Lastly, with ARA, you have complete visibility of the process from start to finish. If something goes awry, you have a much better view of where the error originated from, making recovering much faster and easier, and the damage much less significant.

Ultimately, a successful ARA strategy eliminates the need to build and maintain custom scripts for application deployments, while simultaneously reducing configuration errors and downtime. It allows strategic allocation of resources, giving users the most “bang for their buck” from their developers and other IT professionals. Because of all these factors, Application Release Automation is a hugely effective strategy for increasing both the quality and speed of your deployments.

About the Author/Anders Wallgren

Anders Wallgren is Chief Technology Officer of Electric Cloud. Anders brings with him more than 15 years of in-depth experience designing and building commercial software. Prior to joining Electric Cloud, Anders held executive positions at Aceva, Archistra, and Impresse. Anders also held management positions at Macromedia (MACR), Common Ground Software and Verity (VRTY), where he played critical technical leadership roles in delivering award winning technologies such as Macromedia’s Director 7 and various Shockwave products. Anders holds a B.SC from MIT. Connect with Anders on LinkedIn or Twitter.

July 8, 2015
The Wonder Pill for Mobile Application Success

Dr. James Brown stunned the British Science Festival by announcing a wonder pill that could reduce your weight, lower the risk of high blood pressure, cancer, diabetes, depression, and anxiety. With everyone in attendance eagerly awaiting what it was, he plainly exclaimed that the “wonder pill” was daily exercise.

The most effective solution was the simple one. The same applies towards developing a successful mobile application.

There is a sure fire way to generate greater sales, leap frog the majority of your competition, promote wider brand recognition, and win loyal customers. The wonder pill: make sure your mobile application works.

“The majority of mobile applications get a one star rating from users due to poor quality. Either the apps don’t work properly or they perform too slowly. The best way to guarantee the financial success of a mobile application is to simply test it properly so the user has an ideal experience.” says Tal Barmeir, CEO of Experitest. If a mobile application has a function that isn’t working, 80% users will simply delete it. In some cases they will tell their social media followers about how lousy the company is. They won’t pan the app mind you – they will put down your entire company. Performance is no different. A slow performing application, even if it works, might as well be broken.

If a daily walk is the simple way to quality health, mobile application testing is the simple way to a quality app. If you can guarantee that your application works properly, your mobile application just leapfrogged the majority of its competition that cannot make that claim.

Is Mobile Testing as Simple as a Walk in the Park?

To guarantee top quality of your app you have to test it over all types of devices, and for each device you have to test over several mobile operating systems. For each permutation of device and operating system you need to make sure your application works fast. But be warned! Just because it works fast over a cutting edge local network in Singapore doesn’t mean it will perform as well in rural Indonesia.

Add in smartwatches, other wearables, and apps managing the internet of things and the matrix of test scenarios you need to cover can seem endless.

The best testing tools make it look easy. You can set up an on premise mobile device lab in your company, and using the cloud attach as many devices as you want to it. Each device can have a different operating system, creating for yourself a large a matrix of devices and operating systems, expanding your test area to meet your needs. Your quality assurance testers can access these devices by remote using automation testing tools from wherever they are in the company. A manager can decide to assign devices to different testing teams working in different time zones to create a 24 hour nonstop testing operation. By including a Network Virtualization tool, testers can simulate any local network condition on earth. They can verify that their application works as smoothly in Nairobi as well as it does in New York City. The best testing tools also constantly update to enable you to test for new technologies like wearables and the IoT.

That’s why mobile testing is the wonder pill of mobile application success. It makes your mobile application works. It makes sure your mobile application performs fast. It makes sure your customers and prospects know that your mobile app is the best there is – and they tell their friends about it.

About the Author/David Fink
David Fink is a product evangelist for Experitest, a leading provider of mobile test automation, manual testing and monitoring tools for mobile applications. David worked on Wall Street during the internet boom of the 1990s and 2000s. Watching the same transformation taking place in mobile, he writes about what we can learn this time around to make this new phase in digital growth usher in smoother to benefit everyone involved. Reach him on Linkedin

July 7, 2015
Best Practices to Prevent Scrum Failure
In my years of experience as a scrum master, I’ve encountered numerous situations where scrum has resulted in less-than-ideal outcomes, a Scrum failure if you will—not because the agile framework is flawed, but because the company didn’t understand, or disregarded, how scrum is supposed to work.

For example, I was once a scrum master for an e-commerce website production team where a new feature was considered so critical that it was prioritized over everything else in the backlog. Minimal time was taken for the development of user stories and my team had to make assumptions because the business stakeholders were unavailable. When we were nearing completion, the stakeholders came out of hiding and identified more business rules that required not only additional work, but some re-work of things that were already completed, including global changes to the company’s customer-facing website. The impact to the business had not been appropriately analyzed, and when the new feature was rolled out, the value delivered was negative, causing refunds and ultimately financial loss.

Sadly, scenarios like these are not uncommon. The scrum framework is ideal for situations where priorities change often, but many companies struggle with finding the right rhythm to implement and manage those changes. Scrum fails when companies don’t build a shared consensus on which principles should form their definition of agile delivery, or fail to educate everyone involved on how the agile processes should work. One very common example involves the most basic of agile artifacts: the sprint plan. I can’t tell you how many times I’ve completed a long day of planning only to run into an executive who wants to add something to the sprint. This is where understanding the methodology and sticking to it are important, especially because changing the sprint plan will impact your team’s morale, as well as its productivity. As much as we love our stakeholders, they should not swoop in at the last minute and bypass the product owner.

For those unfamiliar with scrum terms, the product backlog identifies the longer term priorities for a project or product team. The sprint backlog determines current sprint priorities and sets a shared goal for delivery team execution. Product backlog priorities can (and should) change all the time, but sprint backlog priorities should be locked down for the duration of the sprint. At the beginning of each sprint the team participates in a planning meeting where the highest priority stories are pulled from the product backlog and reviewed. The final output of a planning meeting is a prioritized sprint backlog that represents the plan for the sprint. Once the sprint plan is determined, it should be protected. There are always exceptions, but to ensure scrum works, the scrum master must protect the plan as much as possible.

So, how can you make sure your team moves as fast as possible but can still address changing priorities? Below are a few lessons I’ve learned:
1. Determine the Appropriate Sprint Length
First, and most importantly, determine a sprint length that is appropriate for your organization. I would recommend that companies new to scrum should avoid one week and four week sprints. One week sprints move too fast for new teams and are too heavily burdened with meetings. Four week sprint teams have a hard time staying with the sprint plan, especially new teams who don’t fully understand the impact of changing priorities mid-sprint. I’m not saying that four week sprints never work, but I find that a four week sprint length makes it difficult to keep the team on track. Two week cycles have traditionally been the preferred length for agile enthusiasts. When I’ve worked on project teams where code will not be moved to production at the end of the sprint, a two week sprint duration works well; however, my production teams seem to prefer three week sprints because they can develop new features in the first two weeks and use the last week to do production preparation work. Whatever length is chosen, it needs to work with your product environment. For example, if you’re currently running three week sprints but your scrum master is constantly having to battle to protect the sprint plan, then maybe a two week plan would be more appropriate.
1. Educate Your Stakeholders
Once the organization finds the appropriate sprint length, educating stakeholders is essential. How can we expect stakeholders to work with implementing scrum when we don’t help them understand how it should work? Actually communicating the sprint calendar is a good first step. Stakeholders also need to understand the product owner’s role and how they will assist them in assigning their requests the appropriate priority in the product backlog. This can be quite beneficial as many stakeholders who desire to add to the product backlog don’t realize that other higher priority features may not have been completed yet. Working with the product owner provides an impact analysis for the overall product. Sometimes it’s as easy as saying, “Is this more important than x?” Just that simple question has saved me many times as a scrum master. Sometimes stakeholders just want to get the request on the backlog and don’t mean for everyone to stop work on current priorities. If the request is determined to be the highest priority item at the time, then analysis of the desired behavior can begin. By the time the next sprint planning meeting happens the team will have a much better idea what the request is about and will be more likely to complete the feature with the value envisioned by the requestor.
1. Build Trust
Lastly, and perhaps most importantly, the team must build trust among the stakeholders. This has had a huge impact on my scrum teams and has stopped most emergency requests. Teams build trust by setting realistic sprint goals for themselves and then accomplishing those goals on a consistent basis. If the stakeholders know what is planned will get done, they have much more confidence that their request will be completed in a timely manner. This is not an easy task but can be accomplished by improving sprint planning.

The most effective way to improve sprint planning, and one that I find is often ignored, is utilizing the sprint retrospective. This is where the entire team reviews their performance during the sprint and discusses what went well and should continue and what areas need improvement. The team then decides what one or two things will make the most impact on the next sprint and brainstorm for ways to improve. The retrospective works well because the team decides what to change and begins to build a culture of continuous improvement. As far as agile benefits go, this is one of the most impactful to organizations.

Another way to build trust is to communicate changes to the plan early and honestly. Let stakeholders know why there was an adjustment to the plan. Was it due to a bad estimate? A delayed approval? Changing priorities mid-sprint? Once stakeholders start seeing the impact they have on the progress, they can assist the team in moving faster too.

The Result

When properly implemented, scrum can realize amazing benefits, but scrum teams and stakeholders have to work together to understand and implement the scrum procedures.

Here’s another true story: When I was a scrum master for another website project team, we had to halt development based on a third party delay. Because we were using scrum, this was no problem—we closed out the last sprint, complete with a proper demo, and the next day held our sprint planning meeting. We started a completely different project that was estimated to take approximately three months, matching the estimated delay for our primary project. We worked with a new product owner and successfully implemented the new feature as planned. The revenue generated from the feature was estimated to pay back the development costs in less than a year, and when completed, the team was able to return to the primary project.

That’s how scrum is supposed to work.

About the Author

Kathy Loveless is an AIM Consultant specializing in agile coaching, agile project management, scrum process improvement, and strategic development. She has an impeccable background in effective agile coaching and transformation and is passionate about immaculate communication and delivering results. AIM Consulting’s expertise in application development and product and project management covers the entire software lifecycle, from architecture to optimizing processes that ensure high quality outcomes with the most efficient delivery. To get in touch, tell us about your challenge.
July 6, 2015
How ITIL Can Co-Exist With DevOps
The original ITIL framework definition was published in the 1980s, but contrary to many opinions, it has evolved since then, provided one takes a more flexible view. Fast forward to ITILv3, and the framework has been shifted to the view that IT is a service that needs to support business initiatives and strategy.

In parallel, the “DevOps Movement” has continued to gain momentum, and is now the “culture” at many forward-thinking organizations. So, that begs the question, “Can I have my ITIL framework and still transform my IT Org to embrace DevOps?”

First, as a refresher, let me quickly outline the key components of ITILv3.
1. Service Strategy — The service lifecycle must stay focused on meeting the business goals and objectives.
2. Service Design — Architectural blueprint for the service
3. Service Transition — Outlines the change management and release cycle practices
4. Service Operation — Focuses on the delivery and control processes for the service
5. Service Improvement — Focuses on how to continually improve the processes for Service Delivery
Now let’s lay out an engagement model for the transition of the culture to DevOps.
1. DevOps Strategy — Establish metrics that outline Business Objectives and Value
2. DevOps Design — Establish Capacity, Availability, and Security Best Practices
3. DevOps Transition — Establish Organizational Structure and Define SLAs
4. DevOps Operation — Shift to Agile; Programmatic CMDB and Change
5. DevOps Improvement — Automate Incident, Problem, and Event remediation; Feedback loop
As you can see, there isn’t a huge gap between the two worlds, and I assert that the main changes are cultural, not technical, but one can use technology to start shifting the culture.

This is where the concept of a Security Fabric plays an integral role. Here is an example of how an Enterprise Reference Architecture can be organized to combine the best parts of ITIL and DevOps:
1. Policy Definition — Allows you to define Users, Groups, Hosts, Layers, and Interdependencies
2. Secrets Management — Allows you to store Secrets in a vault and access them programmatically in a secure manner with a complete Audit Log
3. SSH Management — Allows you to securely manage SSH public-keys and session interaction with a complete Audit Log
4. Host Factory — Allows you to instantiate and register new hosts or containers into your infrastructure
5. Identity/Service-Defined Authorization– Allows you to control interaction between your hosts or containers via Policy and Gatekeeper
IT now has a powerful platform to map Business Objectives to Identity, Hosts, and Processes. Once that mapping is established, previously static and brittle activities such as managing Secrets and SSH public-keys, can be moved into the Continuous Delivery pipeline, which maps directly improving Service Transition, Operation, and Continual Improvement.

Agility and Scale are programmatically added via various tools, which allows for frictionless and auditable Change requests, as well as rapid response to Incidents, Events, and Problems.

The diagram below attempts to pictorially show the co-existence of ITIL and DevOps and how the features of a Security Fabric are common between the two.

The shift to a DevOps culture should be approached as a continual evolution. ITIL was a great starting framework, but over time added too much structure and rigidity. You can leverage the Best Practices of DevOps to overcome that and transform into an extremely agile and productive environment.

About the Author/Mike Kail

Mike Kail was Yahoo’s CIO and SVP of Infrastructure, where he led the IT and Data Center functions for the company. He has more than 24 years of IT Operations experience with a focus on highly scalable architectures, prior to joining Yahoo. Most recently, Kail served as VP of IT Operations at Netflix. Prior to that, he was VP of IT Operations at Attensity, where he was responsible for the Americas data center operations team; including managing various big data systems with their Hadoop cluster, HBase, and MongoDB components. He has been recognized widely for his insightful industry commentary on Twitter, and was recently named by the Huffington Post as one of the “Top 100 Most Social CIOs on Twitter.” He holds a B.S. in Computer Science from Iowa State University. Reach out to him on Twitter or LinkedIn.
June 24, 2015
Bimodal IT and Remodeling Traditional IT for Greater Agility
Bimodal IT is a Gartner concept created to help CIO’s understand that IT organizations need to support both traditional and agile modes of IT solution delivery and operation in order to make wise choices about infrastructure, process, people, and tools. Much attention has been given and rightfully so to the disruptive and innovative “Mode 2” teams building on the practices established by web-scale giants like Amazon, and Google. However, in enterprises there still exists a sprawling majority of “Mode 1”, traditional, industrialized applications run by teams that must focus on risk aversion, security and compliance. While these will likely not match the innovative velocity of agile IT teams and applications, traditional IT teams are still under pressure to get faster, more agile, innovative, productive and cost-efficient. Yet, the Mode 2 playbook can’t be applied wholesale to Mode 1 transformation work because it often assumes a greenfield situation—kind of like a new home construction. By contrast, Mode 1 modernization is more like renovating an existing home that must proceed in steps that address the present realities of applications, legacy infrastructure, business mandates, organizational interactions, and technical skills.

Understanding Bimodal IT

The Bimodal IT concepts is meant to distinguish between applications and teams that are traditional in their focus, and those that are agile. In her blog post on Bimodal IT, Gartner analyst Lydia Leong explains Bimodal IT thusly:

“Traditional IT is focused on “doing IT right”, with a strong emphasis on efficiency and safety, approval-based governance and price-for-performance. Agile IT is focused on “doing IT fast”, supporting prototyping and iterative development, rapid delivery, continuous and process-based governance, and value to the business (being business-centric and close to the customer).”

The reason that Gartner promotes the idea of Bimodal IT to CIO’s is to help them see that it can be dangerous trying to optimize processes, human resources, infrastructure, and tools to cover such different modes, and as a result, optimizing to neither. Rather, the right move according Gartner is to allow agile teams room to develop a brand new set of capabilities that can then help transform the whole organization. Lydia continues in her blog:

“We’ve found that organizations are most successful when they have two modes of IT — with different people, processes, and tools supporting each. You can make traditional IT more agile — but you cannot simply add a little agility to it to get full-on agile IT. Rather, that requires fundamental transformation. At some point in time, the agile IT mode becomes strategic and begins to modernize and transform the rest of IT, but it’s actually good to allow the agile-mode team to discover transformative new approaches without being burdened by the existing legacy.”

The Contrast Between Agile and Traditional IT

Traditional IT and Agile IT are different in a number of ways. Perhaps an effective way to explain the difference is to picture what these applications and teams look like.

The classic agile IT initiative in the enterprise is around a mobile, e-commerce or web-centric application that is aimed at moving into new markets, improving engagement with new customer segments, or disrupting a current market. The application is fully virtualized, and the preferred infrastructure is a public cloud such as AWS. The team is conceived as a unified, DevOps team of line of business product managers, developers, testers, and operations folks working collaboratively on an agile development methodology. Automation is built deeply into the process, with infrastructure as code tools used to manage public cloud virtual machines and related infrastructure components via well-documented RESTful API’s, plus other DevOps toolchain elements automating and managing most processes, including continuous integration, continuous delivery and continuous deployment of code into production.

Traditional IT stands in pretty stark contrast. Let’s consider an industrialized, mission-critical application that has evolved slowly for years. The application (which is usually composed of anywhere from 5 to 10 or more components) is likely running on a mix of legacy (think RISC UNIX), dedicated (x86-based) and virtualized servers (VMWare and perhaps Hyper-V) running in a private or hosted private datacenter, using a mix of traditional and software-defined storage and connected by Ethernet and Fibrechannel switches. The application components themselves may be legacy code, making them difficult to quickly port to more modern infrastructure. For example, I was talking recently with a colleague who told me about his sister who still writes code in Fortran for a major bank. With such a large, complex, long-lived and mission-critical application suite, the IT organization maintaining it and the developers creating updates and upgrades are highly concerned with quality, reliability and predictable performance. The nature of the overall team working on an industrialized application is a far cry from the sort of “two pizza” DevOps teams on the agile IT side. Rather, it’s more of a loose consortium of teams such as developers, testers, security and compliance, plus outsourced contractors and vendors. The development process is waterfall-based, managed by an application lifecycle management (ALM) suite of some sort. Automation is scant, and in particular it takes a long time to pull together infrastructure for those dev, test and other teams to perform their jobs. Despite that, a production-like environment for testing, security, compliance, etc. is considered necessary to minimize downtime and other quality risks. As a result, release cycles can stretch into many months.

Modernizing Traditional IT

When my wife and I remodeled our 1908 Craftsman-style house about 15 years ago, we discovered that an earlier renovation had actually cut out a third of a key supporting beam, leaving one part of the house literally hanging on a beam that was only attached to one side of the house. Needless to say, we were a bit alarmed to find that out. When dealing with mission-critical applications and infrastructure, you can’t be careless. As mentioned earlier, simply taking the playbook for a greenfield, agile IT application and trying to apply wholesale it to a legacy, industrialized application usually won’t work. Making traditional IT more agile requires a step-wise renovation—a risk-mitigated path that leads towards agile and DevOps practices, but without a great leap into the dark. The steps need to establish the building blocks to agile/DevOps practice without disrupting the application, without introducing downtime, and while delivering solid returns along the way. Successful legacy application and service ‘remodeling’ efforts observed in highly traditional organizations such as telecommunications and cable operators provides good ideas on how to take those steps:
- Identify or Procure Automation Skills: The initial steps to remodeling Traditional IT are about incrementally improving current practice with automation and at the same time making it ready for agile and DevOps. This means that to be successful, you need to get some automation skills into the mix. A team that can’t take hold of its own destiny in modernizing a Traditional IT application is not going to have a sustainable change, and that means identifying or procuring some automation expertise and protecting that person’s time to focus on the renovation rather than on firefighting.
- Private Cloud IaaS: One of the most serious obstacles to agility is the painful amount of time and effort it takes to allocate and provision infrastructure for developers, testers, etc. and the lack of standardized, consistent infrastructure environments, due to manual processes. This is exactly the issue that drives some development teams to go shadow IT and start developing on AWS. Often times, these development teams then discover that integrating that software into the production environment is plagued with translation issues. Nonetheless, the first order of business then is for the infrastructure team to establish IaaS that can serve up full production-like infrastructure environments to devs, testers and other constituents. This isn’t trivial when you’re dealing with a combination of legacy servers, bare metal x86, VMs, and perhaps networking, security and other devices.
The IaaS delivers a number of key improvements. First of all, it speeds up access because developers, security, compliance and other personnel can now get access to infrastructure in the space of minutes rather than the space of days or weeks. More than just speed, IaaS enables asynchronous productivity by various teams to happen 24×7, so that quality progresses in an even fashion, unimpeded by office hours. This is especially helpful if there are global teams, contractors or vendors involved. IaaS also provides standardization and consistency across all efforts. Finally, IaaS saves costs. Significant amounts of operations personnel time that was devoted to painstaking manual infrastructure assembly can be repurposed to higher value activities, and infrastructure utilization rates can easily double or triple almost immediately, which reduces the need to purchase more capital equipment.

A word here about GUI’s. One of the key notes of dissonance that often occurs is when people hear developers say that they don’t want GUI’s, they want API’s. That in my view is absolutely valid, but more so for Mode 2 Agile IT teams. It turns out that IaaS GUI portals can be rather helpful in providing an easily understood reference point to multiple teams that may have highly varying understandings of the end-to-end picture of the application and its infrastructure. For example, if an IaaS self-service portal provides a full visualization of the end-to-end infrastructure involved, a developer who might assume that latency is always near zero might become more aware that there is a WAN in the middle of the end-to-end infrastructure.
- Test as a Service: Many development and testing teams have taken some stab at test automation. However, test automation tends to live and be maintained at the individual level. The natural follow-on to IaaS is to build a standardized regression or certification of automated tests that can be offered from the IaaS portal—test as a service (TaaS). This allows teams to access production like environments, do their development work, then easily launch a comprehensive certification test. Tests naturally should report their results into the ALM tool so that individuals and teams can review results in near real-time to keep the process moving. The benefits are similar to IaaS: increased velocity, standardization, productivity and efficiency.
- DevOps Evolution: With a combination of IaaS and TaaS, the stage is now set to deploy a DevOps toolchain and start doing cross-functional technical collaboration, and agile methodologies. For example, the IaaS and TaaS can be launched by a unified continuous integration server upon software development check-ins from any team. Of course, DevOps doesn’t just mean tools and automation, but the process of coming around collaborative devtest infrastructure and common certification tests to build a common, agile development practice is one of the steps towards a more broadly responsive cycle.
Conclusion:

Traditional IT may not be the sexiest beast in the technology forest, but it’s often mission-critical. Gartner’s Bimodal IT concept encourages CIO’s to let go and let Mode 2 Agile IT teams do their thing to create new culture and practices. Translating that to Traditional IT requires a more step-wise approach. Progressing from manual processes to IaaS to systematized, self-service test automation as the foundation for DevOps evolution is a sound way to proceed. With these steps. Traditional IT can “do IT right” and do IT faster.

About the Author/ Alex Henthorn-Iwane

Alex Henthorn-Iwane joined QualiSystems in February 2013 and is responsible for worldwide marketing and public relations. Prior to joining QualiSystems, Alex was vice president of marketing at Packet Design, Inc., a provider of network management software, and has 20+ years of experience in senior management, marketing, and technical roles at networking and security startups.

Through his roles at QualiSystems, Packet Design, CoSine Communications, Corona Networks and Lucent Technologies he has acquired expertise in cloud computing and the opportunities presented through virtualization. He has written for Embedded Computing, Virtual Strategy Magazine, Datamation, SDN Central, Datacenter Knowledge and InformationWeek.

Connect with him on LinkedIn, Twitter, or SlideShare.
June 24, 2015
Three Ways Dev and Ops Teams Can Manage the Integration Lifecycle
Many enterprise organizations did not invest enough money into integration as they built their cloud or hybrid infrastructures. As if integrating on-premise systems and applications wasn’t tough enough, in today’s cloud era with the rise of hybrid environments, connectivity has only gotten more complicated – especially when combined with the increased speed of deployment. Puppet Labs found that high-performing organizations deploy code 30 times more frequently than their lower-performing counterparts, meaning dev and ops teams are struggling to break down data silos while simultaneously managing constant technology upgrades for line-of-business users.

There’s no doubt that, whether public or private or hybrid, the cloud is now part of most companies’ infrastructures, and its share will only increase. Goldman Sachs estimates that spending on cloud computing infrastructure and platforms is expected to grow at a 30 percent CAGR from 2013 through 2018 compared with 5 percent growth for overall enterprise IT, while Forrester predicts that 25 percent of all software applications purchased will be cloud-based by 2020.

With the speed and agility the cloud imposes on businesses, dev and ops need to come together to look at data integration throughout its entire lifecycle, from developing new apps or services on top of existing apps, all the way to the ongoing service, maintenance and upgrades of those applications. The two teams must join forces to manage this process as one comprehensive integration lifecycle, where dev and ops play equal roles solving for new data integration challenges brought about by hybrid deployments and siloes created by line of business users’ cloud purchasing habits.

Today’s dev and ops teams have to look at data integration as a service, not just a one-off implementation they can hand off. Here are three strategies to help bring the two teams together to jointly manage the data integration lifecycle.
1. Assess Current Integration Challenges Collaboratively
  Without alignment between dev and ops teams, integration lifecycle can’t be managed successfully. Organizations need to align on the people and process portion first, asking questions like: what systems does each business unit need data from? How likely is it that their needs will change, and how can you plan for a flexible integration as a service? What capabilities will ops need after dev implements the integration?
After reaching agreement, teams will need to jointly assess and select a data integration as a service technology or platform that allows for joint requirement setting, collaboration, testing and ongoing adjustments throughout the lifecycle of the applications. Selecting the right integration platform as a service at the onset is critical to minimize cost, time and wasted dev and ops cycles down the line as business and technology needs change. The most agile integration option typically consists of a plug-and-play, flexible platform that makes use of APIs and also allows for deeper customization down the road.

2.Understand the Four Key Stages of Integration Lifecycle Management
Dev and ops teams are now part of the same cycle dealing with frequent updates and agile product development. It’s no longer about developing, testing and then having ops maintaining, but about a seamless and transparent handoff from dev, to ops and back throughout the entire application and data integration lifecycle.

To capture the most value from disparate organizational data and to minimize ongoing data and application synchronization costs, dev and ops teams need to adopt a joint integration management strategy that accounts for the four distinct phases of data and application connectivity. These include:

3. Design phase. After understanding business use cases and selecting a platform, dev and ops must collaborate on priorities – what near-term functionality can they design for based on the plug-and-play layout, and how can they tackle that low-hanging fruit while maintaining the flexibility to meet future requirements?

By starting with the right technology and a common integration framework, dev and ops teams can design connected core systems and data early on with plug-and-play API connectivity for solutions like CRM and ERP, while maintaining the ability to easily expand functionality with custom/deeper integration capabilities with marketing automation, business intelligence and other apps.

In addition to functional capabilities, the design phase should address performance and scalability by taking into account throughput and latency. This process involves planning to meet performance expectations, even with the customization and extensions users will likely want down the road. Dev and ops should ask themselves questions like: which company systems will be on premise, in the cloud or hybrid? What are the performance, scalability and security tradeoffs of each approach?
1. Develop phase. Dev and ops need to work together in the development phase to carefully document key aspects of the integration while building connectivity for specific use cases. This documentation will be critical to the seamless handoff between the two departments, so they both need to be involved in the process. Strategic collaboration and thorough documentation can improve the efficacy of maintenance and debugging activities, help companies adjust to staffing changes and more easily transfer integrations to and from the two teams during the ongoing development and roll-out process.
2. Deploy phase. Given that you’ll likely deploy in a hybrid environment, make sure you test both on-premise and cloud integrations to ensure that data is flowing freely. To help test integrations and move them into a production environment, look into staging and production environment tools that streamline the process. Infrastructure costs should also be considered, as they may increase unexpectedly if you don’t watch them throughout the deploy phase.
3. Run phase. Continuously monitor to confirm that the data is flowing as designed. If dev and ops selected the right technology platform, the teams won’t have to worry about ongoing API management, because the platform takes care of it.
The right platform will also lower downtime and maintenance costs with a multi-tenant management environment that allows the expert for any given situation to dive into the integration backend or runtime. The platform should also provide ongoing diagnostic reports and proactive alerting to allow ops to home in on potential performance issues.
1. Build a Responsive IT model
  With dev and ops coming together to take advantage of the speed and agility offered by the cloud, the resulting just-in-time services for businesses are turning IT from a cost center into an innovation center. The data integration lifecycle is another result of the new cloud-shaped world, helping dev and ops teams create a more responsive IT model and allowing them to better meet business needs while driving innovation. While shared environments bring with them an additional element of risk, a responsive IT model is an important step towards both short-term and long-term success in the modern connected enterprise.
These three steps allow dev and ops teams to offer the value, speed, efficiency and opportunity that today’s enterprises require. By adding collaboration and agility at every stage of the integration process to plan, implement, track, debug and update continually, dev and ops teams can scale to respond to the data-driven needs of the business more precisely.

How have you overcome cloud and hybrid application and data connectivity challenges in your organization? Connect with me in the comments below, or on Twitter at @scribesoft.

About the Author/John Joseph

John Joseph, VP of marketing for Scribe Software, earned a master’s degree in business administration from the MIT Sloan School of Management, a master’s degree in electrical engineering from the University of Southern California and a bachelor’s degree in electrical engineering from MIT.
June 23, 2015