Troubleshooting “silent failures” in an OpenStack cloud environment has been a tough problem to solve for many production users. eBay’s chief engineer of cloud, Subbu Allamaraju, whose team has been operating PayPal on an OpenStack-based private cloud since March 2015, has publicly defined one of his ongoing challenges as: “Too much noise, too many alerts, and still working on how to detect troubles before customers find out.”
For enterprises and service providers in various stages of rolling out OpenStack, tackling silent failures invariably starts with two questions:
Why and where do silent failures occur?
Can my Ops team receive push notifications of warnings earlier and proactively remediate them before silent failures materialize?
Silent failures happen because OpenStack Ops teams often don’t know which service domains to tune into for early-stage warnings. Even if they do, the warnings get drowned out by noise. For instance, your metrics and logs for the compute domain show a successful VM creation. Yet, a tenant is still not able to use her requested VM instance – a silent failure – while the warnings actually show up in metrics and logs outside compute domain, such as DHCP agent failure and identity database query timeout.
To catch a situation like this, OpenStack Ops teams need a real-time, data-driven approach that automatically detects early warnings in DHCP agent and identity service, then correlates them with VM creation, and finally push-notifies only the relevant domain experts to collaboratively remediate “tenant can’t use requested VM.” All noise has been removed and related warning signals have been clustered and contextualized together – automatically.
And Operators shouldn’t have to rely on writing rules to model silent failures. Writing rules will be too time consuming and are unable to cover all failure scenarios. There are too many service domains – at least 13 of them (e.g. compute, identity, dashboard and storage) – plus their service daemons and virtual switch tunnels that stitch them all together.
Using real-time, data driven management to resolve OpenStack silent failures has been a recent success for OpenStack. For example, a large OpenStack production shop was able to proactively detect OpenStack cell failures by receiving early notifications from outside the cell domain – CEPH and JBOD I/O failures.
By becoming data-driven and ingesting all events and status messages across the entire OpenStack into real-time, machine learning algorithms, OpenStack Ops team automatically receive cleaned and contextualized alerts, then use modern social collaboration technologies to rapidly troubleshoot issues, capturing tribal knowledge for the future. The new mantra for these teams should be “clean, contextualize and collaborate,” the 3 pillars of real-time situation management.
But real-time situation management doesn’t just stop here. It scales across the entire ecosystem that lives in and around OpenStack. Data sources include the ever-pervasive RabbitMQ message bus across all services, metrics and logs across the diverse tenant operating systems, and metrics and logs across all technology and vendors. In essence, monitoring the entire technology and vendor independent reference architecture can be rolled up and simplified as “situation management” vs. reacting to many seemingly unrelated “domain silo alerts.”
Real-time situation management can also sit above multiple OpenStack instances for load sharing and DR purposes, and the interactions with public cloud. For instance, Amazon CloudWatch can be quickly incorporated as a data feed, giving OpenStack Ops teams a “manager of managers” view across OpenStack hybrid clouds.
Real-time situation management essentially provides Ops team with a single pane of glass to see and resolve failures earlier, with amplified signals and suppressed noise. Doing so without relying on extensive rules offers much needed flexibility for Ops team to keep up with so many “unknown unknowns,” as the OpenStack community continues to evolve its entire reference architecture and ecosystem.
About the Author/Feng Meng
Feng Meng is a Senior Director, Product Marketing at Moogsoft. His expertise lies in IT service assurance, with 15-year technical marketing background in Application Performance Management (Bytemobile, acquired by Citrix), Network Performance Management (Cisco), Infrastructure Monitoring (VSS Monitoring), and Server Management (HP). Twitter: @fengmengmoog@moogsoft Linkedin: https://www.linkedin.com/in/fengmeng1970
News from the AWS Summit in NYC today is that Chef is now available as a one click install on AWS via the AWS marketplace. While the ability to use Chef on AWS is not new (it is the leading configuration management tool on AWS), being on the marketplace makes both install and billing a one click deal. This should make it really simple to get Chef installed and running for anyone looking to leverage the many benefits Chef brings to configuration management and automation. Literally with one click install what could be simpler?
As part of this new arrangement not only will Chef on the AWS marketplace be available, but it will also be a featured offering. Chef and AWS already enjoyed a cozy relationship (they are both based in Seattle and there is a cross-pollination of personnel). This deal takes the relationship to a higher level. Both Chef and AWS are dedicating significant resources to marketing, sales and support of this initiative.
I know from speaking to AWS in the past and other marketplace vendors that AWS will often let partners list on the marketplace, but they don’t often then dedicate resources to roll out, marketing and support. Further, as far as I know Chef is the only configuration management/automation solution that is available via the marketplace like this. I don’t know if this will remain the case or for how long, but for now it is somewhat of an exclusive relationship.
I had a chance to catch up with Ken Cheney, VP of Business Development and Jay Wampold, VP of Marketing at Chef about this expanded relationship the other day. They both emphasized that this is much more than just listing Chef in the AWS marketplace. Some estimates I have seen have Chef deployed on something like 70% of AWS deployments. Ken and Jay believe that this number will continue to remain high and may even go higher as a result of the marketplace presence and the push from both Chef and AWS.
On top of their previously released expanded relationship with Microsoft via Azure, it seems Chef is putting a stranglehold on two of the largest cloud platform providers in the world. That doesn’t mean that other configuration managers don’t have presence on AWS or Azure, but Chef seems poised to be in a dominant position on these platforms.
As we see more enterprises continue to migrate more and more infrastructure to the cloud this presence will become more and more important. For now you can get Chef fast on the AWS marketplace.
In the past five years business cycles have greatly accelerated, forcing companies to respond faster to customer needs and competitive demands. This trend has been driven by business adoption of consumer-oriented technologies like social media, smart phones, b2c internet services. We all expect to have a constant stream of new services and capabilities delivered to us quickly and with little disruption.
There is currently intense competition across all industries, creating a digital “survival of the fittest” business environment. If a business function can be enhanced by a digital solution, it will be. Most enterprises are now dependent on the rapid implementation of new digital technology and the software solutions to remain competitive.
This extraordinary change has been achieved in part through adoption of Agile development, which promotes close collaboration between business users and developers. In Agile, environments, software development is broken into well-defined intervals with the goal of releasing software quickly, generally within less than a month. Updates are ongoing and nearly continuous, allowing product development teams to pare delivery time by 50 percent or more.
Adobe is a good example of this new way of doing things. After deploying CloudMunch DevOps platform Adobe has increased the pace of application development by as much as 60 percent.
Already, companies like Twitter, GameStop and Etsy use Agile processes as part of DevOps for continuous innovation. In fact, in a recent IPO prospectus, Etsy noted it updates code every 20 minutes, or 70 times per day. In 2014, the company executed 10,000 code “deploys”—a number that was inconceivable just a few years ago.
However, there is a disconnect between traditional IT processes and new Agilde and DevOps delivery processes. Here are three barriers to integrating Agile and DevOps in the enterprise:
Traditional operational IT processes are much longer than intervals used in Agile. While IT is often planned and measured in quarters on an annual budget cycle, Agile development may happen on a two-week sprint. Once built, a business group is eager to use these new capabilities, but most IT organizations aren’t prepared to deploy new capabilities that quickly.
The supporting software ecosystems are very complex, since applications are built by standing on the shoulders of giants. The comprehensive features supporting software allow development teams to focus on the desired business capabilities rather than low-level utility components. Since all of these components are updated frequently, the process of tracking and deploying the right supporting ecosystem presents a complex problem.
Agile teams have autonomy in the use of development tooling, which results in a number of different and disconnected development tools being used. This also makes it difficult to implement these processes and provide the metrics that provide a consistent view of the enterprise portfolio of development projects.
Using modern tools for success in today’s hybrid cloud
To support rapid implementation of infrastructure needed to support the Agile delivery process, enterprises are increasingly moving more of their essential business operations to the cloud. For example, Adobe has embraced CloudMunch, a DevOps platform to automate delivery of software updates to dozens of its products, pushing multiple updates more quickly and easily to its customers.
Today, most companies have a hybrid IT infrastructure, comprised of both on-premise and cloud systems. While they have moved many functions to the cloud to take advantage of the benefits of cloud computing, they still keep many systems in-house. In the past, they’ve used development tools like Chef and Jenkins to manage application development and infrastructure automation, but these aren’t enough in today’s hybrid environment. Increasingly, businesses need to plug in and integrate their developers’ chosen tools into a single workspace to apply Agile development to a DevOps environment with a single dashboard. This will be critical if enterprises need to do Agile and DevOps at scale
This ability to manage Agile development in a hybrid cloud environment will become even more important in the next decade. Already, new platforms have emerged, allowing development teams to better orchestrate multiple toolsets & processes in a single “pane of glass” to streamline collaboration and application delivery. To stay competitive, businesses should assess their development infrastructure to ensure they don’t get left behind as technology progresses.
The impact of these accelerating changes in virtually every industry is astounding, affecting every facet of an organization, from company culture to management styles and business infrastructure. This is driven by changing customer expectations — breakneck speed and complexity have become the new normal. Companies are being forced to seek new development methods and leverage the cloud to create software that has the capacity to continuously adapt to ever-changing user needs.
These new approaches mean companies need to intensify collaboration and address the disconnect that exists between the old and new worlds. Successful integration of DevOps and Agile development will play a key role in the success of companies in today’s digital economy. In the new landscape, it’s about rapid, continuous high quality software delivery for the digital world, and companies that can do this at enterprise scale will outpace those that struggle to adapt.
About the Author/Pradeep Prabhu
Pradeep Prabhu, Co-founder and CEO CloudMunch, Previously Pradeep was Vice President and Head of SaaS for Infosys, where in his 18-year career he was instrumental in growing the tech company to a multi-billion dollar business. Before pioneering Infosys’ SaaS business, Pradeep incubated the Infosys-Microsoft solutions and alliance program and the Microsoft Technology Center. His previous leadership roles include establishing the Infosys Seattle office and delivering many successful software programs for marquee clients like Amazon, Boeing and Microsoft and conceptualizing and launching the internet banking product eFinacle, which today is installed in banks around the globe. Follow on twitter @cloudmunch
Open source and DevOps go together like chocolate and peanut butter, or like milk and cookies (these metaphors are making me hungry). Basically, they’re inseparable and—while each is good in and of itself—they’re better when united. HashiCorp announced today that Atlas, a platform for a variety of leading open source tools, is now available to the general public to help organizations make DevOps even better.
HashiCorp has a number of popular, industry-leading open source tools at its disposal. Vagrant manages development environments. Packer automates building artifacts like Docker containers, OpenStack images, and VMware images. Consul is a tool for runtime orchestration. Serf is a protocol used by Consul for failure detection and message distribution in a server cluster. Terraform automates infrastructure provisioning on cloud platforms like Microsoft Azure and Amazon Web Services. They’re all great tools in and of themselves, but Atlas ties them all together into a powerful, cohesive whole.
Atlas has been available in beta for a while with tremendous success. Nodes under management consistently doubled month-to-month during the beta, and heavy hitters like Mozilla and Cisco jumped on board to test it out.
DevOps seeks to accelerate development and deployment. One of the primary attributes of DevOps is automation to facilitate automatic, error-free, auditable implementation of routine, repeatable functions. Many organizations strive to embrace DevOps tools and principles and leverage open source tools like Consul or Vagrant but use manual, time-consuming, error-prone processes that are difficult to scale and challenging to manage effectively.
HashiCorp believes that Atlas is the solution—a platform that brings it all together and enables organizations to manage end-to-end from configuration management to container deployments across multiple cloud environments. Each of the open source tools is fine alone, but the ability to manage and orchestrate them as a whole is very powerful.
“We initially rolled our own solution for infrastructure CI and deployment. As the team grew, our homebrew solution struggled to centrally manage configuration, Terraform state, and access control,” said Chris Lonnen, Internal Tools and Systems Architect at Mozilla. “Atlas remedies all of these challenges for us and provides a beautiful interface for audit logs, infrastructure history, system-wide monitoring, and more. We’ve never lost 9s betting on a HashiCorp product.”
According to the HashiCorp press release, you can start using the hosted version of Atlas as of today, but on-premise deployments of Atlas won’t happen until late 2015. If you want more information about on-premise Atlas you can sign up at this link: https://atlas.hashicorp.com. The development features like Vagrant box builds and hosting and Packer builds and artifact storage are free, and the first 10 nodes are free as well. Pricing is $40 per node per month beyond the first 10 nodes.
You are confused. But I am confused as well. For approximately two years now this DevOps thing has been both exciting and frustrating. And in ways that are totally different for each of us. At the intersection of all the disparate confusion is the “perfect storm”. What that means for now is that we are all hunkered down in the basement, reading books on “DevOps” to pass the time, and better understand the thing that is attempting to destroy us.
In the application economy, you have to deliver software as if your business depends on it … because it does! It’s a bold statement, but true. Think of nearly every interaction a person might have today—be it for work, commerce or play. Most have a digital dimension, which relies on digital technology and platforms.
In order to survive and grow, every company needs to become a technology company and every business needs to become a digital business. Successful transformation requires Continuous Delivery, the new business imperative that enables you to rapidly develop and deliver applications that drive superior user experiences and engage your customers and staff.
But the traditional “software factory” or process for transforming an idea into a customer experience is throttled by a number of bottlenecks in the delivery pipeline. That means delivering innovative, high-quality applications, faster and more frequently can be a chaotic and complex process, particularly if your application delivery systems and processes were designed to only push out one or two releases a year.
Join moderator Alan Shimel, DevOps.com; and experts Georg von Sperling, DevOps and Technology Evangelist at CA Technologies and Julie Craig, Research Director at Enterprise Management Associates (EMA) on July 21, 2015 to discuss ways to overcome today’s application economy challenges with solutions designed to transform your software delivery lifecycle into a continuous delivery pipeline of digital innovation.
Recorded: Tuesday, July 21, 2015
Recording
Q&A
1. Based on your experience tell me : continuous delivery versus traditional development. The continuous delivery model is perfect for every business ?
GvS: As a goal, yes, absolutely. Because continuous delivery practices enhance what is already being done and simply promote good behavior over creation of waste. Removing bottlenecks to increase quality, frequency and velocity simply makes sense, regardless of the business you are in. Now, that said, continuous delivery is not necessarily perfect for every business if you define it as continuously delivering to production. But what development organization would shun away from getting early and amplified feedback in User Acceptance Test (UAT)? To accomplish this, you still require every one of the concepts in continuous delivery.
ANBU: If you think of achieving Continuous delivery as a means to achieve modularity & lose-coupling in order to achieve higher workforce productivity, every business will be motivated to do it. As a concept, it is perfect for every business, but the objectives of the process and their order of their importance will vary industry to industry. As long as you make these adjustments, you will benefit a lot.
2. One of our struggles is the impact of Continuous Delivery on our Environment Strategy and the Mechanics used to maintain them. For example, applications dependent on other applications / components will require those other applications / components in the environment infrastructure in which testing is occurring. How do you see Compnay Environment Strategies changing to support Continuous Delivery? When does it become necessary to change an application’s architecture to reduce or remove these dependencies? Or are there testing methodologies that can accurately test changes without the need for large QA Environments?
GvS: The path with the least resistance is seemingly learned from the airplane manufacturing industry. When there is a new wing design that is required to undergo quality validation, the dependent systems and datasets are so complex, they could not have ever released those designs if they threw up planes into the air with different passenger or luggage loads (data) in different weather conditions (environment) attached to a full airplane body (dependencies). Likely, this would have resulted in the same adverse effects that we experience when pushing a digital asset from our IDE to the first environment – often catastrophic. Virtualizing dependent services with actual behavior – just like in a wind tunnel – is a good approach. This also allows discovery of the dependencies in consumable, shareable format which, should you find additional needs to revisit the architecture, actually aid in the process of redesign.
ANBU: I would recommend a model-based approach in your situation. In your stated example, if you model ‘infra-as-code’ and try to achieve modularity at that level, so that developers and testers call it out during their plan/design/development process to make their ‘availability and use’ defined, the problem can be solved. It requires an assessment on all the functional components (infra included) in order to model it. There are tools out there that would help you achieve such a modeling.
3. How do you measure the success of DevOps other than faster & quality releases? are there any metrics that can be collected periodically?
GvS: In the context of Continuous Delivery, there are many metrics that can and should be captured. Many more in the context of DevOps. We could likely devote an entire session on this. Let me focus, however, on continuous delivery metrics. On top of focusing on business-oriented outcomes, much like in DevOps, some operational metrics could be captured. Examples are:
– Lead time to production. Measuring cycle time from the work starts to the time the team’s definition of ‘done’.
– Defect resolution time. Really successful teams can fix defects as quickly as they are discovered. The more left-shifted the defect discovery is, the more quickly they are resolved.
– Production defect leakage. Measure both: new defects found and defects previously found and still made it to production.
– Defects. It is bad enough to ship buggy apps. Continuously doing so is even more terrible. Measure at each stage in the cycle, and notice a left-shift if CD is going well.
– Broken build time. The longer a build sits around broken, the commitment to quality is just not there.
– Production downtime during deployment. If you achieve zero-downtime, you can stop measuring here.
– Regression test duration. Teams with manual practices will measure in months or weeks. CD teams will likely measure in minutes or hours. Also include lead-time to regression test.
– Performance test outcomes. If you find tons of functional bugs in performance testing, there is likely a bottleneck not addressed early in the cycle.
– Touches. Number of tickets, emails or other communication required to do a deployment in each stage. If it isn’t “zero-touch”, there is improvement potential.
This is not a complete list, but certainly one to start with
ANBU: GVS’s response is complete.
4. How do you get legacy code/systems into the CD model? Or is CD only for new code/projects/systems?
GvS: The practices advocated with CD aren’t specific to the tenure of the project or the age of the technology. They are arguably all good practices and make perfect sense. Now, if you are an enterprise on the journey to adopt CD practices, the recommendation is certainly to begin with projects that have the highest business impact. A good heatmap analysis does go a long way to begin. If it turns out that it is your legacy code/system, so be it. Parallelize work streams, accelerate quality and automate & orchestrate the delivery pipe.
ANBU: Legacy code/systems are generally monolithic/tightly coupled. Hence the problem. Unfortunately, they are also part of the corporate string of applications, so we can’t stay away from them. We have seen examples where these systems stay as one ‘component/workstream’ and plug into the larger CD stream, creating a hybrid-model. Yes it is not perfect, but will help you implement CD. Additionally, the maintenance/re-engineering of these legacy systems are opportunities to change the code to adopt to the larger CD strategy that you put in place. If they are truly legacy, there won’t be major ‘patch’ work so they stay what they are.
5. How do you reconcile product management with CD, i.e., the definition of a story to encapsulate a deployable or atomic element that can be delivered? Often times stories are part of a larger epic that needs to be delivered as a whole and not on their own. so the issue is deploying atomic units/stories creates the fear of delivering partial functionality. How does a business deal with this and overcome this
GvS: If you pardon me, but it seems a misinterpretation of CD itself. Deployment and delivery are two, if not three, different things. I believe you may have latched onto the “live” deployment of an atomic change to production. Given the right strategy in modeling the pipeline itself, all CD elements are still valid, the atomic element may go “live” with many other atomic elements. BUT, unlike in the past, the fear of failure in that deployment is minimized as quality drastically improved and, the same, already tested, processes are utilized to set the atomic unit “live” as have been validated in lower environments. It is a journey.
ANBU: one of the benefits of CD is achieving the readiness of production deployment of (ALL or SELECT atomic units) on a push of a button. It does not force you (but you have the choice) to deploy every atomic unit into production If what you said “larger epic as a whole or none” is important, then your definition of the ‘select’ atomic units should correlate to it. Configuration mgt and release mgt should have proper ‘gates’ to make sure that they roll-out properly.
6. My question on metrics is because in legacy orgs it takes a lot more time & energy to get to CD, so measurement through metrics is critical.
GvS: Absolutely, metrics and measurement go hand-in-hand. Looking at CALMS (Culture, Automation, Lean, Measurement and Sharing) for guidance, influencing the organizational culture through measurement and sharing which both are derived from automation is the nirvana. Along with the will of the IT organization, there also has to be executive buy-in to the strategy. Yes, grass-roots is fine, but a lot can get accomplished if the company stands behind the strategy. Having defined positive business outcome metrics as part of that journey helps reinforce that postion.
ANBU: Agree with GVS.
7. What is the Quality as a result of continuous delivery? What is the tradeoff with respect to the Business impact to defect leakages in production?
GvS: Quality metrics should show drastic improvement if CD tenants are implemented. There was a question on metrics already. This is not about continuously deploying bad code, which really just makes things worse, but about continuously delivering, where quality is certainly a determining factor.
ANBU: You need to baseline ‘quality’ before and after CD. There is an absolutely no way, these measures should negatively trend, if you properly implement CD. Defect leakage in any model will have the same implications, just that in CD – the mean time to resolve (MTTR) these issues should be faster.
8. For a business to adopt a CD model, doesn’t all disciplines/functions have to change to suit the atomic delivery concepts.
GvS: Once nirvana is achieved, likely there was a change in all disciplines and functions. It is a journey that must begin. The imperative is there for the business. But, much discussed in the context of DevOps as well, CD is a journey of transformation – not something you do in a day.
ANBU: YES. this is a life-cycle process, hence the impact is for everybody. But you can start from one unit/organization and spread it out to evolve it. if you have the luxury, start a new project/unit with CD as the foundation.
9. How do you see ITIL and DevOps interacting to maximize the benefits of continues delivery?
GvS: There is a great amount of synergy between the two. And actually, how much easier would it be to have “Standard Changes” in the change management process, enrich a CMDB with actual data that isn’t stale, have service models up to date and have focused problem management.
ANBU: A well defined devops strategy can help you implement ITIL more effectively. Configuration management and release management are two particular areas where the ITIL can provide the maximum benefits to the CD strategy.
10. Jez Humble and David Farley suggest not using feature branching, because this incourages back room channel development, and to only use the master branch to do CD, but other companies in this space says feature branching is ideal because git-flow encourages this, is there a prefered strategy in your branching strategy to do CD. Is there a need to better define what is meant by ‘continuous delivery’? I am concerned with people stating that CD means code gets in untested (which is from my understanding an anti definition) and fully tested code is a part of CD.
GvS: I believe Jez’ argument, and I agree, is why it is essential to have a deployment pipeline that is triggered off trunk to ensure all necessary validations, including functional testing. A deployment pipeline is a pattern that organizations adopting CD use. The key is that the validations should be performed against a build off mainline. I also advocate a number of good approaches that help in the validation effort, such as automating the generation of functional tests and having the right data sets available for relevant validations.
ANBU: No comments.
11. Does continue delivery means continuous Deployment to production? For Horses, frequency of deployment is not too frequent but they would still need to have Continue delivery to achieve benefits of Test Automation, Agile Methodology etc.
GvS: I would argue no. Continuous delivery does not equate continuous deployment to production. If I may be so bold, I would state continuous delivery is a practice to allow deployment to production at any time with a high degree of quality and a low amount of risk with zero downtime and zero touch. A business leader recently said to me that whereas, in the past, “deployment to production was an IT decision, it is now a business decision at a push of a button”. But, CD certainly played a part in this paradigm shift, as the entire delivery process, including testing, is visible and predictable.
ANBU: Continuous Production deployment is not the ultimate goal of CD. It is to let you manage multiple workstreams in parallel as atomic units, accelerate quality through high degree of automation and provide an ability to learn/continuously improve the processes. By means of this, you make your changes ready to be deployed faster than you normally do. You should aim to achieve the frictionless automation across the life-cycle, not just in QA activities.
Your Host
Alan Shimel, Editor-in-Chief DevOps.com, An often-cited personality in the security and technology community and a sought-after speaker at industry and government events, Alan has helped build several successful technology companies by combining a strong business background with a deep knowledge of technology.
About the Panelists
Julie Craig, Research Director at Enterprise Management Associates (EMA)
At EMA, Julie’s focus areas are Application Management, public and hybrid Cloud, Integration Technologies, DevOps/Continuous Delivery, and Application Performance (APM). Julie has over 20 years of deep and broad experience in software engineering, IT infrastructure and integration engineering, and enterprise management. Her experience in commercial software companies included development of communications interfaces and management of programming teams. Georg von Sperling, DevOps and Technology Evangelist at CA Technologies
Helping organizations adopt Continuous Delivery without creating more technical debt, Georg is a practitioner, an evangelist and a general geek on software development methodologies, technology, patterns, systems and delivery practices. He advises, helps customers visualize, plan and implement transformational approaches to delivering digital assets from idea to production for medium to large enterprise organizations.
Anbu G. Muppidathi, Vice President at Cognizant Technology Solutions
Anbu has over 25 years of experience in the IT industry and is with Cognizant for the past 20 years. Anbu’s customer management responsibilities include advising the C-suite on breakthrough thinking, competitiveness, strategy, operations and transformation. His business management responsibilities include strategizing on QE&A business unit’s growth, managing the customer relationship, delivery and business operations for the customer engagements in Americas. Prior to QE&A, Anbu was with Cognizant’s Banking & Financial Services vertical for more than a decade managing a large portfolio of North American customers.
In order to meet the many challenges of producing loads of different app/server projects large and small in extremely rapid succession, Appster collaborates on, automates, and dashboards everything it can. “Unlike traditional product companies (i.e. one product with lots of developers) or corporate IT shops (i.e. silos of products and services that rarely change), Appster has to handle a wide spectrum of software projects across a myriad of industries and vastly different client profiles,” says Martin Halford, CTO, Appster. Many of the projects Appster works on are mobile apps.
To do this, the DevOps tools and processes must tackle each new and unique development project in a continuous project stream with the same fervor that any large development project requires. “Our DevOps focuses on automating testing, builds, and deployments for highly custom, highly complex app/server software development projects,” says Halford.
The tools that help Appster to accomplish this include the XCode IDE, the Eclipse IDE, Atlassian’s BitBucket, SourceTree (for source control), and Jira (for Agile project and issue tracking), Jenkins for build automation, Amazon Web Services, and Klipfolio for dash-boarding, lists Halford. “And due to the short turn time and short duration of app/server projects—between 6 to 16 weeks—Appster does not at present practice Continuous Integration,” says Halford.
Appster has three main objectives: to deploy only from BitBucket, to protect the non-development servers from the developers, and to display the build statuses via dashboard to all. Deploying only from BitBucket holds the source code repository accountable as the “single source of truth”, says Halford. In order to protect the non-development servers, developers do not have root level access to the testing, staging, or production servers, only to the development server in this four server stack, Halford explains. Further, the QA Team has access to the development and test servers as well as limited access to the staging server, but no access to the production server. Only the Dev Ops Team / Linux Admins have access to the production server; they also have responsibility for the maintenance and administration of the entire server stack, Halford says.
App Dev At Appster Prior To DevOps
Prior to moving to DevOps approaches, Appster developed apps using manual steps and documented processes that were prone to human error, says Halford. “The process was also prone to human shortcuts and attempts to hot fix builds by circumventing source control and documented protocols such as those that demanded that no one make direct updates to the production server,” Halford says. These issues were not immediately uncovered.
A Typical Trip Through Today’s Appster Development Sprint
CTO Martin Halford shared his take on a round trip through Appster’s development process today. Appster will first generate development servers from saved generic Linux server images, which the Dev Ops Admins will also secure. Servers will ultimately include development, testing, staging, and production servers (or DEV, TST, STG, PROD). Appster will then generate source repositories in BitBucket, and Jira Projects that it will link to BitBucket. Appster will also create a product backlog using an Agile Scrum Software Methodology.
According to Halford, the development team will begin to develop software using Sprints, and each Sprint will follow this process:
A. The developers will check out their code and associate tasks with User Stories and / or Bugs in Jira.
B. They check code in as Changesets that are attached to tasks, which are in turn attached to User Stories or Bugs.
C. Developers automate and deploy builds as needed to the development server.
D. At the completion of a Sprint, the QA Manager selects completed tasks (User Stories/Bug Fixes) for deployment to the test server.
E. BuildMaster initiates the build, which pulls Changesets from the source repository, generates the build, and deploys it to the target server for testing.
F. After a successful Smoke Test, the QA team performs more involved tests to verify the build.
G. BuildMaster follows a similar process to deploy the build to the staging server for User Acceptance Testing.
H. The next Sprint begins and the cycle repeats itself.
Concluding Thoughts
“DevOps can heighten the effectiveness of mobile development by forcing and enforcing discipline on all parties, preventing human error, speeding QA testing cycles, effectively stopping defects, and highlighting inefficiencies in the existing development process such as could be alleviated through additional developer training and mentoring,” says Halford. Further, having a dashboard that is visible to all parties providing transparency into server issues so that Dev Ops Admins can respond immediately during build or production is critical to the DevOps process.
The concepts of communication, collaboration, abstraction, automation and orchestration are cornerstones of the rapidly growing DevOps movement. At the same time reliance on virtualized infrastructure and Infrastructure-as-a-Service has exploded, making manual provisioning and management simply not feasible anymore; it takes too long and locks up too many resources. Modern DevOps methods and tools have emerged, allowing IT organizations to move faster and with higher quality, thus giving them the ability to respond to the business with more agility.
Now security teams have an opportunity to learn from the DevOps experience. Manual policy provisioning and security operations in highly dynamic IaaS environments simply doesn’t work, for the same reason it doesn’t work for DevOps teams – the pace of change is simply too to fast to handle manually.
Applying security policies based on static parameters and making manual rule changes just before production leaves little time for provisioning the policies. This impacts release quality, increases risk of errors and slows down the DevOps cycle.
Trying to use DevOps orchestration tools to provision security can leave companies exposed since these tools lack critical controls and don’t integrate with the rest of the security infrastructure.
To solve these challenges, IT and security teams need to adopt platforms and processes that match the speed and agility of their DevOps brethren.
Here are the key ingredients you should look for in security solutions that can move at the speed of DevOps:
Built-in Automation – Security automation means that any control (e.g. firewall policies, configuration vulnerability scans, intrusion detection, multi-factor authentication) can be deployed and managed without human intervention. Most desirable is full-lifecycle automation, in which policies are set once and tied to some context, after which underlying controls are 100% automated at each stage of the control’s lifecycle, from deployment to de-provisioning. Automated collection of audit and operational data is also critical, especially in environments where infrastructure components are only operational for short periods of time. Even though short-lived, these ephemeral resources are still in scope for auditor inspection, even if not running at audit time. Well-implemented automation enables security organizations to keep up with the scale and rate of change associated with dynamic infrastructure models. Security accuracy and effectiveness are both improved by automation, and potential for human error is removed—especially if API instrumentation enables cooperation of otherwise disparate technologies.
Security Orchestration – Platforms that enable security orchestration centrally manage the composition, deployment, and management of individual control components into more complex, service-oriented security systems. By composing many individual controls into a larger system, security orchestration is considered to be a higher order function than simple control automation. In many implementations, orchestration also addresses licensing, metering, chargeback, and other security resource consumption issues that are important in service-oriented cloud computing and software-defined infrastructure environments.
Instant Visibility & Continuous Enforcement at the Workload – Public clouds have no natural perimeter and network segmentation, which leaves individual servers exposed. In private clouds, malicious East-West traffic inside the network is undetected by perimeter tools and can become a serious threat. So choose a security platform that extends your investments in network security directly to the workload itself. The solution should be on-demand and easy to deploy. Many of these platforms have an agent-based model, so make sure the agent is ultra lightweight to eliminate drag on the virtual server, is non-intrusive to the workload and is easy to integrate into DevOps continuous deployment model. The agents should be deployable through orchestration tools, with scripts or manually, even on live systems without reboot, to speed up the process even further.
Flexible Policy Definition – A modern security platform should allow security policies to be defined by logical application groupings instead of static network parameters, which protects new workloads automatically and overcomes natural limitations of traditional network security tools.
Security at Every Stage – The DevOps model has multiple stages, many of which are conducted on various cloud services and on other virtualized architectures. This leaves assets vulnerable to attackers, so baking in security at each stage prevents this issue. Just as importantly, development teams need to know how security will impact the application being developed, so incorporating security early in the process makes a ton of sense.
Layered Approach – Having a platform that provides layered security (not just a firewall) in the DevOps model, is key. Integrating multiple functions from different vendors would prove enormously daunting from an orchestration perspective. So make sure layered security functions like file integrity monitoring, security configuration monitoring, strong access control and vulnerability management are baked into a single platform and included on every system throughout the lifecycle.
Seamless Integration with Orchestration Tools – Make sure your security platform integrates seamlessly with the orchestration tools you’re already using. Jumping back and forth between tools can slow things down, introduce errors and lower your overall security posture.
By implementing a security solution that incorporates all of these attributes, IT can bring security into the high speed, high quality DevOps model that is now required to provision and manage modern infrastructure.
About the Author/Amrit Williams
Amrit Williams is the CTO for CloudPassage. Previously Amrit was the Director of Emerging Security Technologies and CTO for mobile computing at IBM. Prior to IBM, Amrit was a research director in the Information Security and Risk Research Practice at Gartner, Inc. where he covered vulnerability and threat management, network security, security information and event management, risk management, and secure application development. Previously, Amrit was a director of engineering for nCircle Network Security, and undertook leadership positions at Consilient Inc., Network Associates, and McAfee Associates.
Back in my days at StillSecure a major problem we had with our vulnerability management platform was managing vulnerabilities over a large number of hosts. How can you effectively manage so many hosts, visualize them across the entire deployment? It gave our GUI designers and developers fits.
In today’s world of virtual instances, containers and apps, managing a large number of hosts is even a bigger challenge. It is a daunting task that even the largest companies struggle with. In fact in some ways the bigger the company, the greater the amount of hosts and the harder it is to manage. I wanted to highlight today how one company, Datadog has tackled this problem with their Host Maps feature.
The trick to managing large numbers of hosts is to balance a global view with an individual host view. It is harder than it sounds though. It is almost like joining Einstein’s Theory of Relativity with Quantum Mechanics. How can we use one view to see things on both the galactic scale and the molecular level at the same time. Datadog has developed one of the best ways to do this with their instant grouping and drill down capability.
Instant grouping allows you to virtually (no pun intended) define groups on the fly by a large number of tags and filters. So for instance you can use standard AWS tags. You can also group hosts by tags and they will be visualized this way.
This was where we ran into the wall on our vulnerability manager. We were never able to create groups on the fly. You could define groups by certain filters but drilling down from there was out of reach. With Host Map feature you can create a group of hosts that have a certain feature or attribute in common and then drill down into that group with the click of a mouse. The YouTube video below shows you much better than I have described what I am talking about:
Datadog’s designers have also made great use of color selection and shaping to help make visualization much easier for the end user. Pretty nifty use of simple technology.
Not to turn this into a love fest for Datadog Host Map, but I am not aware of others who have instituted this kind of on the fly grouping and drill down capability to address this Big Data type of problem. If you know of one, I would love to hear about it.
From the point of view of looking at how the state-of-the-art has evolved from when we were trying to tackle this issue 10 years ago, it is impressive to see how far GUI has progressed. I think it is a clear case of “mother is the necessity of invention”. It would be near impossible to manage the kind of host deployments we see now without this type of functionality.
I did have a chance to speak to a few Datadog customers when I reached out to Datadog about Host Maps. It was unanimous that they feature was as valuable to them as it looked to me. Now of course one could say that Datadog wouldn’t refer me to customers who didn’t like the product. But my impression was that this feature was indeed a big help to them.
As we continue learn to better manage data flows and manage ever growing infrastructure, I think we will continue to see the evolution of management GUIs like what Datadog has done here.
What about you? What GUI functionality have you run across that you think is worthy of praise? I am sure others would like to know.
Dr. James Brown stunned the British Science Festival by announcing a wonder pill that could reduce your weight, lower the risk of high blood pressure, cancer, diabetes, depression, and anxiety. With everyone in attendance eagerly awaiting what it was, he plainly exclaimed that the “wonder pill” was daily exercise.
The most effective solution was the simple one. The same applies towards developing a successful mobile application.
There is a sure fire way to generate greater sales, leap frog the majority of your competition, promote wider brand recognition, and win loyal customers. The wonder pill: make sure your mobile application works.
“The majority of mobile applications get a one star rating from users due to poor quality. Either the apps don’t work properly or they perform too slowly. The best way to guarantee the financial success of a mobile application is to simply test it properly so the user has an ideal experience.” says Tal Barmeir, CEO of Experitest. If a mobile application has a function that isn’t working, 80% users will simply delete it. In some cases they will tell their social media followers about how lousy the company is. They won’t pan the app mind you – they will put down your entire company. Performance is no different. A slow performing application, even if it works, might as well be broken.
If a daily walk is the simple way to quality health, mobile application testing is the simple way to a quality app. If you can guarantee that your application works properly, your mobile application just leapfrogged the majority of its competition that cannot make that claim.
Is Mobile Testing as Simple as a Walk in the Park?
To guarantee top quality of your app you have to test it over all types of devices, and for each device you have to test over several mobile operating systems. For each permutation of device and operating system you need to make sure your application works fast. But be warned! Just because it works fast over a cutting edge local network in Singapore doesn’t mean it will perform as well in rural Indonesia.
Add in smartwatches, other wearables, and apps managing the internet of things and the matrix of test scenarios you need to cover can seem endless.
The best testing tools make it look easy. You can set up an on premise mobile device lab in your company, and using the cloud attach as many devices as you want to it. Each device can have a different operating system, creating for yourself a large a matrix of devices and operating systems, expanding your test area to meet your needs. Your quality assurance testers can access these devices by remote using automation testing tools from wherever they are in the company. A manager can decide to assign devices to different testing teams working in different time zones to create a 24 hour nonstop testing operation. By including a Network Virtualization tool, testers can simulate any local network condition on earth. They can verify that their application works as smoothly in Nairobi as well as it does in New York City. The best testing tools also constantly update to enable you to test for new technologies like wearables and the IoT.
That’s why mobile testing is the wonder pill of mobile application success. It makes your mobile application works. It makes sure your mobile application performs fast. It makes sure your customers and prospects know that your mobile app is the best there is – and they tell their friends about it.
About the Author/David Fink David Fink is a product evangelist for Experitest, a leading provider of mobile test automation, manual testing and monitoring tools for mobile applications. David worked on Wall Street during the internet boom of the 1990s and 2000s. Watching the same transformation taking place in mobile, he writes about what we can learn this time around to make this new phase in digital growth usher in smoother to benefit everyone involved. Reach him on Linkedin
Alan Shimel, Editor-in-Chief DevOps.com, An often-cited personality in the security and technology community and a sought-after speaker at industry and government events, Alan has helped build several successful technology companies by combining a strong business background with a deep knowledge of technology.
Georg von Sperling, DevOps and Technology Evangelist at CA Technologies
Anbu G. Muppidathi, Vice President at Cognizant Technology Solutions
Amrit Williams is the CTO for CloudPassage. Previously Amrit was the Director of Emerging Security Technologies and CTO for mobile computing at IBM. Prior to IBM, Amrit was a research director in the Information Security and Risk Research Practice at Gartner, Inc. where he covered vulnerability and threat management, network security, security information and event management, risk management, and secure application development. Previously, Amrit was a director of engineering for nCircle Network Security, and undertook leadership positions at Consilient Inc., Network Associates, and McAfee Associates.
