devopsy.kinsta.cloud

Tag: monitoring

Taming 2 Cloud App Performance Enemies

Whenever you hear the words “the network’s slow today,” or, even worse, “the network’s down,” you know somebody miscalculated. It may have been the person who decided to buy the wrong hardware, or maybe the one who signed up the wrong telecom provider.

Often, that slowdown or outage was the result of somebody underestimating server demand or overestimating the amount of redundancy built into the network setup. It’s always tricky finding the right level of redundancy that will keep users from waiting for their data without overspending on cloud capacity.

A common mistake IT managers make when switching from in-house infrastructure to cloud infrastructure is failing to take advantage of the flexible, affordable, redundant capacity offered by Amazon Web Services, Microsoft Azure and other cloud providers. That’s the conclusion of Yan Cui in a April 20, 2015, post on his “The Burning Monk” blog.

Cui points out that latency is inevitable because things go wrong—and usually, those things are beyond your control. That’s why you have to design your applications with latency in mind. Too often developers work inside a bubble, where all hardware and network parameters are geared to the tasks at hand. In the real world, hardware, networks and software break. If your app isn’t designed to accommodate such failures, you’re leaving your users in the lurch.

Get Real: Inject Latency in Your Test Environment

To model app performance in real-world settings, it isn’t enough to test servers and databases under anticipated loads. You have to take the extra step of mimicking the glitches that are as inevitable as they are unpredictable. Cui proposes injecting random latency delays on every request. He even provides a simple PostSharp attribute that automates the process.

Cui applies the concept of error-planning even further by introducing errors in the mechanism used to throttle requests to prevent bad guys from spamming their servers using such tools as Charles and Fiddler, or any of the spammers’ own bots. By making the errors visible in the dev environment, designers can anticipate them and engineer ways to handle the throttling errors “gracefully.” Once again, a PostSharp attribute is applied to randomly inject throttling errors; Cui states that the same approach can be used to plan for and respond automatically to session expiration, state out-of-sync, and other service-specific errors.

These and other error-detection and -correction methods fall under the umbrella of Design for Failure, a principle championed by Simon Wardley as a key characteristic of next-generation tech companies.

Simon Wardley’s Design for Failure contrasts the development approaches of traditional shops and those he identifies as “next generation.” Source: The Burning Monk

Cui explains how modeling latency and other errors within the development process has been applied by Netflix’s use of Chaos Monkey and Chaos Gorilla, as well as the mechanism Uber devised to allow its client applications to fail over seamlessly in response to a data-center outage.

Learn How to Combat Latency from the experts: Gamers

Nowhere does server performance matter more than in the world of online games, where one-microsecond hiccup can spell the difference between winner and loser, champion and chump. In an April 17 article on Venture Beat, Cedexis’s Pete Masden explains that a 500-millisecond network delay makes an online gamer twice as likely to walk away, while a 2-second delay in load time leads to abandonment rates as high as 87 percent.

Addressing latency begins by measuring it, according to Masden. That’s why we added native monitoring to Morpheus when we built the cloud management platform. Being able to see the status of all your organization’s apps and business services at a glance isn’t just a nice convenience—it’s a competitive advantage. To be able to glance the status of all your organization’s apps and business services, group databases, app servers, messaging queues and web servers by system dependency and impact to view an overall status as well as each app’s current state is crucial to knowing what’s happening in your organization.

One approach to cloud-server latency is to create separate instance, which is common for failover but less so for addressing regional network demand. Rather than the traditional active-passive model, however, Masden recommends an active-active approach when implementing multiple redundant instances to ensure all servers are up to date and optimized. Atop the multiple instances runs the global traffic analyzer to ensure no single instance becomes overly congested.

Is ‘Zero-Latency Computing’ Even Possible? Why not?

If you think network latency is a problem now, just wait until such technologies as 5G, the internet of things (IoT), virtual reality (VR) and robotics go mainstream. Equipment and bandwidth upgrades, and more sophisticated software-defined networking won’t be sufficient to address tomorrow’s application-performance needs. That’s the conclusion of Nitin Serro in a Sept. 29 post on the Channel Partners blog. Serro identifies three technologies as key to meeting future performance requirements: network functions virtualization infrastructures (NFVi), fog computing and zero-latency computing.

The applications made possible by 5G and related technologies come close to eliminating latency by shifting more intelligence directly to the network, not merely by providing faster speeds. Source: GSMA Intelligence, via The Unwired People

While NFVi and fog computing focus primarily on the network and data layers (compute, storage and network infrastructure), zero-latency computing targets the application layer, where information has to be exchanged in real time between diverse components, most of which link directly to users. According to Serro, nothing short of an entire re-invention of the network infrastructure—from backbone to edge—will prevent today’s bottlenecks from becoming tomorrow’s blockages.

Move the Network Intelligence Closer to the Data

The network-capacity challenge we now face is unprecedented, according to Silicon Republic’s John Kennedy, who writes in an Oct. 3 post that even the much-touted 5G network will come out of the gate hobbled by the need to accommodate its many predecessors. Rather than the “step change” in network speed that most analysts expect, 5G will deliver a tremendous boost to network intelligence. This may be the technology’s greatest asset: a smarter network means the data doesn’t have as far to travel.

From an application-design perspective, latency can be traced to three sources: the need to link to legacy infrastructure, the need to link to third parties via APIs and the innate unpredictability of the internet itself. Marqeta’s Dave Matter examines these situations from a payment-processing perspective in a Sept. 13 post on App Developer Magazine.

As with the network infrastructure itself, the solution to application latency is to move more of the intelligence into the app itself and then rely on open APIs to link data sources in real time in an environment that’s closer to users. Once again, the data and associated analytics tools have less distance to travel, and much less latency to deal with. Ultimately, the solution to latency—in hardware and in software—will be fewer, shorter trips.

— Brian Wheeler

October 10, 2016
Alerts Keep DevOps Teams Responsive, in Sync
Continuing our five-part series on “Continuous Monitoring: The Role of DevOps and APM,” this blog post will talk about how alert notification can help DevOps teams be more productive and agile.

Amid the constant, rapid change and fluidity typical in DevOps organizations, there are some undeniable absolutes. These absolutes, present at each stage of the DevOps life cycle, demand your teams’ attention: A build or test succeeds or it fails; an outage of an app in production is restored to service or it lingers unavailable; users applaud or they blast out complaints. And with each of these states throughout the DevOps life cycle there are absolutes common at every step: Your team is either communicating or they are isolated, and, subsequently, either in sync or operating in confusion and chaos.

This is still common, but avoidable. The means of communication are more abundant than ever. We can text, email, use collaboration apps or even the obscure method of speaking into a telephone to another human in real time. Yet, despite all these methods, DevOps teams still find themselves taking too long to become aware of issues that need immediate attention or action. Or, we realize too late that multiple people are responding to or chasing down the same issue. Ironically, the easier it is to communicate and even automate communication across all of the different methods, the more likely we get inundated and saturated by what we really need to be paying attention to.

What’s the best way to cut through all the noise? DevOps teams can:
1. Filter the junk and cut down on the noise to begin with—without missing something important.
2. Direct the relevant info only to those people that need to take action, and only when it’s the appropriate time.
3. Require an acknowledgment that the recipient is taking ownership of the issue.
4. Or, of course, all of the above.
There have been tools available over the years to tackle these noise-slaying objectives. Some do a great job getting info to our team’s pertinent DevOps information, but with no way to filter out the irrelevant or redundant stuff. And others can do some of everything, but are complex to set up and maintain and expensive to use.

DevOps teams I’ve talked to want to be able to:
- Immediately and automatically send critical, actionable alerts.
- Route alerts to the right people immediately, but only when it is the appropriate time to do so (don’t wake up the wrong teammate!).
- Speed alert response with automated acknowledgment and escalation.
- Automatically communicate through text, email, voice mail and mobile app, all custom-designed for DevOps use cases.
The prevalent expectation goes beyond easy-to-use, simple to start and maintain solutions. Effective tools also must meet the increasing demand for agile and efficient collaboration among DevOps team members who often use multiple communication tools.
- Filter out the noise, but don’t miss the important stuff:
  - Filter alerts, (e.g. by problem severity), so you can focus on real problems and disregard IT operations noise. Create custom groups of contacts to make sure the right notifications reach the right team member, and streamline the process of determining who to contact when specific problems occur and how to contact them.
- Notification channels:
  - Define experts who can respond to problems and store their contact details so they can be notified automatically of problems in their area of responsibility. Users should be organized into groups and so you can send notifications to several users at once. Notifications should able to be sent by email, SMS, mobile app, voice message and chat apps.
- Alert management and notifications:
  - Tools with policy-based notifications can let you define which type of subject matter you want each person or group to be notified about. You should be able to create filters based on the alerts that occur, customize the filters and assign users and groups who are notified when matching problems occur. Unacknowledged alerts should be escalated automatically after a set time period to the appropriate contact. Filters such as the severity and status of alerts can be created for your IT monitoring requirements.
- Scheduling:
  - You’ll want to be able to easily set up on-duty and on-call shifts. The process of scheduling is considered to be one the most time-consuming tasks that prevent users from utilizing a notification tool. A good scheduling capability should be intuitive and require minimal steps to use. Users should have the ability to customize or use out-of-the-box templates. Users also should be able to manually create shifts or build in shift patterns, such as follow the sun, to get their scheduling done faster. Scheduling capabilities also should be smart enough to schedule users or groups for weeks in advance.
- Alert viewer:
  - You’ll also need to be able to monitor alert and notification status in management views, perform actions on alerts, and track them in real time. Seeing key alert information such as alert severity can help you enable to easily prioritize your to-do list. You should be able to track alerts from their receipt by the system through acknowledgment to resolution and changes in an alert history, and use predefined and real-time filters to designate alerts you want to see.
Utilizing the capabilities such as these can empower you to improve responsiveness, save resolution time and focus on actionable notifications requiring immediate attention.

About the Author / James Moore

James Moore is responsible for offering management and strategy for IBM IT Service Management hybrid cloud-based solutions including Runbook Automation and Alert Notification. James joined IBM from the Candle acquisition in 2004, where he was product manager for Candle’s Application Response Time product lines. James has more than 15 years’ experience in business service management, application performance management and event management. Connect with James on LinkedIn and Twitter.
October 5, 2016
Putting Ops Back in DevOps

What Agile means to your typical operations staff member is, “More junk coming faster that I will get blamed for when it breaks.” There always is tension between development and operations when something goes south. Developers are sure the code worked on their machine; therefore, if it does not work in some other environment, operations must have changed something that made it break. Operations sees the same code perform differently on the same machine with the same config, which means if something broke, the most recent change must have caused it … i.e. the code did it. The finger-pointing squabbles are epic (no pun intended). So how do we get Ops folks interested in DevOps without promising them only a quantum order of magnitude more problems—and delivered faster?

The problem is exacerbated when the topic of DevOps comes up, as it is most often a developer doing the talking. And the topics under conversation are things every developer would have substantial interest in and pretty much nothing that would interest the operations part of the equation. Many Ops folks feel as though they have been dragged into this DevOps phenomenon by osmosis, without their consent, and now they are supposed to be reaping huge benefits from it (by magic, as near as anyone can tell). Developers working faster is traditionally not something Ops folks look forward to; rather, it may be their worst nightmare. So, if the advocates of DevOps all tend to be developers, how do we get operations folks to become our evangelists as well?

QA Impacts on Evangelism

It begins with quality—not the ethereal concept of quality, but rather the specific responsibility for who owns it. Finger-pointing be gone; developers now own QA. If the software development lifecycle (SDLC) environment is broken, developers own it. Ops can sit back, toast marshmallows and enjoy the show. Developers should be including whatever configuration file changes their application needs as part of the deployable package. So nothing is done manually anymore. This means no one could forget something, as machines do not forget; they do what we tell them. If by chance, some random operations staffer decides to doink with a config variable, the deployment software should simply overwrite the manual change with what is in the deployable package created by the developer. Even malicious changes can be overwritten, so now, all change is included in the development class environment in the one and only deployable package. That is the only package that moves forward.

If developers own the QA, there is now a shift in the force: The operations team has a new role, one of coach and adviser, instead of doer. The software now is the doer, the developers are the packager and the operations staff are the witnesses of how it works. If it breaks, we all know who must fix it (and who is to blame). But once blame has been permanently affixed, something else changes, namely the culture. Now that the operations team does not take the blame, it remains interested in seeing its operations run without error. This allows operations to take an active “helping” interest in getting things fixed faster. It does not really matter who is to blame, once that argument has been settled; now it only matters if they can make something work. Harmony between factions begins to emerge.

Scalability Impacts on Evangelism

It continues with scale. Self-service is the nirvana of your average operations staffer. If the responsibility for provisioning can be moved into a push-button self-serve automation, nirvana will have been achieved. Imagine the offload of repetitious work tasks that consume the life of your average operations staffer. Instead of setting up (and taking down) environment after environment of one class or another for one application or another, requesters can all do it themselves with the push of a button. Even the accounting or financial impact is something the accounting team can guide the user on; operations does not have to worry about the cost of its operations anymore (or, at least, worry less). If too many environments are being provisioned, the accounting team has the power to change the rules and keep it from happening automatically without operations having to be the bad guy.

At this point, Ops can resume a focus on something it tends to enjoy more: monitoring. Ops staffers enjoy being able to provide real-time reporting on where things stand at any point in time. It is even more rewarding to provide more meaningful information such as who is using this application, what users are doing with it and when are they performing that activity. To monitor that kind of information, operations must partner with developers at the outset of a project to determine what kind of application logs are needed later for monitoring. Without the daily barrage of mundane provisioning tasks, an operations engineer is free to focus on more rewarding (translate: higher skill-level) tasks. The days of changing tapes for the backup may have reached their end (or pick your similar set of mundane tasks which DevOps automation can fully eliminate).

And Speaking of Monitoring

It merges into the future. How do you create an operations staffer as a DevOps evangelist? Start by exploring potential that only DevOps can enable. All those nagging alerts that are created when an environment goes down while a deployment is made … all of that could be turned off by the DevOps automation, then turned back on when the deployment is completed. Again, it all would be machine-driven, with no human mistakes anymore, ever. This will begin to cut down “noise” in monitoring/alerting systems into only useful information. However, integrating monitoring systems into the normal DevOps continuum extends beyond just an on- and off-switch.

Forward-thinking organizations begin to consider testing and monitoring as two parts of the same evaluation discipline. When a test is needed and when a monitoring characteristic is needed now can be determined in total rather than just in a development capacity or operations capacity. A holistic view of what can and what should be evaluated about an application, and how that will be achieved, brings the operations team to the table with the development and testing teams to plot holistic solutions that will live on beyond the development project timelines. Examining monitoring relative to version-controlled artifacts allows for organizational constructs within the monitoring discipline that never were possible before DevOps enabled them.

The Masters of Hardware Abstraction

Containers, or similar types of technologies, abstract an application from the hardware details it runs on top of. Technology such as this is a trend in DevOps that greatly simplifies the world of hardware from your average application developer. So who will retain the knowledge about the jungle that lives and breathes underneath? Ops has an extended role in understanding what lives underneath the abstraction layer that lives on top. Over time, only Ops will understand these particulars. They will become the only in-house experts about which cloud provider to use, which sets of physical hardware performs best and under what conditions.

It is this kind on unique expertise that makes an operations engineer extremely valuable (translate: pricey). It is the kind of knowledge careers are made upon. It is ageism-neutral. Older employees do this today, and they will continue doing this in the future; the only difference will be fewer and fewer people will know how it is done. If the domain of application development is the province of our youth, then the domain of hardware abstraction is easily the province of our more experienced employees. These realities create personal value for our operations teams, and they are only possible in a post-DevOps embrace.

Impacts to the Bottom Line

The adoption of DevOps by the Ops portion of the equation is critical to overall success. Passive-aggressive resistance by operations staffers can delay or destroy altogether the value of DevOps. Overcoming this is a first step. But creating operations evangelists goes an order of magnitude further: With active operations staffers becoming internal DevOps evangelists, the entire operations team can begin to look forward at what DevOps will ultimately enable for them. The value they will carry back into development efforts becomes substantial and the entire company benefits. Careers can emerge that were not possible before it—and, just as importantly, so can profits.

To continue the conversation, feel free to contact me.

August 26, 2016
Ending Alert Fatigue with Modern Security Incident Management

RECORDING

August 1, 2016
Datadog Expands Monitoring Solution Across Stack for Microsoft Customers
The leading cloud-monitoring service Datadog announced their latest integrations with Microsoft Azure Resource Manager and Microsoft Azure SQL Database

Datadog agent allows customers to seamlessly monitor their stack, from cloud hardware down to custom metrics for their application
July 11, 2016 09:00 AM Eastern Daylight Time
NEW YORK–(BUSINESS WIRE)–Datadog, the essential monitoring service for dynamic cloud infrastructure, today announced expanded support for the Microsoft Azure cloud platform. Following the release of Microsoft Azure Resource Manager (ARM) – the latest way to deploy virtual machines on the platform – Datadog will now support Microsoft Azure SQL Database as well as cloud servers deployed through ARM. This will empower large enterprise companies running their infrastructure in the cloud to correlate metrics from these new services with application performance. The new capabilities will be demonstrated in the Datadog booth 1724 at the 2016 Microsoft Worldwide Partner Conference taking place July 10-14 in Toronto, Canada.

“We make it a top priority to build out integrations for new services from major cloud platforms as they are released,” said Amit Agarwal, Chief Product Officer, Datadog. “By utilizing Datadog, enterprise customers are now able to effectively manage the performance of their large scale Microsoft Azure-hosted applications.”

Datadog already boasts a number of Microsoft integrations, from monitoring the virtual machines deployed on Azure, to collecting metrics and events from native windows services such as Windows Management Instrumentation and Event Viewer.

Datadog is compatible with the following Microsoft hardware and middleware:

Hardware

Classic Virtual Machines

VMs deployed through Azure Resource Manager

SQL Database

Middleware

Windows

Internet Information Services

SQL server

Windows Event Viewer

Windows Management Instrumentation

Application Custom Metrics

StatsD for C#

“Datadog is a key monitoring tool for enterprise companies embracing the cloud,” said Eduardo Enciso, Global Business Strategist, Microsoft Corp. “Their new integrations with Microsoft Azure allow our mutual customers to use the capabilities in Microsoft Azure VM and Microsoft Azure SQL Database, with full visibility across multiple Azure services and applications.”

Datadog is available for one-click deployment through the Azure Marketplace and also from the Microsoft Azure Portal. They will be demoing their latest integrations at Microsoft’s Worldwide Partner Conference in Toronto.

About Datadog

Datadog is the world’s leading monitoring service for cloud-scale applications, bringing together data from servers, databases, tools and services to present a unified view of your entire stack. These capabilities are provided on a SaaS-based data analytics platform that enables Dev and Ops teams to work collaboratively to avoid downtime, resolve performance problems, and ensure that development and deployment cycles finish on time. Since launching in 2010, Datadog has been adopted by thousands of enterprises, including Twilio, Airbnb, Netflix, EA, Spotify, Warner Bros. Games and AdRoll.
Contacts

Mindshare PR
Heather Fitzsimmons, 650-279-4360
heather@mindsharepr.com
July 11, 2016
Iterative Indicators: Measuring DevOps ROI

It’s well-established that application of ROI measurements to IT investment remains an imperfect science, and the same goes for ongoing adoption of DevOps methodologies.

The fuzzy math of vendor-driven ROI calculators aside, even the most respected industry watchers have long struggled to create concrete parameters for assessment of technology spending. This, in large part, has been related to the fact that the requirement for continued investment has outweighed the value of short-term metrics.

At the same time, it’s hard to put a price on the influence that paradigms including mobility, virtualization and the cloud are having on today’s business world. The scope of the so-called Applications Economy is only truly beginning to be realized.

For in-depth discussion of DevOps ROI, register today for the upcoming webcast “Maturity Metrics – Analyzing the 2016 State of DevOps Report” with noted industry expert Gene Kim.

With DevOps, the challenge of ROI measurement evokes similar characteristics as with these other bellwether movements. The affect that leveraging DevOps to become more agile and respond adeptly to changing customer expectations will have, long-term, ultimately will be realized as much in organizational viability as in any immediate cost-savings or other factors.

That said, the recently published 2016 State of DevOps Report, assembled by the researchers at DevOps Research and Assessment (DORA), goes further down the path toward creating meaningful ROI metrics than any other such previous effort.

For starters, the 2016 report builds on previous year’s conclusions that so-called “high-performing organizations”—those already deeply engaged in DevOps practices—are “decisively outperforming their lower-performing peers in terms of throughput,” and this gap continues to widen.

Moreover, DORA contends that high performers deploy applications code 200 times more frequently than slow performers and achieve 2,555 times faster lead times. Perhaps even more importantly, in particular as it relates to the ability to improve applications performance, the report estimates that leading DevOps practitioners lay claim to 24 times faster recovery times and three times lower change failure rates.

The State of DevOps Survey, completed by roughly 4,600 technology practitioners, also found that high-performing teams spend far less time on so-called “unplanned work and rework”—most often required to address existing problems—allowing them to spend nearly 50 percent more of their time on new work. The latter brand of effort is typically aimed at adding or improving feature functionality—or, in simple terms, innovation.

As increased software velocity and reduction of MTTR represent the most critical differentiators in today’s environment, where an end user may wait only seconds for an app to load before abandoning your brand altogether, these conclusions obviously carry impressive weight.

Finally, based on a framework established by research analysts at IDC which suggests that hourly application downtime costs can range from $1.25 to $2.5 billion for a Fortune 1000 firm, and that the average cost of a critical application failure is $500,000 to $1 million per hour, the State of DevOps Report allows for the conclusion that high-performing organizations save an average of more than $91 million per year.

Much of this advantage is found, the researchers note, in the form of lower applications change failure rates, or those updates to software code that result in subsequent performance issues or outages. While high-performing DevOps practitioners experience an average change failure rate of 7.5 percent, medium-performing organizations see that figure rise dramatically, to a whopping 38 percent.

If the undeniable adage that “time is money” applies in today’s Applications Economy as much—if not more than ever—than these ROI calculations would appear to infer substantial meaning.

Later this month, Gene Kim, one of DORA’s founders and a key contributor to the State of DevOps Report, will host a webcast with CA Technologies expert Aruna Ravichandran to discuss DevOps ROI and many other related topics. Register for that event today.

July 8, 2016
DevOps Matchmaking: APM + CI = A Perfect Marriage

As anyone who has ever been set up on blind dates by friends can attest, finding the right match in today’s world is a tricky feat.

There’s a lot of evidence suggesting that the frantic pace of life makes such matchmaking more difficult than ever. That said, when the right set of complimentary traits exist and there’s a willingness to find common ground, things can work out.

Spike Jonzian-notions of artificial intelligence aside, certain technological partnerships are increasingly being perceived as “a perfect marriage” within the context of DevOps automation.

Register today for the upcoming webcast: “Using APM to Turbocharge Continuous Integration”

These technological matrimonies closely support the underlying tenets of DevOps methodology—collaboration, experimentation and increased velocity. In fact, it’s easy to wonder why no one ever conjoined them before. But we were using traditional waterfall development and operations and so couldn’t conceive of such a thing. On the other hand, emerging availability of tooling supporting new integrated workflows represents a significant catalyst.

Finding the Perfect Fit

One such DevOps confederation getting a lot of attention is the matchmaking of continuous integration (CI) and application performance management (APM) technologies. Tying together these once-disparate systems makes a lot of sense, despite the fact that few tools would seemingly appeal to more (traditionally) polarized constituencies.

Continuous integration has been a staple of the development community for many years. It emerged as teams evolved the practice of long build times into more frequent check-ins to deliver code and look for potential errors more quickly. Today, with the rise of CI tools such as the Cloudbees Jenkins platform, some development teams are touting tens of thousands of builds per day.

On the other end of the spectrum, squarely on the turf of IT operations, APM has been evolving rapidly over the past few years as organizations have sought to improve use experience and applications reliability via dedicated monitoring. Enabling this ability to identify and address emerging issues has resulted in APM becoming the fastest-growing segment in the IT ops management market, according to Gartner.

DevOps matchmaking factors in when you bring these tools together to create an automated workflow that spans their respective use cases. A cross-functional team can leverage APM-level monitoring and KPIs within the domain of CI and testing to identify potential performance obstacles introduced in new code. Doing so then rather than waiting to respond later in the applications life cycle benefits everyone.

Reaping the Benefits

For example, say an e-commerce applications developer has implemented a new inventory search functionality—something considered a key differentiator by business management. The developer knows this feature could rely heavily on some back-end database system and wants to ensure it’s tested properly, even as the code is being introduced.

Integrating APM and CI allows the entire team to monitor the process and test various conditions related to the update, tracking multiple builds and looking out for related performance issues. Such integration also allows for centralized tracking of other key attributes such as build number, status and build date, to further ease management.

Most importantly, by tying these two critical aspects of the applications delivery life cycle together, the collaboration and feedback loops required to truly embrace DevOps methodology become ever more valuable as these core Dev and Ops functions become intrinsically linked.

If this type of DevOps advancement gets your heart pumping, you’re not alone. Later this month, we’ll host a webcast here on devopsy.kinsta.cloud featuring experts from CA Technologies and Cloudbees. They’ll be discussing the benefits of matchmaking with APM + CI integration. They’ll also detail real-world benefits organizations leveraging integration of our respective technologies (CA APM and Cloudbees Jenkins) enjoy.

We obviously feel that this is a DevOps tooling marriage that will mature, change and stand the test of time. We’d love to garner your questions and feedback on the topic.

Which is all good, because thankfully there’s no such thing as Tinder for DevOps … yet.

June 21, 2016
Enterprise Cassandra Deployments
Successful Cassandra deployment in the cloud takes a certain amount of skill and knowledge. In this blog I will share best practices for a successful Cassandra deployment in the cloud such as Amazon Web Services (AWS) and Google Cloud Platform (GCP). I will also discuss how you can protect and recover the data in your Cassandra cluster when deployed in the public cloud.

Cassandra Deployments in the Public Cloud

There are two main ways to have successful enterprise Cassandra deployments in AWS:
- Use a ‘managed’ Cassandra deployment. This is a reasonable approach; the only downside is the inability to customize data placement beyond what is typically provided by the vendor.
- Manage the cluster yourself, if you want to have full control over your Cassandra deployment and customize it to scale and perform according your business needs.
Please refer documentation from Datastax or Apache Cassandra for detailed best practices for Cassandra deployments. The following best practices are reiterated here to simplify deployment of a scalable Cassandra cluster in AWS.

Storage

Both AWS and GCP offer two types of local storage options for their compute instances. For AWS, the options are Elastic Block Storage (EBS) and Ephemeral Storage. In GCP, the options are Persistent Disks and Local SSD. Both options have their own trade-offs. Pick the one that best suits your deployment.

Persistent Storage

EBS volumes have performance characteristics similar to network storage such as a NAS or SAN. Because storage from EBS volumes is multitenant in nature, it is not suitable for distributed, log structured databases such as Cassandra. If you want to ensure reliable performance through EBS volumes, use “EBS optimized instances” with “provisioned IOPS.” Since the IO traffic to EBS volumes goes over the instance network, EBS optimized will provide reliable performance, but does not guarantee any increased performance. To get the most performance out of EBS volumes, use instances with multiple disks and separate the volumes used for “Data” and “Commitlog.”

If you have to use EBS Volumes, go with “EBS Optimized Instances,” with “Provisioned IOPS” and split data and Commit Log on different volumes.

Ephemeral Storage

Ephemeral storage is a great option to use for downstream Cassandra environments such as test/dev and staging. Since ephemeral volumes are typically SSDs, there is no need to separate the data and commitlog into separate drives. To optimize performance, choose EC2 instances with multiple ephemeral storage drives and stripe them together into one logical volume. Send data and commitlog to the logical volume. Remember that when you stop the EC2 instance, the data in Ephemeral storage will be deleted.

Stripe all Ephemeral drives into a single volume to get maximum performance.

Data Distribution

Cassandra uses Snitch to determine the topology of the cluster. The Snitch is also used to determine the data center and rack for each node in the cluster. Replication strategy is used to determine the placement of data across the different nodes in the cluster.

AWS
1. EC2Snitch works well for single region deployments.
1. Ec2MultiRegionSnitch works well for cluster distributed across multiple AWS regions. Please note, Multi-DC, Active-Active configuration makes your cluster resilient to region failures and for multi-dc cluster, use “local quorum” to improve performance.
Google Cloud Platform (GCP)
1. Use GoogleCloudSnitch for Cassandra deployments in GCP.
1. GoogleCloudSnitch treats regions as Cassandra data centers and availability zones as racks.
Other Features

Both AWS and GCP offer a series of operational capabilities that seem to make it easier to maintain high uptime on Apache Cassandra cluster. These capabilities have to be chosen carefully depending your deployment needs.
1. Load Balancing: Network-level load balancing service is offered by both AWS (Elastic Load Balancer) and GCP. It is not recommended to put a load balancer in front of Cassandra. It is better to use a client connector that can handle this, consider open source utilities such Presto and Astyanax
1. Autoscaling: Both AWS and GCP offer Autoscaling facility for compute instances. However, although this functionality looks very enticing, putting all C* nodes in an autoscaling group will not produce the desired result. Instead, create a separate autoscaling group of ‘1 node’ for every node in your cluster. This will ensure that anytime a node goes down, it is automatically brought back up. This will ensure the cluster data placement is not disturbed when nodes go down and nodes coming up do not cause excessive data movement.
Scalable Backups and Reliable Point-in-Time Recovery

Backing up your data is a must-have for enterprise organizations. The impact of data loss is so high that no enterprise wants to risk relying alone on replication. Protecting data in a public cloud environment may seem daunting, but there are several options available that can make this easier compared to an on-premise deployment. A simple option may look like one of the native snapshots:

Storage Snapshots

I have seen few customers use snapshots of EBS volumes as a way to protect the data in their cluster against logical and/or operational errors. This solution will work but with some severe limitations. On top of the overhead of maintaining the scripts to take snapshots, there is a large overhead in storage costs. Taking a snapshot of volumes does not allow for incremental backups of your data. This causes an exponential bloat in the amount of data stored in snapshots, leading to high cost of the backup solution.

Amazon S3 or Google Cloud Storage

Using an S3 bucket as the secondary storage enables enterprises to use low-cost, highthroughput storage, while reducing overall operating costs for storing backup data. Here are a few recommendations when using S3:
1. S3 Region: Amazon S3 buckets are region-specific. A bucket created in one region will provide better performance characteristics to other infrastructure in the same region. Subsequently, it is important to ensure the S3 bucket used as the secondary storage is in the same region as the production database being protected and the Datos IO cluster.
1. Storage Class and Lifecycle Policy: Amazon S3 allows users to create lifecycle policies on buckets. These policies dictate the performance and response time characteristics of objects in the bucket. STANDARD_1A is more suitable for longer lived and less frequently access data, and is not suitable as secondary storage. Additionally, do not turn on any lifecycle policies on the buckets used for the secondary storage. All buckets should be in the standard storage class.
Conclusion

As enterprises leverage the agility provided by public cloud environments, it’s important to understand the various facets of Cassandra deployments to ensure the efficient use of your money and time.
May 23, 2016
Q&A: Sematext’s Thies on Docker Monitoring, Part 2

In part one of my conversation with Stefan Thies, Sematext DevOps expert. we spoke about Docker’s use of Sematext Logsene logging, anomaly detecting and alerting software and SPM monitoring software. Let’s continue the conversation.

David Geer: What makes Logsene centralized? Where is it centralized a typical deployment? What are the advantages of it being centralized?

Thies: Traditionally, systems store logs from multiple applications on every server. This makes it difficult to search logs for troubleshooting or auditing purposes. To find a relevant log entry, operators must log in to each server, find the right file on this server and search in this file for a relevant message.

This can easily take several minutes. With systems storing logs on multiple servers, it is difficult to correlate data, so minutes can turn into frustrating multi-hour root cause-hunting sessions, often involving multiple team members. Logsene provides a central full text index for all log data coming from multiple servers and applications. There is no need to log in to multiple servers and search/grep multiple log files. All team members see all data at the same time and can collaborate much more efficiently.

Having a central place for search makes it much easier to find logs quickly, run analytic queries or centralize alert rules in a single place.

Geer: What is it about what Logsene does for Docker logging?

Thies: Sematext Agent for Docker interacts with the Docker API to automate the collection, parsing and processing of logs, metrics and events. It detects containers automatically (no setup or configuration for a new container!) and supports many log formats for applications on Docker Hub such as Nginx, Apache, Elasticsearch, Solr, MongoDB, MySQL and many more.

Geer: What does your statement mean, “Some DevOps engineers even think of Logsene as ‘ELK Stack on steroids’”?

Thies: The ELK Stack is a combination of open source tools: Elasticsearch, Logstash and Kibana. These tools are very popular for log shipping (Logstash), indexing and search (Elasticsearch) and visualization (Kibana dashboards). However, these tools are missing many enterprise features like multiuser user interface, role-based access controls, secure transport of logs or out-of-the-box configurations and an easy query language to search logs. This is where Sematext Logsene adds the relevant enterprise features to these open source tools to make them easy to use and securely integrated into the enterprise information technology infrastructure.

Maintaining Elasticsearch clusters is another problem for many people. It requires a lot of expertise. Using Logsene does not require this sort of expertise. The Logsene SaaS provides a managed solution where users don’t need to know anything about maintaining Elasticsearch clusters or the required infrastructure. Sematext takes care of that for them. Sematext customers can benefit from these managed services and pay a fraction of the cost compared to self-hosted ELK stacks, where storage, computing and networking infrastructure and manpower for highly skilled workers counts.

Geer: What is Tutum? What is so important about ease of launching SPM and Logsene Docker on Tutum?

Thies: Docker acquired Tutum in 2015. Tutum provides management of Docker containers as a cloud service. Tutum is a platform made to operate “Dockerized” application stacks. In the case of Sematext Agent for Docker, the Tutum integration deploys Sematext Docker Agent automatically to all cluster nodes in a few seconds. These nodes could run on different cloud platforms such as Amazon, Digitalocean, Rackspace or on premises. Again, a big time-saver for people running larger infrastructures or hybrid clouds. Supporting Docker products and having excellent integration with Docker products is important for Sematext.

The Sematext Docker Agent runs on all solutions offered by Docker e.g., Docker Swarm and others such as Google Kubernetes, Google Container Engine, RancherOS, CoreOS, Amazon ECS or Hashicorp Nomad.

Geer: Please discuss the Sematext Docker and Docker image?

Thies: Sematext Docker image is available on Docker Hub, and we shared the Tutum Stackfile for Sematext Docker on Stackfiles.io. The easiest way to get it is to use Sematext, which generates the stackfiles for you, including tokens.

Docker Hub is the central registry for public application images including related installation instructions. Docker users can find out quickly how to run a “Dockerized” application with the help of the instructions in Docker Hub. Typically, this requires a little configuration work before a container can be started with the right settings.

Stackfile.io is the place where people share preconfigured application stacks (e.g., a web service together with the required database). Sematext provides a preconfigured setup for Sematext Docker Agent to monitor all containers in Tutum Cloud. This means one can launch Sematext Docker Agent and monitor all containers running in Tutum Cloud with just a single click for the chosen stack file and by entering a unique identifier (Token) for the application. That is really all it takes.

When you log in to Sematext SPM or Logsene the general installation instructions have all required configurations displayed. Sematext users can simply copy/paste the relevant commands including unique identifiers (tokens) for their application to get the Sematext Docker Agent deployed. This, too, is a big time-saver for our users and it gets them up and running in just a few minutes.

May 9, 2016
Q&A: Sematext’s Thies on Docker Monitoring, Part 1
Stefan Thies is DevOps evangelist Sematext Group. I recently spoke with him about Docker’s use of Sematext Logsene logging, anomaly detecting and alerting software and SPM monitoring software. Read on for part one of our conversation.

David Geer: Technically speaking, for the purposes of this article, what is logging?

Thies: Logging means to store the output of programs (applications, servers) running on a computer. Typically, log messages that software applications produce are stored in log files. However, software running in Docker containers writes its log messages to a virtual terminal (console). The Docker Daemon (the server process managing the containers) has several facilities to forward log messages via log drivers or the Docker API to external applications, which are responsible for storing the log messages. This is not the traditional way of working with log data. This changes the log collection and storage mechanisms and forces many organizations to change their workflows for log management by introducing new tools.

Typically, you would use a service called syslog to receive and store log messages on servers, but this requires special configuration for all Docker containers. While Docker made it easier to deploy applications, the new technology made logging and monitoring much harder as you could not use the traditional tools apart from integration with Docker.

This is where companies like Sematext provide added value, by enabling:
- Integration into the Docker Ecosystem
- Cloud storage for log files
- Indexing for full text search, analytics, visualization and alerting on log data
Geer: What has Sematext Logsene contributed to logging, technically?

Thies: Logsene Log Management includes all required workflows and security procedures to handle log data. Logsene provides role-based access controls for access to log data. Logs are collected and shipped to Logsene servers (running in the cloud or on premises) where the data gets stored and analyzed.

Other important features are:

• Full text search
• Analytics
• Alerts
• Anomaly detection

Geer: In what way is Logsene/Sematext available to users and organizations that seek solutions to capture logging for monitoring their Dockerized distributed applications?

Thies: Logsene is available on premises and as SaaS in the cloud. It fits in hybrid environments as well such as running the Dockerized application in-house on bare metal while shipping logs to Logsene SaaS. In this case, only Sematext Agent for Docker runs on premises.

Sematext Docker Agent automatically detects running Docker containers and automatically collects logs from the containers. The Sematext Docker Agent can transform unstructured log messages into structured data for analytics. We do this with integrated parsers and a pattern library, which detects log formats to structure logs. The final task of Sematext Docker Agent is to ship the structured logs from all containers securely to Logsene, where indexing for full text search and analytics queries takes place. The Logsene user interface provides all tools for search, visualization, anomaly detection and alerts on log data.

Geer: What is SPM for Docker? What role does it play?

Thies: Both Logsene and SPM collect data via the Sematext Docker Agent. SPM is a performance monitoring, anomaly detection and alerting solution for many server applications like Web servers (e.g., Apache, Nginx), databases (e.g., Cassandra, MongoDB, HBase, MySQL, etc.), big data engines (e.g., Spark, Storm, Hadoop, etc.) and full text search engines (e.g., Elasticsearch, Solr). Sematext Agent for Docker collects not only container logs but also performance metrics of all containers at the same time, plus Docker events.

SPM for Docker provides operational insights into application performance on the Docker platform. It is not limited to performance monitoring—it integrates with Logsene for log management in a single user interface. DevOps teams can find log messages in correlation with potential performance bottlenecks. Most organizations use different tools for performance monitoring and logging. Sematext has recognized that metrics and logs must be available in a single platform to provide detailed operational insights without having to switch between different tools and without overhead for things like integration of multiple tools, multiple user interfaces, logins managing access permissions for multiple tools, and billing or licensing from multiple vendors such as NewRelic or Splunk.

Watch this space for part two of my conversation with Stefan Thies of Sematext.
May 3, 2016