devopsy.kinsta.cloud

Category: DevOps Practice

The NVMF Boogie: How Kubernetes and NVMF Will Rock
I recently wrote a blog about how Kubernetes creates a global computing grid that delivers unprecedented compute flexibility. Then I wrote a blog about the importance of going fully Kubernetes-native. In today’s blog I would like to drill down into the role that the NVMe over Fabrics (NVMe-oF or just NVMF) standard will play in enabling Kubernetes infrastructure flexibility.

What is NVMe?

NVMe is a protocol that dictates how a CPU, moves memory via the PCI bus to a storage device. NVMe communicates over a set of rings (per CPU) where commands get submitted from any CPU to the underlying NVMe device. The design of NVMe eliminates intermediate layers between the CPU and the storage device. NVMe devices consist of a controller, queues, namespaces, namespace IDs and the actual storage media with some form of an interface.

Storage media can be grouped into sections called namespaces with an ID. In the context of NVMF, namespaces provide a way to enforce access control for the disks/consumers. Namespaces are analogous to an OS partition, except the partitioning is done in hardware by the controller and not the OS (you can still have OS partitions on namespaces). Some NVMe namespaces might be hidden from a user (e.g. for security isolation). A controller connects to a port through queues and a namespace through its namespace ID. A controller is allowed to connect to multiple namespaces and a namespace is allowed to be controlled by multiple controllers (and thus also, multiple ports).

Imagine smearing out this NVMe device across multiple computers and you get to the next important concept, a storage fabric.

What is NVMF?

When you put a network between the PCI bus and the storage device, you use NVMe over Fabric (aka NVMe-oF or simply NVMF). NVMF enables fast access between hosts and storage systems over a network. Compared to iSCSI, NVMF has much lower access latency, in practice adding only a small latency difference between local and remote storage. NVMF delivers a breakthrough in throughput and seek time relative to traditional device attached storage:

Source: Samsung.

This is possible because NVMF uses RDMA to efficiently access the memory of another computer without using the CPU or operating system of either. RDMA is a transport protocol that bypasses the kernel stack, offloading the workload to network hardware. This means the transport layer takes place in the RDMA NIC itself, so data is transferred directly to or from application memory without having to involve the normal OS-level network stack, achieving low latency with low CPU consumption.

Of course, NVMF performance is gated by the networking infrastructure used, whether Ethernet, Infiniband or Fibre Channel. The November 2018 final ratification of the NVMe/TCP transport specification has accelerated adoption of NVMF because it allows storage fabrics to be deployed on standard network hardware.

NVMF can be used with white label storage systems (JBOD), feature-rich block storage systems (SAN), and storage systems built with previous generation (SATA) devices. This means data centers can simultaneously leverage low latency devices while also using more traditional storage subsystems. NVMF also supports using storage class memory as a cache in the data center compute hierarchy.

Why is NVMF Important?

Think of NVMe as a replacement for SCSI, and NVMF as a replacement for iSCSI and iSER. NVMF has several distinguishing technical attributes that add up to huge performance scalability at the storage infrastructure layer:
- Protocol Efficiency: The NVMe-oF protocol is designed specifically for efficient memory transfers at scale. Designed from the ground up for scalability, the NVMe-oF protocol dramatically reduces the internal locking needed to serialize I/O and improves the efficiency of interrupt handling. NVMe-oF also supports MSI-X and interrupt steering, which prevents bottlenecking at the CPU level and enables massive scalability as systems scale. Combined with the large number of cores in modern processors, NVMe-oF delivers massive parallelism to connected devices which increases I/O throughput and lowers I/O latency.
- Command Set Efficiency: NVMe-oF has a streamlined and simple command set that uses less than half the number of CPU instructions to process an I/O request that SAS or SATA does, providing higher IOPS per CPU instruction cycle and lower I/O latency in the host software stack. NVMe-oF also supports enterprise features such as reservations and client features such as power management, extending the improved efficiency beyond just I/O. NVMEOF devices can approach DDR4 performance levels and, with SCM, become an integral component of the computing memory hierarchy.
- Queue Parallelism: With 65,535 parallel queues, each with a queue depth of 65,535 entries, NVMe-oF supports massive parallelism. Every CPU has a private queue to communicate with a data device and is capable of achieving high speeds because there are no locks between queues that live on separate cores. Because each controller has its own set of queues per CPU, I/O throughput increases linearly with the number of available cores.
- Integrated Security: NVMe-oF includes a tunneling protocol that provides security features produced by the Trusted Computing Group (TCG) and other related communities. Features planned for NVMe-oF devices and systems include simple access control, data at rest protection, crypto-erase, purge-level erase and others. Additional information on NVMe-oF security is available in the Trusted Computing Group and NVM Express Joint White Paper: TCG Storage, Opal and NVMe.
NVMe-oF is designed not only to solve the limitations of scale-out storage, but also to lay the foundation for a storage fabric that dynamically shifts as workloads demand.

What Makes NVMF Important to Kubernetes?

Kubernetes applications implicitly need the infrastructure flexibility enabled by NVMF. NVMF and Kubernetes makes distributed computing within sites easier.

The key point of my Kubernetes-native blog was “(Kubernetes-native) applications break apart into their constituent elements and can deploy and scale across the latency pyramid where they fit best. The elements dance across the infrastructure following their constituents in what I imagine as a containerized ‘ballet’ focused on delivering the best possible user experience at the lowest possible latencies.”

However, multi-cloud, multi-site and edge deployments introduce significant additional complexity for organizations operating Kubernetes at (or beyond) enterprise scale. Business needs are pushing IT teams to move workloads between sites, clouds, WAN edge and rapidly be able to clone, burst and then destroy workloads anywhere around the world as needed. Yet Kubernetes deployment across multiple regions, sites, clouds and WAN edge remains problematic.

OpenEBS is the solution for latency sensitive stateful Kubernetes workloads that need universal data agility. A NVMF-based disaggregated storage physical fabric mates perfectly with the disaggregated data agility software infrastructure offered by OpenEBS. NVMF allows data to physically float across a single storage fabric while OpenEBS allows stateful workloads to be loosely coupled across multiple storage infrastructures. And the OpenEBS MayaStor project natively integrates to NVMF to simplify the developer experience.

Together, NVMF and OpenEBS become the dance floor for those Kubernetes-native application components to do their infrastructure-aware (and infrastructure-intelligent) ballet. Given its performance and scalability, however, perhaps this is better called the NVMF Boogie.

To learn more about containerized infrastructure and cloud native technologies, consider coming to KubeCon + CloudNativeCon NA, November 18-21 in San Diego.

— Charles Piercey
November 7, 2019
What Are AWS Application Patterns?

The cloud-native approach is not just about compute power and cloud services. A “set it and forget it” attitude won’t work with cloud native; there are far too many moving pieces and there is just way too much complexity. Instead, this approach requires constant testing, optimization and improvement. This is epitomized in the two key phrases of the DevOps methodology: continuous integration and continuous delivery.

To make this a reality, however, it takes a strategic approach to defining application patterns that will govern the way in which cloud-native applications are built and shipped, rather than a one-off effort. Since AWS is the leading cloud provider, we should discuss which application patterns are relevant to the AWS platform. There are many aspects to consider when it comes to AWS application patterns, and we will cover them here.

Operational Excellence

The best starting point for application patterns is a commitment to operational excellence. Typically, the term “Ops” is used to identify the IT team, but in this case, operational excellence rests on the shoulders of Dev, QA and Ops teams. In cross-functional DevOps teams, every member should strive for operational excellence at every step–even in the smallest of tasks.

In their white paper on operational excellence, AWS encourages its users to manage operations as code. This helps avoid human error, enables automation of complex processes and allows for quick and easy rollback when errors occur. Failure, according to AWS, is an opportunity for learning. They even advise performing “pre-mortems” that try to anticipate potential points of failure before a deployment.

AWS Trusted Advisor is a useful tool that enables this type of accountability and shared responsibility within a team. It checks and reviews your new and existing AWS workloads looking for opportunities to save costs, improve performance, tighten security or follow best practices for scaling operations.

Security

Security is a pattern that is non-negotiable. The term “DevOps” has inspired another variation called DevSecOps that signifies this fact. Security operations take center stage when talking about DevOps.

The interesting thing about DevSecOps is it actually results in more freedom, flexibility and trust across teams when done right. This is unlike traditional approaches to security which created silos, stifled creativity and slowed down innovation. Security is now a key piece of the puzzle, not an afterthought. AWS has many machine learning-based security solutions such as Macie which automatically identifies and secures personally identifiable information (PII).

Along with this, configuring role-based access controls using AWS IAM is essential to have greater control over who or which applications access data and for how long. Access Control Lists govern access to data stored in AWS S3 buckets.

The news is full of instances of data breaches due to weak security policies and access controls. Capital One is a recent example of a data breach that affected 100 million of its users. While security in the cloud is more complex than before, AWS has the necessary tools and systems in place. It’s up to organizations to implement a security strategy that prevents misuse of their data.

Reliability

Today, there is a low tolerance for downtime because of the potential loss of revenue. With the stakes so high, reliability is a pattern any application custodian would have high on their list of priorities. AWS has multiple reporting and health monitoring tools to keep you constantly on the pulse of your applications.

Architectural patterns like microservices enable better resilience by decomposing monolithic applications into more manageable chunks called services. This “divide and conquer” approach leads to higher availability, which is critical for SaaS applications.

Patterns like chaos engineering seek to intentionally kill random instances and services and force the DevOps team to figure out workarounds and backups for every scenario.

Application Performance

With the explosion of data, applications are always dealing with lots of data to transfer, store and process. An application’s performance depends on three factors: the backend data storage layer, the middleware components and the frontend.

The backend data layer is the most critical component when it comes to performance. Choosing the right storage formats, database types, storage devices and organization patterns all contribute to the speed of data access. AWS has an amazing array of storage services, including S3, RedShift, Aurora and Glacier. AWS gives its users every type of storage solution, all of which should be considered to improve data management.

AWS is devising new ways to query large datasets without bearing the load on your existing systems. RedShift Spectrum is one example of this. It is a great way to “minimize the need to scale Redshift with a new node, which can be expensive.” If you’d still rather not have to manage any RedShift clusters on your own AWS Athena is a great serverless alternative

That gives you all the power you need to query large datasets in S3 without having to maintain any clusters of your own.

This pay-per-use model is making its ways across the entire AWS stack with services such as ECS and Fargate offering completely serverless experiences when operating containers at scale. What started out with AWS Lambda is catching on fast, and many AWS services like the ones mentioned above enable serverless experiences with a focus on application performance.

Cost Optimization

When moving to the cloud, pricing is completely different. Rather than paying for perpetual licenses, you now pay a small fee for every small resource that you use. Though small on its own, you’ll soon realize that cloud costs can shoot up easily if you’re not paying close attention to your spending.

Cost optimization is necessary as you scale operations in the cloud. Cost optimization includes looking for more efficient services, whether that’s within the same vendor platform or a different platform. At a minimum it may require a lift-and-shift approach to transition from expensive old infrastructure to inexpensive new alternatives. For example, shifting workloads from VMs to containers to serverless within the same platform can show great cost savings, since containers and serverless are generally more cost-effective than VMs.

It pays to leverage the expertise of experts in this area. They should ideally be people who have spent years using AWS, understand Well-Architected best practices and have found ways to cut costs and derive more value from the same budget.

Application Patterns Enable Extensibility

Microservice applications adopt a pluggable architecture. This enables them to be easily integrated with external applications. Typically, API-based communication patterns are most common. AWS API Gateway is a service that manages API design for AWS applications. In some cases, an SDK is the better choice–for example, when you need more control over the developer experience and integrations. It’s important to know when to choose either option.

Application Patterns Enable Portability

Portability is the ability to move workloads across instances without interrupting their functioning. As instances change, the workloads which they support shouldn’t be affected. This is only possible if there is a clear separation layer between the workloads and the infrastructure that runs them.

Application Patterns Enable Shift-Left

The DevOps methodology is known to encourage a shift-left approach to software delivery. This means that not only Dev, but also QA and Ops contribute to the design and development stage of an application. This fosters deeper collaboration across teams and breaks down silos.

A shift-right approach holds true as well. Shift-right means developers are responsible for the code they write. They are equal owners with operations teams for the resilience of the apps they build and also the bugs they introduce into the system. This practice of putting ownership and responsibility where it belongs is very healthy for a DevOps team.

AWS facilitates pipeline management using tools such as CodePipeline, CodeCommit and CodeDeploy. They each perform similar but complementary roles and help build a CI/CD pipeline. They make it easy to get started with CI/CD within the AWS platform.

For organizations further down the line in their cloud journey, AWS’ pipeline management tools may seem too simplistic for something like GitOps. Instead, they’d want to look outside AWS for tools such as Jenkins X or Weave Flux which are designed with GitOps automation in mind. Whether it’s the simple or advanced route, a pipeline brings great benefits over a manual development process.

Conclusion

Though tools enable the implementation of helpful application patterns, on their own, they are ineffective. Application patterns need a deliberate strategy and an approach for a start. Only after a strategy is in place, can you move to deciding which tools to use. With a strong strategy in place, any of the tools mentioned here can be swapped out for other better tools and the results will be the same. It’s important to not be distracted by the next shiny tool but keep a keen focus on the strategy for implementing AWS application patterns.

As we’ve seen here, application patterns are diverse, but they all build upon each other. Without even one of these pillars, organizations would not be able to realize the goal of cloud-native applications. On the other hand, if they give careful thought to each of these application patterns, organizations would deliver cutting-edge applications that can only get better with the passing of time.

This sponsored article was written on behalf of Eplexity.

— Twain Taylor

November 7, 2019
DevOps at a Distance

How do you successfully follow DevOps practices when the two parts of your team are 5,000 miles (8,000km) and two time zones apart?

Airwallex is a global fintech founded in Melbourne, Australia just four years ago. It is Australia’s fastest-growth tech unicorn with a valuation of over $1 billion thanks in part to investments from companies such as Tencent Holdings, Sequoia Capital, Square Peg, Horizon Ventures, DST Global and MasterCard.

The business came into being when co-founders Jack Zhang and Max Li–who both worked in the banking sector–started a cafe in Melbourne’s Docklands area and realized that expensive foreign exchange rates hurt their profit margins when they imported products such as coffee cups.

Zhang was a software developer working in investment banking and foreign trading, and he knew technology could be applied to the problem.

Four years later, the company operates a network that allows SMEs to make easy and affordable payments to more than 130 countries and in 50 currencies.

Airwallex has 400 employees spread across offices in Hong Kong, London, Melbourne, Shanghai, Shenzhen, San Francisco, Singapore and Tokyo. That headcount includes 50 engineers in Melbourne and 84 in Shanghai.

“Communications between the two teams is incredibly important,” said Peter O’Donovan, Airwallex engineering lead.

So the approach has been to treat all of his people as if they were remote workers.

Tools

Extensive use of Slack–even when both people are in the same office–keeps everyone in the loop, he explained.

Similarly, Zoom videoconferencing is used for standups, and physical agile walls are replaced with digital equivalents.

These tools also allow for the growth of skills across regions, and help ensure everyone is “working to a common pattern.”

Such shared channels for collaboration and communication help accommodate overlapping working hours (Melbourne and Shanghai are three hours apart when daylight saving is in effect in Australia), as well as the need for engineers to be on call to deal with emergencies.

This approach puts everyone on a similar footing, whether they work in one of the offices or remotely.

DevOps leaders need to “focus on building relationships first,” he advised. Frequent video conferences are one way to build bonds with people in different locations. Understanding each others’ mindsets, establishing common ways of working and setting clear expectations are essential, and “doing it from day one is important.”

Scaling up

Airwallex plans to double the size of its engineering staff in both cities during 2020. O’Donovan points out that these established standards and practices help with the onboarding process and allow new hires to become productive more quickly.

That is especially true as the company has selected tools that most people will find familiar: in addition to Slack and Zoom, Airwallex also uses GitHub and G Suite. “These tools are easy to adopt and easy to consume.”

“Melbourne is a challenging market,” said O’Donovan, because DevOps skills are in demand and Airwallex wants to hire and retain quality talent. But there’s a virtuous circle–helping staff build a strong skill set and surrounding them with other talented people makes it a better place to work, so attraction and retention becomes less of a problem.

O’Donovan sees DevOps as a key to scaling the company. Having all the teams using the same tools and patterns yields economies of scale by avoiding unnecessary repetition of work, and it allows developers to focus on functionality rather than plumbing.

Furthermore, regulation and compliance is particularly important in the finance sector, and those considerations need to be built into the processes. “If you can do that seamlessly, you’re doing a good job,” said O’Donovan.

For example, making vulnerability scanning and the segregation of duties the defaults in the CI/CD pipeline addresses such issues in an automated and auditable manner.

You need “the right mindset for finance,” he warned. Building in compliance from the start works; trying to add it as an afterthought does not.

Don’t segregate people, advised O’Donovan, because everyone needs to be working collaboratively toward the same goal. With the right tools and automation, a cross-functional team can deliver better code, faster.

— Stephen Withers

November 7, 2019
Becoming a Data Company: A Tale of Two Trends

Data is back, baby! Well, that seems like an odd statement. It never quite left. Nevertheless, Google’s recent acquisition of Looker and Salesforce’s acquisition of Tableau seems to highlight this mindset in magnified ways. With eyes set on new leaps in data science and artificial intelligence, it’s easy to understand how our views on data now feel like a shifting pendulum.

The theme of the last decade was “every company is a software company,” driven by ready compute availability from cloud platforms. In the coming decade, it will be time for our data practices to catch up with the ongoing revolution in software development practices–or in other words–for “every company to become a data company.”

Over the past few months, as I have conferred with customers, attended conferences and connected with colleagues, there’s a perception of growing vigor in data conversations. Developing a data competency is rapidly growing in importance to business now, just as establishing a software competency became a priority 10 years ago.

However, my most curious observation has been the discussions around data and software are insular. Conversations either focus on how to move, manage, cleanse and leverage data for analytics, or how to build and release high-quality software quickly. The people who hold these conversations are either data people or software people who live in isolated worlds, seem to have divergent concerns and connect only at the lowest common denominator–the database storage–when the software guy or gal needs to store the data somewhere. There is little talk of the deeper connection and synergy between the two worlds and how it relates to larger trends.

The Next Wave of Digital Transformation

If you zoom out a bit, there’s more than meets the eye. There is an elegant symmetry between the software and data worlds, and it alludes to how the modern enterprise will morph in the age of digital transformation. Just as functions and organizations were reshaped to support new cultural and technology trends such as DevOps, organizations shall be reshaped in a data-first world, too. As software competency grows and matures, the need to acquire data competency becomes more urgent. What seem like separate trends are a maturing continuation of the uber trend of digital transformation, an attempt to acquire competencies in software and data (and necessarily so in that order).

As the diagram below shows, enterprises have been pursuing a journey of developing a software competency and becoming software companies. Those that make sufficient progress in software competency quickly realize that the significantly increased data they now have acquired requires new competencies to harvest and extract value from it. A data competency becomes necessary to completing digital transformation and so begins the journey to becoming a data company. Software and data are two sides of the same coin of digital transformation with the echoing resonance of reassuringly familiar themes. The diagram below highlights some of these.

Mirror, Mirror on the Wall

Speed and safety are central drivers in both competencies. The DevOps movement was driven by these two (often at odds) requirements to deliver software quickly from development to production without breaking the production deployment by ensuring that what is traditionally known as “non-production” accurately mimicked production. The goal is to build and test code so when it is deployed to production it doesn’t break what is already working.

In the data context, as more organizations understand the value of their data, they realize the importance of speed to get data quickly from where it is produced (production) to those that need it (outside of production, i.e. non-production). Safety in the data context is not just about not disrupting production while extracting that data, but also about meeting regulations and remaining compliant with data privacy policies in the process.

The elegant symmetry here is further accentuated by the fact that while the primary concern in the software context has been around getting software from internal development environments to customer-facing production environments, in the data context, the concern is getting data from customer-facing production environments to internal non-production environments. This completes the virtuous cycle of experimentation and learning that modern enterprises must master (see below) in their journey of digital transformation. It requires that organizations rapidly iterate to product-market fit through trial (software release) and learning (data science, analytics, machine learning) and back again.

In this way, digital transformation becomes a virtuous cycle driven by the complementary movements of DevOps and DataOps–the cultural movements, philosophies, technologies and collaborations that underpin the software and data journeys respectively, and the primary drivers of organizational metamorphosis that enterprises are undergoing during the digital transformation journey.

— Karun Bakshi

November 6, 2019
Using Open Source for Better DevOps Outcomes

Many insurers have jumped on the DevOps bandwagon in order to improve the efficiency and effectiveness of their systems and systems implementations. Very few insurers, however, spend much time thinking about the open source software community as a way to improve their DevOps approach–and the skills of their developers–and in that regard they’re missing a golden opportunity. In fact, insurers often restrict their developers from updating or contributing to open source libraries because they don’t realize the potential benefits available to them by participating more actively in the open source community.

A 2016 survey by 451 Group indicated that the more companies used open source solutions in their DevOps portfolios, the more effective the DevOps function was. More specifically, the study reported that among companies with at least 60% of their DevOps portfolio as open source, a full 54% had reached the “fully deployed” DevOps state, with all of their development occurring via DevOps models.

By comparison, only 30% of the surveyed companies that used less than 60% of open source tools in their DevOps portfolios had reached this state of DevOps adoption. Clearly there’s a link between open source use and DevOps success. Given these results, insurers should focus on the benefits of supporting and encouraging their DevOps groups to embrace the open source community.

It’s a dated but still persistent view: Participating and contributing to the open source community means developers are giving away their time by creating and contributing intellectual property (IP) that may benefit others outside the organization. However, the truth of the matter is participating DevOps engineers, developers, analysts, etc. are often fixing bugs and making improvements to code libraries other than their own but that can have value for their own DevOps efforts.

The very definition of open source –software for which the original source code is made freely available and may be redistributed or modified–makes it fertile ground for skills and solutions improvement. Additionally, that process of analyzing and repairing code represents some serious skills development. It’s the kind of development skills that can be difficult to obtain by just working on in-house applications and platforms–which are often proprietary and dated–lacking modern coding techniques.

Since open source literally means a developer and/or organization has the ability to make the code fit their needs however they see fit–they control their own version of the product that may diverge from the original open source project–the opportunity for DevOps solutions improvement is substantial. Moreover, any DevOps related IP a developer creates and contributes is not lost and is certainly not stolen–the IP can be used directly by the developer and that developer can continue to benefit from the IP of other developers on a gratis basis.

Utilizing DevOps solutions from the open source community can be both time and cost effective and can be a positive boon to organizations. While some insurers may not knowingly allow open source code in core systems or applications, its use in a DevOps ecosystem is another matter. That’s especially true since DevOps is a newer and less mature IT function that requires new tool, process and solutions development. Leveraging open source solutions can expedite that process. In fact, many of the key DevOps tools used today either are or started as open source solutions for DevOps problems. One would be hard pressed to find a DevOps development stack–from building/developing to monitoring–that didn’t use at least one open source tool as part of the stack.

Often the fear or apprehension to using open source libraries (for DevOps or anything else) is that anyone could have written it, thus its quality is or could be sub-par. While in some cases this can be true, the same holds true for closed source proprietary software, as many insurers could attest. Simply put, open source tools and libraries are worked on by developers of the exact same caliber–if not the exact same people–who work on closed source proprietary tools and libraries. The only difference is there are often less people involved with proprietary solutions development. And since there are relatively few proprietary DevOps solutions in the marketplace to choose from, open source should be considered a viable channel for DevOps solutions.

Even if there were more choices for proprietary DevOps solutions, the financial savings involved in combining open source solutions with DevOps can be significant. Importantly, open source solutions for DevOps are particularly useful when the right solution for a problem has yet to emerge, almost a given in a maturing DevOps ecosystem. Rather than rushing out to buy something that promises to fix the problem–or writing something themselves–DevOps developers can find open source solutions and use those to experiment, test and otherwise isolate and identify the true nature of and solution to a DevOps challenge.

Another reason DevOps groups should avail themselves of open source techniques and tools is the fact that the tools and techniques available for DevOps–proprietary off the shelf and built inhouse–are still not quite there yet. The tools available are almost always missing something, resulting in developers at many organizations spending most of their time–and the organization’s money–trying to develop fully functional solutions from the ground up.

More often than not, the resulting tool or solution is never as complete or well-built as the open source tool or solution that already exists. Most organizations would be better served tasking their developers with scouring open source libraries, identifying the “best fit” DevOps tools and then spending the time required creating the updates and improvements to the tools that have already been mostly built, saving time and money.

Finally, there are lots of DevOps staff out there who participate in the open source community–in their spare time anyway, or have done so while working on previous projects. That means hiring DevOps staff with open source experience is a shortcut to potential open source techniques and tools. These are people that can hit the ground running by bringing their open source experiences to bear while they’re getting up to speed on the proprietary work they may be assigned. Considering these benefits, it’s past time for DevOps groups at insurers to dive into the open source waters.

— Tanner Gibson

November 6, 2019
IoT, AIOps and OMG!

The task of keeping networking straight while expanding the definition of network to include both physical and virtual networking has fallen heavily upon the op-centric side of DevOps–or what I like to call devOPS. What is defined as networking has grown, sometimes at a ridiculous rate. While the terms are familiar, how you manage them in truly virtualized environments is different than traditional hardware management. More agile, but different.

And that means those who deal with networking in DevOps have to know more, manage more and debug more–all while working on other parts of the DevOps toolchain. It’s been a slog through increasing capability with increasing workload.

There is a confluence of new and different things that promise to set this lopsided progress right. We’ll look at them individually, then talk about the sum of the parts.

Internet of Things (IoT)

For most of us, when we hear IoT, we think of millions of devices slamming data into a centralized service. iPhones, for example, returning information to Apple. Or highly specialized devices such as utility meters that report far more frequently than humans could go out and read them. But IoT is many things, and increasingly IoT is networking devices. They hold a wealth of information about the state of the network at any given moment, and that information is increasingly being funneled into APIs that IT or management apps can take advantage of. Considering that some of those management apps are also getting data from application tooling and server health status, there is a wealth of information coming together in one place.

AI Operations (AIOps)

A networking device can be doing a lot in any given second, and generating data as it works. Servers can be on their way offline when they feed the information out, or maybe sitting idle and wanting more work shipped their way. Application bottlenecks might be identified in the wealth of application tooling information, but buried under the wealth of other response data being shipped back. AIOps aims to help the devOPS side deal with these issues by limiting what makes it to human attention and drawing correlations between issues. Finding that an over-burdened network connection is associated with an apps performance degradation is an easy one to talk about, even though it is pretty simplistic. AIOps can make the connection before a human gets involved, and then the staff member can decide what to do about the actual root cause.

It’s not at all perfect yet–we’re not generally talking about AIOps making a ton of decisions, rerouting traffic or spinning up new instances to resolve issues at this point–but it is definitely in the “useful tool” category. The cost in person-hours to analyze and reduce problems is definitely cut by the use of AIOps as it exists today, and that benefit will only grow moving forward.

The biggest problem is the one that bedevils all data interchange, and in fact is the biggest issue in the world of big data–data standardization.

Enter OMG

No, my OMG in the headline wasn’t what you thought. It was the Object Management Group (OMG). They’ve been around forever, and have done a pretty good job of developing useful standards, particularly in the area of data interchange. I’ve followed them for a long time, and used their standards since … well, anyone remember CORBA? That was my first set of OMG standards.

Now they’ve started an Artificial Intelligence Platform Task Force (AIPTF). This group will be looking at all areas of AI to set standards, but the interesting bits for this blog will be in data standardization and communication. Standardizing data and how it is communicated is the next big step in this process, and hopefully the OMG will set us down that path. They’re not specifically looking to address infrastructure big data needs, but their work will likely guide AIOps moving forward. And there is huge potential in that movement. If AIOps gets some standards that help it to be more proactive, we may actually end up with auto-fixing algorithms for obvious and easy issues. We already have the systemic ability in container managers to increase the number of instances serving an application if the existing ones get over-loaded, if we could expand that type of auto-growth to include networking issues and consider cores used by instance to find the best solution. Members of the team that are more devOPS will have more time to focus on new development. And that’s a huge plus.

The Future is Bright

Whether the work at OMG benefits us will depend upon whom is involved in the various subgroups they have planned to implement, and cooperation with other ongoing efforts at standardization. Yet I’m hopeful that, like with that original CORBA spec, they will provide a roadmap that others can follow to put us on the path to the much-maligned self-healing network. Giving devOPS a chance to keep up, while leaving them ultimately in control of what happens. Current iterations of AIOps do this to a lesser extent, but more inputs and more automated analysis of those inputs will only make AIOps more powerful. Standardization will make AIOps more able to make use of data and better focused on their specific problem at hand. I see the confluence of all three trends to be a huge positive to the industry if we manage to take advantage of it. And some vendor out there will see the potential, taking advantage of it for us, we just have to find them and use their solutions.

Meanwhile, paw through the data available to you by hand again, and keep knocking those walls down.

— Don Macvittie

November 6, 2019
When Gateways Go Away with Cellular IoT

Gateways have been an essential piece of three-tier IoT architecture for years. Their traditional spot between the edge and the cloud is in flux as cellular IoT technology rolls out. When are gateways still the right approach? How will gateways go away with cellular IoT? Here’s the trade-off between traditional three-tier and cellular IoT architecture for DevOps teams.

Spanning the Gaps

IoT edge devices and enterprise networks are often incompatible. Enterprises and clouds run on IP packets and heavier protocols such as HTTPS, TLS and XML. It takes processing power to run these–power often beyond the reach of smaller battery-powered edge devices. Even with improvements, Wi-Fi is enterprise network compatible but too heavy for many edge devices.

That’s why wireless sensor networks such as Bluetooth, Thread, ZigBee, Z-Wave and others evolved. These compact, efficient stacks run on tiny microcontrollers, sipping power as sensors send data in small bursts. Alongside wired control networks are more wireless technology such as 6LoWPAN, ISA100 and WirelessHART. Real-time protocol conversion in a gateway brings data from these devices into IP-based enterprise networks.

In some cases, a gateway might be as simple as a smartphone, tablet or a small board. Some gateways have faster processors, AC power, better cooling and packaging for harsh environments. Faster gateways can also pre-process data streams in an edge computing role. This reduces network traffic and speeds up decision-making without relying on cloud analytics.

Setting up a few gateways inside an industrial facility is easy. Clusters of far-flung sensors call for many gateways spanning the gaps. As IoT applications grow, gateways carry development and deployment costs. Every time another gateway drops, DevOps must secure and update it.

A Towering Alternative

There was no alternative to three-tier IoT architecture until recently. Wireless sensor networks appropriate for edge devices lack range for many use cases. What if instead of extending device range one could extend network coverage? Low-power, wide-area network (LPWAN) technologies take on that challenge.

Three main classes of LPWAN technology exist today: Sigfox, LoRaWAN and cellular IoT. Sigfox is a low data rate uplink solution connecting devices and Sigfox towers within 30 to 50km in rural areas. LoRaWAN uses unlicensed spectrum limiting its duty cycle to around 1% and requires a wireless gateway for a five to 7km rural range. Urban ranges are shorter for both technologies, still a few kilometers. Both work for sensors that mostly sleep and transmit small amounts of data.

Cellular IoT technology got a late start, with good reason. Specifications for faster, yet low-cost, low-power radios with cellular-grade range and security was a massive undertaking. The 3GPP defined two paths: LTE-M and NB-IoT. LTE-M offers a 1Mb/s data rate and lower latency. NB-IoT concentrates on lower data rates without duty cycle limitations. With about 200kb/s data rates up and down it is still much faster compared with Sigfox or LoRaWAN. NB-IoT also penetrates buildings better, avoids interference and targets a 10-year battery life in devices.

Mobile network operators (MNOs) worldwide are racing to get NB-IoT and LTE-M networks deployed. In the latest GSMA Mobile IoT Rollout Report, NB-IoT is leading in adoption. These rollouts are not tied to expensive 5G infrastructure, so MNOs are investing incrementally.

After towers are ready, the last piece is support for Remote SIM Provisioning (RSP). Cellular IoT devices typically work with an eSIM like in the latest Apple iPhones, with one major difference. An iPhone user scans a QR code, pulling a profile over RSP.

Pushing Across Networks

Cellular IoT devices use a push model for RSP. When an edge device with an eSIM powers up, it uses a bootstrap network profile. It can then communicate with an RSP server to load an operational profile of choice. With this over-the-air profiling one device configuration can go practically anywhere in coverage. It opens powerful use cases such as agriculture or oil and gas, with wide-spread sensors in both indoor and outdoor settings.

All this happens without a DevOps team planning and investing in infrastructure. MNOs are making the major investment. The variables for IoT application developers are network coverage, operator profile creation and setting up devices for streaming. If one MNO proves costly or unreliable in a specific location, devices can receive another profile over the air.

Cellular IoT can deploy one sensor, or many, without a gateway. The cellular IoT advantage becomes even stronger with more dispersed locations. A simplified network architecture looks more like the IoT should look–with every device addressable.

Alternatives exist because they serve some use cases well, with some persisting for years after new technology arrives. For some DevOps teams risk is a strong factor, and they may prefer the control a gateway-based architecture offers. In tight sensor clusters, the life cycle costs for a few gateways may turn out lower. If gateways aren’t cutting it for an application, they may go away quickly with cellular IoT rollouts.

I see ubiquitous coverage driving cellular IoT momentum. Providing IoT services is a priority for mobile network operators as consumer growth flattens. There are plenty of resources for understanding cellular IoT technology in depth–get in touch with me if I can help point you in the right direction.

— Don Dingee

November 5, 2019
The Challenge of Scaling Metrics in Distributed Systems
Whether migrating from a monolithic architecture or building a distributed system from ground zero, there are many benefits to leverage from a microservices architecture–faster software deployment cycles, enhanced scalability, improved isolation of risk. However, distributed systems are complex and as such, accompanied by many challenges, particularly when it comes to the scalability of metrics.

The Importance of Observability in Distributed Systems

In order to realize the benefits of a distributed system, a strategy of data-driven optimization is essential. The paradox here is visibility simplifies complexity, while the complexity of visibility is dramatically increased in distributed systems.

As New Relic points out in their eBook Cloud Native Is the New Normal, traditional monitoring approaches need to be adapted to meet the demands of decentralized architectures, including:
- Overcoming the loss of end-to-end monitoring capabilities by learning how to connect the dots across a complex network of environments.
- Developing tactics to quickly drill down into issues across interconnected parts to reduce mean time to resolution (MTTR).
- Introducing proactive reliability strategies that focus on measurement first, followed by improvement.
Finding the Right Balance Between Monitoring Nothing and Monitoring Everything

Most application engineers leverage some sort of metrics tooling such as Grafana, Prometheus or Graphite to track vitals for their system health. While monitoring is essential, there is such a thing as bloat when it comes to metrics ingestion; and this overload translates to scalability challenges that affect performance, cost and reliability.

So, how do you know what to monitor? In order to derive the most valuable insights from your system, you must first understand and identify the components, and more specifically which attributes within those components drive key metrics. Knowing this requires an intimate familiarity with how all the pieces work together in your distributed system.

Google’s Site Reliability Engineering book provides a framework for determining what to monitor in distributed systems. They break signals down into four different categories:
- Latency: Understand resource requirements and identify bottlenecks.
- Traffic: Monitor system response to varying levels of load; most useful when paired with other indicators.
- Errors: Track frequency and nature of interruptions in request/response flow.
- Saturation: Surface capacity-related issues.
Even when you’ve identified the most important metrics to track, scalability is still a challenge that must be addressed, particularly when dealing with applications that handle massive volumes of data across a complex application architecture.

M3: Uber’s Open Source Large-Scale Metrics Platform

A solution directly addressing the challenges of distributed metrics at scale is M3, an open source project developed by Uber’s engineering team.

Before using M3, the Uber team sent metrics to a Graphite stack with storage handled using the Whisper file format in a shared Carbon cluster. From there, they created Grafana dashboards to visualize the data and used Nagios for alerting based on Graphite threshold checks via source-controlled scripts. As the volume of metrics continued to expand, manual processes to scale the solution became unsustainable and repeat failures resulting in permanent data loss led them to build a more fault-tolerant, scalable system.

M3 is focused on providing a reliable storage solution for large-scale metrics over long retention periods, solving inherent single node-associated limitations with Prometheus by offering a multi-tenant store for Prometheus metrics. A key component of this solution is M3DB, the scalable storage backend for M3.

M3DB, built in Go, is a distributed time series store and reverse index with configurable out-of-order writes. By enabling the storage of real-time metrics over long periods of retention, M3DB can replace a Prometheus cluster. In developing the M3DB solution, the Uber team essentially deconstructed Prometheus into microservices to enable the scaling of each component individually. By taking this approach, M3DB is able to offer better reliability, efficiency and performance.

At a minimum, M3DB can be leveraged as remote storage for Prometheus (or used to replace any other remote storage solution). There are, however, other features the solution offers, including an external aggregation engine and a separate query engine.

The M3DB project is still under development with a growing community of contributors.

To learn more about containerized infrastructure and cloud native technologies, consider coming to KubeCon + CloudNativeCon NA, November 18-21 in San Diego.

— Molly Wojcik
November 5, 2019
Breaking Up and Making Up: Composable Persistent Storage for Kubernetes

Kubernetes is rapidly emerging as the standard for application management in the data center. Kubernetes defines applications in a container model for deployment and delivers most of the software-as-a-service features of virtualized infrastructure at a lower cost and with less performance degradation. In this article we’ll cover the intersection of Kubernetes and scale-out data-intensive applications, what problems Kubernetes solves for those applications and why composable infrastructure provides the remaining piece for a total solution.

For heavy-duty data-intensive applications, such as Spark, Kafka, Aerospike, Vertica, Cassandra and Elasticsearch, ML-model training/service with TensorFlow, Kubernetes promises a common application orchestration solution. These applications are termed scale-out because they allow additional performance and capacity to meet any use case simply by adding more nodes. The challenge introduced by these applications is managing large production deployments of each application, compounded by managing multiple applications each with their own configuration requirements. Kubernetes provides an answer here for managing these disparate data-intensive, scale-out applications with a common container orchestration platform.

Scale-out applications are distinguished from typical enterprise applications in three ways. First, scale-out applications take a divide-and-conquer approach to problem solving: if you have a bigger problem simply deploy additional capacity and performance by adding nodes. Second, since performance and data capacity are simply increased by adding building blocks, you can exploit cheap commodity-off-the-shelf (COTS) nodes, avoiding the money pit that is the case in big iron hardware purchases required by traditional scale-up applications. Third, the data-intensive applications typically integrate an optimized storage stack within the application itself that effectively uses the cheap local storage available within the COTS nodes. The last point has become a problematic area for scale-out application deployment as overprovisioning is now common to overcome the difficulty in predicting workload behavior at initial deployment time.

Composable infrastructure tackles the last problem presented by embedded local storage for data intensive applications. Looking back over various transformations in the data center, one thing the legacy enterprise storage vendors got right was separating compute and storage. Both during acquisition and throughout the application lifecycle, separating persistent storage components from compute nodes running the actual application yields opportunities for cost efficiencies and flexibility.

Composable infrastructure brings the capability of separating compute from storage for these scale-out applications, which sport their own embedded optimized storage software stack because it does not dictate a new storage architecture, it simply says “take your existing servers with embedded local storage and disaggregate them”–placing the storage on the ubiquitous high-speed low-latency fabric you’ve already deployed for your applications: Ethernet. From the application perspective in a composable infrastructure, storage still appears local and the application runs as it did before.

In this way, applications run unchanged and the infrastructure they run on can be expanded, reduced or replaced in real-time. This is true software-defined infrastructure where infrastructure is treated as code. Being able to modify the node configuration ratio of compute and storage dynamically avoids the issues of purchase time lock-in of possibly under-provisioned nodes (which has simply led to overprovisioning). Composable infrastructure eliminates the downside of purchasing static resources that define the resources available before the workload is run.

Kubernetes launched not long ago in 2015. Many of the early adopters of Kubernetes focused on stateless applications that did not depend on persistent storage to deliver useful services. It is only recently that support for persistent storage emerged (an obvious requirement for data-intensive applications). After a couple of approaches to persistent storage integration, the key players in Kubernetes (and other container orchestration platforms) settled on an orchestration-independent persistent storage API, the Container Storage Interface (CSI). The CSI provides a standard method for storage providers–and composable infrastructure providers–to define and attach persistent storage to containers.

Why should you be interested in composable infrastructure when considering how to deploy your data-intensive applications on Kubernetes? Because Kubernetes only provides application and container orchestration–it does not fundamentally change the way the application works. These data-intensive applications were defined to exploit, and are most efficiently deployed, over COTS node and commodity storage. Each application still has its own storage stack, each carefully optimized for their particular workload. It is likely that deploying storage appliances, or software-defined storage solutions that aggregate pools of HDDs or SSDs, under data-intensive applications will only result in degraded performance and unnecessary costs.

A composable infrastructure solution is only useful as an underlayment for a Kubernetes deployment if it meets several requirements.

First and foremost, the composable infrastructure must scale to 100,000s of devices in support of the applications being deployed on top of the platform.

Second, the composable infrastructure platform must not destabilize or diminish Kubernetes capabilities and scale accordingly with commodity networking in mind. The composable infrastructure solution must share a similar design philosophy with Kubernetes, allowing it to scale. Kubernetes implements a distributed desired state model where work is distributed amongst the mostly autonomous nodes to reach a desired deployment configuration. Such an approach provides scaling on the order required by hyperscalers. You must ensure the composable infrastructure solution you choose provides a similar scalable approach.

Third, Kubernetes emphasizes a configuration state that is durable in the face of any individual node failure or network partitions. Durable configuration state in the face of large, highly interconnected commodity component deployment is critical for correct automatic operation at scale of business-critical applications, and your composable infrastructure solution must similarly support this.

Fourth, Kubernetes defines a model for application deployment in terms of groups of containers called Pods. For a manageable persistent storage solution, the support for CSI and Kubernetes must provide a native Pod view of the storage bindings since the Pod (the managed set of containers) is the proxy for an application node instance and the unit of expansion and migration.

Kubernetes has ushered in an exciting new chapter in data center architecture by providing a lightweight, richly featured application software orchestration platform. Think of Kubernetes as the ideal software application orchestration platform. What is missing is the scalable hardware orchestration solution that supports the dynamic model for application orchestration that Kubernetes defines. A feature rich, well-integrated composable infrastructure platform provides the needed hardware orchestration to enable the application migration and elasticity promised by the Kubernetes solution.

— Brian Pawlowski

November 4, 2019
AIOps: The Path to Greater Resilience and Uptime

Large organizations with complex IT infrastructures are facing a significant challenge when it comes to data. The complexity is such that many IT operations teams are struggling to detect, or predict, the complex failure conditions and circumstances that can adversely impact resilience and uptime.

Monitoring a multitude of systems is a complex task, as the data generated by any system can be vast and far too overwhelming for manual IT operations processes, particularly in relation to analyzing, correlating and prioritizing alerts and remediation.

Companies have much to gain from a better understanding of the data being created from monitoring tools and practices: increasing the resilience of systems, and the speed of remediation cycles delivers significant cost and time savings. Harnessing advanced technologies to deliver more intelligence to teams in charge of performance monitoring is becoming increasingly important, which is where AIOps can deliver significant value on top of existing monitoring and automation tools.

Cutting Through the Noise

Telemetry tools deliver staggering amounts of data. The Microsoft Azure telemetry platform, for example, records 10 petabytes of data per day. When dealing with this volume, the ability to intelligently decipher the severity of alerts becomes vital, especially as many of them may be duplicates or not meaningful in any way. Yet it is IT operations teams that are tasked with understanding how systems are performing, and where resources should be diverted to ensure services are always up and running. To do this effectively, the root causes of alerts must be identified quickly and correctly.

AIOps is increasingly becoming the go-to solution to the problems inherent in dealing with big data. By applying data collection, data modeling and data analytics techniques and using machine learning algorithms to establish patterns, the noise of the data is reduced as operations teams gain more intelligent insights into alerts. So, providing more context to issues flagged within systems is the goal. Eventually, somebody has to take action when an alert is created, so the more intelligence engineers have, the easier it will be to remediate.

The less context delivered with alerts, the more time engineers will need to resolve the issue, as the manual remediation process will likely involve interaction with other teams. Take application performance monitoring as an example. In my experience, operations teams do not have an end-to-end understanding of the applications that are live in the production environment. If an alert is created, the ops team will typically seek assistance from a member of the developer team who built it, or another colleague with a deeper understanding of the application, which of course extends the lead time of the remediation process.

Key Considerations for Deploying AIOps

By establishing data patterns through machine learning and other advanced analytics technologies, the following key drivers for alerts assessment using AIOps can be established:

Frequency: Discovering how regularly alerts are created and whether it warrants automation for analysis, correlation, prioritization and remediation.

Coverage: Telemetry is of course vital to AIOps, as the machine learning algorithms will only be as good as the data they ingest. Putting in place the right telemetry tools is therefore vital to ensure the right data is being gathered. Failing to do so increases the risk of missing critical issues.

Impact: Not all problems require equal energy and resources. Certain issues might only occur every few months and may not be critical. In such situations, it is not prudent to invest significant amounts of time and money into developing the machine learning algorithms that will identify where and when an alert will be created. Again, cutting through the noise of the telemetry will help to identify high-priority alerts.

Probability: What is the likelihood of certain issues recurring? Can automation tools, informed by the machine learning algorithms, be implemented to deal with these, i.e. learn where and when they are likely to occur so they can be dealt with by the AIOps platform autonomously?

Take, for example, the issue of CPU or memory going beyond a certain threshold. Rather than manually contacting customers when an issue is flagged to inform them that services are about to go down before being spun up again, an AIOps platform can automate this process, so remediation happens seamlessly.

Latency: Automation of the remediation process for an alert deemed critical requires increased investment in AIOps. Getting good data from the early stages of AIOps processes will reduce the amount of time taken to both flag and deal with a problem. Containerized microservices, for example, take very little time to recover, so very little investment is needed when applying AIOps tools that automate remediation. Database recovery, however, is a more complex process, the automation of which would require significant investment.

What’s the Customer Value?

For service providers, AIOps is essential if they are to attain the highest levels of resilience, the measure of which all comes down to the Service Level Agreement (SLA). The SLA is measured by a percentage that corresponds to the time systems and are up and running over a year. So, 99.99% uptime is good, 99.999% is great, but 99.9999% is where service providers want to be. To put this into context, downtime for an entire year in the latter scenario amounts to just 32 seconds.

Increased automation is the only way to achieve optimal SLA. To do this, service providers are increasingly turning to AIOps solutions.

Operations teams are in charge of the IT strategies that align with business objectives. For service providers, a key part of this is, of course, ensuring maximum availability and optimal performance of the customer-facing environment. Understanding when and where faults appear, and improving resilience against them, is the best way to ensure this. That is where AIOps truly adds value.

— Prashant Jain

November 4, 2019