devopsy.kinsta.cloud

Category: Doin’ DevOps

A History of Distributed Tracing
Organizations are increasingly using distributed tracing to monitor their complex, microservice-based architectures. Distributed tracing has become essential in microservice applications, cloud-native and distributed systems. Microservices and serverless applications can grow exponentially, which makes observing them at scale very challenging. The traditional logging method becomes expensive, and if there’s an issue, time-series data can reveal symptoms but not the root cause(s). With modern usage patterns of Kubernetes service mesh like Istio and Envoy, for example, there can be billions of microservices calls daily.

What is Distributed Tracing?

Traces describe the progress of a transaction or workflow through a system. Distributed tracing provides visibility into service dependencies and their interdependencies by providing end-to-end visibility. Visualizing transactions in their entirety allows you to compare non-normal traces with normal ones to determine differences in behavior, structure and timing. Through distributed tracing, you can identify bottlenecks in your system. As a traced request flows through a network, it can be tracked.

As requests flow from frontend to backend and to databases, traces provide diagnostic techniques that reveal how a set of services coordinate to handle an application request. Using distributed tracing, developers can troubleshoot requests with errors and high latency. A single trace typically shows the activity of a request or transaction within an application. A trace shows what has changed at each step. As you aggregate the traces, you can see which backend service or database has the most significant impact on performance.

The History of Distributed Tracing

Dapper, a large-scale distributed systems tracing infrastructure, was introduced by Google in 2010. Two years after Dapper was made public, Twitter open sourced Zipkin, which was designed for application performance tuning. Zipkin was the first open source distributed tracing project. Zipkin trace data can be collected and visualized using a UI. In 2015, Uber launched Jaeger, which was inspired by Dapper and named after the German word for hunter. Later in 2017, Uber published a blog post, called Evolving Distributed Tracing at Uber Engineering, explaining the reasons for the architectural choices in Jaeger. In addition to creating Jaeger, its author, Yuri Shkuro, wrote a book about distributed tracing called Mastering Distributed Tracing.

In 2016 Ben Sigelman, founder of Lightstep, wrote a blog post called Toward Turnkey Distributed Tracing, describing OpenTracing as a single standard. Some people refer to this as the OpenTracing Manifesto. OpenTracing allows developers of application code, open source packages and open source services to instrument their code without binding themselves to any particular tracing solution. The goal of OpenTracing was to solve a standardization problem. Trace context must pass through all the components, including application code, dependent libraries, standalone open source services (Nginx, MySQL) and other vendor-specific libraries and services, to collect a complete distributed trace. Collecting a full path without a standard API to define the collection and passing of trace context is difficult. OpenTracing aims to solve this problem by defining a standard API that can be implemented by components from different tracing solutions, allowing the collection of end-to-end tracing data. The Cloud Native Computing Foundation (CNCF) accepted OpenTracing as its third project in October 2016. Two months later, OpenTracing 1.0 was released. After OpenTracing, Jaeger joined the CNCF.

W3C tracing context specification was proposed in November 2019, bringing distributed tracing closer to standardization. Over ten years, distributed tracing evolved from one paper to an active community. With standardization on all the layers, it is moving from just tracing to overall observability, ranging from latency optimization to root cause analysis and application performance management. It is moving from a single backend system to an end-to-end solution that spans multiple systems.

OpenTracing and OpenTelemetry merged in 2019. Using OpenTelemetry, distributed tracing can be implemented end-to-end. It released version 1.0 in 2021. CNCF’s OpenTelemetry project is one of the fast-growing projects, and OpenTelemetry has become the de facto standard for tracing, metrics and logging.

How it Works

Traces in distributed tracing consist of tagged time intervals called spans. It is possible to think of a span as a unit of work. Traces are directed acyclic graphs (DAGs) composed of segments known as spans. According to OpenTelemetry, There are four major components of a trace:
- traceID
- - A unique 16-byte array to identify the trace that a span is associated with
- spanID
- - Hex-encoded 8-byte array to identify the current span
- Trace Flags
- - Provides more details about the trace, such as if it is sampled
- Trace State
- - Provides more vendor-specific information for tracing across multiple distributed systems. Please refer to W3C Trace Context for further explanation.
The spans in a trace consist of contiguous segments of work that are named and timed. There are start and end times for spans, as well as metadata such as tags or logs that help classify them. Parent-child relationships may be present between spans to show how particular transactions traverse the application’s numerous services and components.

A trace represents an end-to-end request; it can contain one or more spans. Spans indicate work performed by a single service with associated time intervals and metadata. The purpose of traces is to provide a request-centric view of spans through tags. Despite microservices enabling teams to work independently, distributed tracing offers a central resource that allows all teams to understand issues from a user’s perspective.

Conclusion

In distributed systems, response latency can have a significant commercial impact. Identifying bottlenecks and understanding how a request moves through a complex system is complicated. Other techniques, like logging and monitoring metrics, don’t provide insight into distributed applications such as those created with microservices architecture. Several open standards and tools are emerging within the distributed tracing space, like OpenTracing and commercial tools that could compete with existing APM solutions. Implementing distributed tracing for modern cloud-native services poses several challenges. OpenTelemetry tracing is necessary for processing high volumes of trace data while generating meaningful insight.
December 6, 2022
What Should Business Owners Know About SSDs?

Without a doubt, SSDs have become the latest trend in hardware storage, and you can expect their use to increase over time. Implementing SSDs in your business can improve productivity and make it more efficient by increasing the speed of your PC long enough to extend its useful life. Here is all you need to know about using an SSD for your business.

What is an SSD Storage Device?

An SSD is a storage device that functions similarly to an HDD. But instead of having moving parts, an SSD stores data on interconnected flash memory microchips that continue to store information even when there is no power flow. These microchips are known as NAND and are more reliable and faster than the HDD.

SSDs are fast because they use the Non-Volatile Memory Express (NVMe), which allows high data exchange speeds of up to 2,600 MB/s, almost five times faster than a standard SSD. The NVMe SSDs are more recent than SATA SSDs and use the peripheral component interconnect express (PCIe).

SSDs Vs. HDDs

SSDs and HDDs differ in several ways, and you should understand the fundamental differences between the two. HDDs are mechanical and use magnetic heads to read and write data while spinning the platters. The number of rotations per minute determines the speed of an HDD.

Meanwhile, SSDs are electronic and use flash memory chips, which makes them faster than HDDs. They also use NOR or NAND flash memory as their storage medium. The NAND is smaller and faster than NOR in read/write operations and can store data when off.

What is the Lifespan of an SSD?

Cells wear out faster when they write more data, and a standard SSD cell has a lifespan of around 3,000 write cycles, which may seem small at first glance. But the SSD uniformly distributes the write operations across all cells, reducing the chances of the cells dying and increasing the drive’s longevity.

SSDs also have backup cells that regenerate dead cells. For this reason, large SSDs have an extended lifespan.

How SSDs can Help Your Business: Speed, Longevity, Energy

SSD is ideal for businesses that handle and store vast amounts of data due to its high performance, speed, and reliability. Here are more reasons why you should use an SSD for your business:

1. Faster Read/Write Speeds
SSDs are faster at accessing your business data, reducing the startup time for programs and operating systems. Compared to a traditional HDD, an SSD runs up to 30 times faster, allowing you to boot your PC in seconds instead of minutes.
2. Longevity
An SSD may be suitable for upgrading drives in a portable PC. It has no spinning drive, which means better durability and a lower likelihood of damage or losing data when you drop your business laptop.
3. Energy Saving
When it comes to going green and reducing energy consumption, SSD storage devices should be at the top of your list. SSDs usually generate less heat and energy since they don’t have moving parts. Therefore, you will save substantial power when using your PC, and more battery life means more productivity.

Challenges of an SSD Storage Device

It has always been expensive to store enterprise-level data on servers, with SSDs being slightly more expensive. As business storage capacity increases, SSDs become more expensive, and an HDD may be suitable if you want to archive a significant amount of data. Apart from archiving data, SSDs are generally the best option for storing and transferring data.

Bottom Line: Speed, Reliability, Improved System Cooling

Modern businesses cannot function without storage devices, and an SSD offers a unique solution that significantly increases productivity by reducing loading times. An SSD promises faster system performance, increased reliability, and improved system cooling. Due to these advantages, your business will save both time and money.

December 5, 2022
How to Select the Right Volume Type/Storage for Your Workload

AWS services are at the top of the cloud services food chain, and a hot topic for nearly every business considering migrating to the cloud. Along any cloud migration journey, many hidden traps await the unprepared. The volumes you’ve already provisioned might seem like a perfect match—but are you sure they suit your workload and carry the best price for performance delivered? Today, we’ll make the difficult task of choosing a storage class for your workload a bit less complicated.

In the World of EBS

AWS EBS currently provides three categories of considerable storage: Hard disk drives, general purpose SSD, provisioned IOPS, and one that you will most likely never need to use—magnetic. Although I’ll provide brief descriptions of each, I’ll focus on the most widely used volume types—general purpose.

GP Storage – the Common Choice

This is the staple storage for pretty much all AWS users. It gives you the best bang for your buck and is commonly used for tasks that don’t require heavy lifting from a drive perspective. GP storage ranges from 1 gibibyte (GiB) to 16 tebibytes (TiB), providing from 128 to 1,000 mebibytes per second (MiB/s) of throughput, which will most probably fit your use case, unless the workload is significantly specific.

There are multiple types of GP storage, each with different pricing and characteristics that I’ll provide a few details on below.

GP2 – Old-Gen SSDs for the Masses

GP2 is the typical solution for boot volumes of EC2 instances, development environments, and lighter/less important databases. This is one of the most popular options out there, although its popularity comes from old habits rather than actually being a perfect shot. GP2 volume performance scales with its size—the bigger they are, the faster they go. Still, these might seem a good option, but to be honest, they aren’t so perfect. And the biggest elephant in the room is the term you’ll have to remember – “burst credits.”

Burst Credits – Math Homework for Your Business Work

Provisioning a GP2 volume, you get a one-time grant of 5.4 million burst credits, which you can use to make it handle input/output better during spikes in demand. More of those tokens means you’ll be able to sustain bigger performance demand for longer, boosting your storage up to 3,000 IOPS as long as you have enough of them in your pocket.
While depleted when you go “turbo,” they accumulate when you’re not using your disks at full speed—saving up unused performance for when you need it most. Burn them all, and your IOPS performance drops to base value and throughput falls to the baseline IOPS multiplied by max I/O size.

The performance of GP2 storage volume can be calculated with this formula:
Throughput in MiB/s = ((Volume size in GiB) × (3 IOPS per GiB) × (I/O size in KiB))

Until you reach the point where your volume is so big that its base performance exceeds burstable, you’d have to learn them by heart. Where is this point? Quite high, at about 1.1 TB (1024 GiB)—a lot of disk, even for a medium-sized business, especially since those volumes can’t be mounted to multiple targets.

Figure 1: Graph of Max IOPS available for certain volume sizes (Source: AWS documentation)

Unless you’re deploying a disk-devouring monster such as an EC2 host instance for dozens or hundreds of Docker containers, or you plan to run an enormous Elasticsearch node for the next few years with no clean-up routines in place, you’re not likely to need this much space. Also, in either of the two cases mentioned, you’d probably run out of CPU and RAM first; there are better alternatives for these cases as well, not to mention that the lack of maintenance is definitely not a good practice.

In addition, for practically every use case—web serving, web applications, databases, casual EC2 instances for use with Docker, and more—pretty much anything is better than GP2.

Luckily, in 2014, the introduction of GP3 brought some great improvements.

GP3 – The “Newcomer”

While not really a “new” player, GP3 volumes are a welcome improvement after experiencing the inner workings of GP2. They perform better and cost less in comparison, allowing you to easily scale your operations according to your needs much easier. Gone are the days of burst credit balances.

The golden rule of this storage class is “mix n’ match.” Baseline performance here is the maximum burst performance of a GP2 volume, 3,000 IOPS, and 125 MiB/s. If your business needs more, you can purchase additional amounts of either, paying extra only for what you’re interested in.

The limits are quite generous. You can purchase up to 500 IOPS per GiB (for at least 32 GiB volumes, capping at 16,000 IOPS), or up to 0.25 MiB/s per GiB (for volumes of at least 8 GiB, 4,000 IOPS or more, capping at 1,000 MiB/s).

If you were previously happy with your GP2 volume, it’s almost guaranteed your company will be just as happy with GP3, and you’ll save a decent percentage of the storage cost, even up to ~20%.

Assuming that you don’t need more than what GP2 provided you with, make the switch. To make it even easier to cut those costs down, AWS is now automatically setting the IOPS and throughput of your new GP3 volume to the exact same as your GP2. Less guessing, less messing around, more good stuff. Just click a few buttons and save some cash, no strings attached.

This is the storage class I would recommend for every “generic” or “typical” workload. It’s a good starting point for almost every business use case that doesn’t expect huge amounts of disk operations, but is ready to scale up when the dreaded flood of demand for disk power comes.

GP3 can easily handle the role of boot volume for EC2 instances. It can host both simple and a bit more complex applications, databases such as MySQL and PostgreSQL, or a blend of those in a containerized environment, especially in a multi-node configuration or as part of a Kubernetes cluster. Zesty Disk for Kubernetes handles this switch automatically if you have the CSI driver installed

When the time comes, it can scale in tandem with demand. In certain cases, it might even be cheaper to provision high-speed storage with GP3 than the IO1/IO2 types, which are dedicated to speed and next on our list.

IO Storage Class – Need for Speed

The IO storage class was introduced to cater to the needs of businesses based on extreme durability, high read/write speeds, and the lowest latency available. It requires certain criteria to be met to show its true potential and costs significantly more. But, in turn, it provides unmatched speeds and reliability, as well as some features you won’t find in any other storage type.

This volume type is great if you seek a volume to provision for your high-speed disk-based database, for example, or to create a single MySQL database host for multiple large-scale e-commerce instances, although there are multiple options available for this from RDS to multi-instance DBs (e.g., MySQL Galera). It’s also a fit if you need to accept a lot of data fast to process with an ML algorithm.

IO1/IO2 Vs. IO2 Block Express – Which Type of “Fastest” do you Need?

IO2 Block Express is pretty much IO2 on extremely potent steroids. You get limits that are four times as high for IOPS and throughput, latency of less than milliseconds, and surprisingly, no additional fee. Yes, you read that right, they’re priced the same. Plus, provisioning additional IOPS/throughput above the limit of a standard IO2 instance actually costs less.

If you decide to use IO2 and meet the requirements (a certain type of EC2 instance), you should definitely consider Block Express—if it’s not already running by default.

And when would you run IO2 anyway? Well, with its power and hefty price tag, this class is dedicated to the most disk-dependent, throughput-hungry, and IOPS-dependent monstrosities. For example, a data-crunching machine for a stock exchange, a global log-ingestion mainframe, or a multi-pod Kubernetes node prepared for very demanding workloads—the titans of storage, no doubt. It’s a go-to when you need a single machine that’s going to pack as much punch as you can get.

How about IO1 then? It’s a compromise between low latency, high IOPS, and price. A good choice when you need a bit more than general purpose, but still not enough to go for the fastest storage out there. Keep in mind there’s also one additional drawback—less durability. A 0.2% failure rate annually does not sound like a lot, but if you’re thinking about dozens or hundreds of drives, absolutely critical workloads, or both, it definitely makes a huge difference.

So, Which Should I Choose for my Workloads?

As you can see, the world of EBS is quite complex, to say the least. So, I made a chart that should ease the selection process for your use case:

Fantastic Disks and How to Tame Them

Choosing the right storage class is a tough decision to make—but the benefits are worth it. With just one shot, you can both improve the performance of your instances and save on costs as soon as your next invoice.

December 5, 2022
Why SaaS Vendors Need to Give Back to Open Source

There aren’t many tech organizations that don’t leverage open source technology. Companies that refuse to import software will rely on a freely available library tool or codebase. Open source is one of the largest and most powerful mechanisms of software propagation. However, open source is in danger.

The Two Sides of Open Source Development

In recent years, the ubiquity and dominance of open source tooling have led companies to adopt them en masse. Products like Elasticsearch, Spring and Akka are incredibly popular as open source, freely available tools. But unfortunately, often only one-half of the open source philosophy is being honored.

The Open Source Philosophy

What does that mean? Well, open source depends on two things. Customers use the software and those same consumers contribute to the codebase. It’s a feedback loop of like-minded individuals who have a shared interest in building and using an outstanding product and are happy to donate their time to realize their collective goals. This agreement has driven some of the most exciting software achievements of the past two decades.

Alas, while it became prevalent to use open source software, the popularity of contributions did not scale equally. Suddenly, small teams of dedicated developers were inundated with demands from huge enterprises that equated “open source” with “free.” This caused the owners of open source projects, like Elasticsearch, MongoDB and Akka, to reevaluate their position and change to the increasingly prevalent source available license.

Growing Risk to the Open Source Community

If users contribute to the codebases they leverage, then those codebases can thrive and no one needs to worry about subscription fees or fines. Moreover, users can maintain the certainty that they are not doing anything illegal when they depend on a piece of software. Without contributions from the engineering community, an open source project’s commercial and practical basis quickly disintegrates.

So Why Should SaaS Vendors Give Back?

SaaS solutions inhabit a particular corner of the market, assisting companies in solving a cross-cutting problem. For example, observability. This is an important detail because many open source tools also operate in this arena. This is exemplified by the fact that it’s common for SaaS vendors to regularly compete against an open source, homegrown alternative. Many SaaS solutions actually base their core infrastructure on the same open source software.

This means that SaaS providers are often in direct competition with the open source solutions they leverage to build their product(s).

SaaS Vendors Depend on Open Source Software

The communities that support vital pieces of open source software, like Apache Kafka or OpenDistro, are helping to unlock innovation in enterprises worldwide. The continued pace of this innovation is inextricably linked to the continued growth and development of the open source tooling on which it depends.

The only sensible course of action is for SaaS vendors to make every effort possible to contribute back to the open source space. They do not need to give up their unique features, nor do they need to release their proprietary software, but the simple act of maintaining patches, improving performance, increasing compatibility and even correcting poor documentation—all of this is a contribution that will benefit the entire SaaS ecosystem.

Those Contributions Will Breed Specialization

A big challenge of using open source software is finding engineers that understand how it works. The plug-and-play nature of many tools means that users never need to dive into the details of a library or application. This, however, comes at a cost. Users often don’t know why the software breaks or behaves unpredictably.

If a SaaS vendor begins to contribute back into the core of the open source tools on which they depend, not only are they benefiting their product and the open source community, but they’re also seeding specialist knowledge in their teams. This specialist knowledge means they can utilize open source features in new and exciting ways, giving them insights and advantages over their competitors.

Open Source Contributions Make Commercial Sense

The SaaS community thrives when open source tooling works smoothly. Conversely, if fundamental open source tools like Kafka or OpenDistro begin to falter, then the success of the SaaS community is also at risk. They are highly interdependent. This means a contribution back to core open source tools is a contribution to the stability of SaaS products. Moreover, growing specialist skills within engineering teams means that when it comes time to leverage a new open-source feature, your developers may have skills and insights that other companies won’t have. This is a clear commercial advantage, and all for the cost of doing the right thing, and keeping the dream of free, open source software alive.

December 2, 2022
Implementing Shift Left Security in the Cloud

While ransomware has been the leading concern for enterprise security teams over the few past years, software vulnerabilities are nipping at its heels. The boom in cloud-based apps and services and increased digitization of work have been a boon for hackers, who are taking advantage of developers’ and DevOps teams’ attempts to work faster and smarter to keep up with demand. One estimate says that four out of 10 zero-day attacks carried out in the last decade happened in 2021 alone.

Many things account for this increase. Developers are stretched and are reusing code, which allows for misconfigurations and vulnerabilities to reappear unexpectedly in different programs, and the use of multiple cloud services fragments security measures and reduces visibility into the code running many enterprise functions. This is why developers and security professionals alike are paying more attention to security throughout the software development life cycle (SDLC), particularly in the early stages.

Shift Left Security Principles and Challenges

The zero-day surge has led to an increased interest in shift left practices as a way to make security a priority in the development process. Shift left culture brings security into the equation much earlier in the software life cycle, before the software is deployed, rather than patch bugs after users report them. This preemptive approach helps head off vulnerabilities that can affect an application’s security posture unbeknown to its defenders.

Shift left principles can also enhance security when developers build applications for cloud platforms—such as Amazon Web Services, Microsoft’s Azure or Google Cloud—where visibility into the proprietary code and security tools of the platform can be limited. In a shift left culture, DevOps embeds least privilege policies as part of the daily work on cloud workloads, to protect network infrastructure and avoid granting excess permissions on those workflows.

For example, setting up role-based access control (RBAC) on Kubernetes containers enforces a least privilege model on those clusters and avoids excessive permissions that can lead to a breach, while removing admin credentials from continuous integration/continuous delivery (CI/CD) workstations shuts off hackers from using those pipelines as an attack vector.

Shift left security is a great concept, but runs into hurdles in execution, mainly the internal conflict between development teams who want to move at the speed of business while security staff takes a more cautious approach. This leads to friction, with developers warning about access denials affecting their work and causing deployment delays, while the security analysts worry about data breaches caused by too many users with unchecked admin privileges.

Working out this friction requires a mindset shift, where DevOps understands that security is part of the development process, and management can trust that developers have security in hand. This also requires tools and training to integrate shift left practices, such as platforms that can bridge the security needs of the CISO (by granting greater visibility into the network infrastructure code, for example) and the agility needs of the DevOps teams (by automating security guardrails and entitlements, for example). The ultimate goal is creating what is known as a DevSecOps model, bringing together development, security and operations into one agile, safe and efficient workflow.

How to Adopt Shift Left Security on the Cloud

Four best practices can help:

● Gain deep visibility: With enterprises often working in multiple cloud environments—and mixing public and private clouds into hybrid infrastructures—visibility into assets typically suffers, which makes getting a clear view of risk difficult. Being able to discover all identities, permissions, configurations and resources in the environment and all the access paths to specific resources can help build a contextual inventory of cloud assets to better manage them and carry out policy analysis.

● Prioritize risk: Risk management is a key practice of any security program, but the flood of alerts from multiple cloud providers and systems can overwhelm security operations centers, which are often short on talent and time. Automation tools can now search and prioritize risks across an entire cloud environment, from development to production, spotting problem combinations and alleviating the workload of sorting manually through alerts across multiple silos, freeing the staff to work on higher-priority mitigation and remediation tasks.

● Just-in-time access: Zero-trust architecture has become the standard to aspire for most security defenses, but it is often challenged by the difficulties of managing access privileges. Just-in-Time (JIT) access, granting limited-time revocable rights, is a useful tool for building a zero-trust security architecture and enforcing least privilege policies to maintain it. Technology tools that actively manage and monitor developer access to cloud environments, including a strong audit trail of privileged activity, give security a means to enforce JIT access and zero-trust in turn.

● Find and prevent vulnerabilities and misconfigurations: Infrastructure-as-code (IaC) has made software deployment in the cloud (much more agile and efficient with the use of virtual machines, containers, microservices, etc. However, security is often an afterthought both pre- and post-production when standing up cloud infrastructure. Organizations should scan code for security policy violations and other risks in order to mitigate flaws before they reach production and to detect and fix problems in production environments. With feedback and automated guardrails built into development workflows, DevOps and security teams can reduce security risks in their code.

Like zero-trust, shift left is on its way to becoming a standard practice for cybersecurity assurance. But building a DevSecOps model takes more than good intentions. It requires practices, tools and training to execute that model, and a mindset of partnership across the enterprise and with outside cloud and security partners. Only a group effort will provide the cloud security assurance an agile business needs.

November 30, 2022
Blue-Green Deployment: What Are the Options?

Blue-green deployment is a change management strategy for software releases. Blue-green deployments require two identically configured hardware environments. One environment is active and serves end users while the other remains idle.

Blue-green deployments are typically used for applications with strict uptime requirements. First, new code is released to staging environments and fully tested. Once the code is reviewed, the team adjusts the router configuration to redirect application traffic to the staging environment. This staging environment then becomes the production environment.

If problems are found after the switch, the team switches traffic back to the old production environment and continues running the original version. Once the new code is validated in production, the team can update the code in the idle production environment, and use it to test the next release. For more background, see this detailed blog about the blue-green deployment pattern.

Blue-Green Deployment Benefits

Here are some of the main advantages of blue-green deployments.

Standby System
A blue-green deployment always ensures a server is on standby, providing a crucial backup when the system fails. In addition to facilitating software updates, this approach improves risk management, allowing you to switch to the backup server during a malware breach.

Redirecting traffic to the other system version is easy because it has the same infrastructure. The system can stay live and functional while the teams fix issues.

Smooth, Fast Releases
A blue-green deployment allows product owners to push software to production quickly. It offers the flexibility to release an update at any time, with no downtime required. End-users don’t feel the impact of each release. DevOps teams can also benefit from smooth releases because they don’t have to race to meet deadlines for scheduled outages.

Simple Rollback Process
Rolling back an update is also fast in a blue-green deployment. Simple rollbacks reduce the risk of experimenting in a production environment. You can implement rollbacks by setting the application to read-only or using a load balancer to handle active transactions during the cutover. This can be significant for Kubernetes security because if security vulnerabilities are discovered in the cluster, you can easily roll back to a previous safe version.

No Downtime
A blue-green deployment allows you to avoid downtime, whether scheduled or unexpected. You can shift users to the other model during maintenance checks and fixes. Upgrades can occur anytime because there is no need to wait for times with lower traffic (i.e., midnight).

Testing in Production
Testing in production allows you to check a live product’s functionality, showing how it performs in the real world. The drawback of this approach is that it can expose end-users to bugs during testing. A blue-green deployment lets you test the product in an inactive production environment, minimizing the risk of bugs making it to the live environment.

Blue-Green Deployment in Kubernetes

Many development teams are migrating to Kubernetes as an environment for development, testing and production deployments. Kubernetes makes it easy to deploy applications and can also be used to automate blue-green deployments.

However, Kubernetes does not provide out-of-the-box blue-green deployments. It has a Deployment object that allows “rolling updates”—this allows you to update your applications without downtime by incrementally replacing pods with the latest version of an application.

This is similar to a blue-green deployment but doesn’t offer all the benefits. Rolling deployments can make it difficult to roll back if there are issues with the new version. You might require Kubernetes health checks to determine the status of pods within a deployment. Applications might take time to roll out, while the pure blue-green pattern allows you to switch over to a new version instantly.

Blue-Green Deployment in Kubernetes: What are the Options?

Argo Rollouts
Argo Rollouts is a set of Kubernetes controllers and CRDs that provide advanced deployment features for Kubernetes such as blue-green and canary deployments, experiments, and other progressive delivery capabilities.

Argo Rollouts optionally integrates with Kubernetes ingress controller implementations and service meshes, leveraging traffic shaping to incrementally switch traffic to new versions during updates. Rollouts can also query and interpret metrics from other systems to validate key KPIs, and automatically promote or roll back releases.

Flagger
Flagger is a progressive delivery tool that automates the release process of applications running on Kubernetes. It reduces the risk of introducing a new software version into production by gradually migrating traffic to the new version while measuring metrics and performing conformance testing.

Flagger implements multiple deployment strategies (canary releases, A/B testing, and blue-green deployments) and integrates with various Kubernetes ingress controllers, service meshes, and monitoring solutions.

GitLab CI/CD
GitLab is a collaborative DevOps platform and open-source code repository for large software development projects.

In addition to a code repository, GitLab also provides CI/CD automation and issue tracking. GitLab repositories allow the hosting of different software versions and entire development pipelines, allowing users to roll back software and environments in case of unexpected problems.

GitLab supports both manually triggered deployments and scheduled deployments to Kubernetes production systems, using progressive deployment (including blue-green deployment).

CircleCI
CircleCI is a CI/CD platform that helps engineering teams build, test, and deploy software, while viewing code changes in real-time in a dashboard UI.

CircleCI can be deployed to virtually any target and can be easily configured to integrate with other services, for deployment strategies such as feature flags, blue-green and canary deployments. It supports both fully automated deployment processes or a process with manual approvals, with the ability to easily customize deployment configuration.

Conclusion

In this article, I explained the blue-green deployment pattern, its benefits, and how it can be beneficial for Kubernetes deployments. I introduced four tools that can help you implement blue-green deployments in your Kubernetes clusters:

● Argo Rollouts – Part of the open source Argo project. Includes Kubernetes controllers and CRDs that provide advanced progressive deployment features.
● Flagger – An open source progressive deployment tool for K8s that supports canary releases, A/B testing, blue-green, and more.
● GitLab CI/CD – Supports both manually triggered deployments and scheduled deployments to Kubernetes production systems.
● CircleCI – Supports deployment to Kubernetes clusters, with deployment strategies such as feature flags, blue-green and canary deployments.

I hope this will be useful as you learn to implement progressive deployment patterns in your Kubernetes environment.

November 29, 2022
Switching From FluentD to Vector Log Aggregation Tool

Log files are extremely important to the data analysis process as they contain essential information about usage patterns, activities and operations within an operating system, application, server or device. This data is relevant to a number of use cases across an organization from resource management, application troubleshooting, regulation compliance and SIEM and business analytics and marketing insights. To manage logs created by these use cases and make use of this wealth of data, log aggregation tools enable organizations to systematically collect and standardize log files. However, choosing the right tool can be quite challenging.

This blog will detail and compare the popular open source fluentD and Vector tools for log aggregation.

fluentD Configuration and Efficiency Calculation

When using orchestration tools like Kubernetes to deploy containers or other API resources, there is a need for a log aggregator to store the pod or node logs in a cloud platform. For a particular requirement, fluentD was used as a log aggregator tool to push K8s pod logs to cloud storage buckets with a sample configuration as shown below:

<match kubernetes.**>

@type <cloud platform name>

project <project name in cloud platform>

keyfile <credential json to access the cloud storage> bucket <cloud storage bucket name> object_key_format <name for the file to be used>

path <file prefix/path where the file have to be stored>

<buffer tag,time>

@type file

path /var/log/fluent/gcs timekey 1m timekey_wait 30 timekey_use_utc true flush_thread_count 16 flush_at_shutdown true flush_mode interval flush_interval 1 chunk_limit_size 10MB retry_max_interval 30 retry_wait 60

</buffer> <format>

@type json </format>

</match>

Using this system, fluentD was pushing only 47.62% of total logs to cloud storage. Since there was a loss of more than 50%, changes were made to the configuration. In most of the changes, the efficiency was somewhere between 40% and 50%, with a maximum efficiency achieved at an average 67% for an entire day. Below are some of the changes made along with the percentage of logs that were pushed to cloud storage:

<buffer tag,time> @type file

path /var/log/fluent/gcs timekey 1m timekey_wait 30 timekey_use_utc true flush_thread_count 16 flush_at_shutdown true retry_max_interval 60 retry_wait 30

</buffer> Efficiency:- 46.32%

<buffer tag,time> @type file

path /var/log/fluent/gcs timekey 1m timekey_wait 30 timekey_use_utc true flush_thread_count 16 flush_at_shutdown true

</buffer> Efficiency:- 49.89%

<buffer tag,time>

@type file

path /var/log/fluent/gcs timekey 10m timekey_wait 0 timekey_use_utc true flush_at_shutdown true

</buffer> Efficiency:- 37%

<buffer tag,time> @type file

path /var/log/fluent/gcs timekey 30 timekey_wait 0 timekey_use_utc true flush_thread_count 15 flush_at_shutdown true

</buffer> Efficiency:- 60.88%

<buffer tag,time> @type file

path /var/log/fluent/gcs timekey 1 timekey_wait 0 timekey_use_utc true flush_thread_count 16 flush_at_shutdown true flush_mode immediate

</buffer> Efficiency:- 66.77%

Vector Deployment, Configuration and Resultant Efficiency

To improve this further, the open source Vector tool by Datadog was also considered. This tool was suitable for K8s setup with a similar configuration as fluentD and was installed in the nodes.

A Helm command was used to clone the official repository in its VMs; the configuration was changed as described below and installed it as an agent. Vector comes in two working modes: Agent and aggregator. While agent is the plain mode that pushes logs/events from source to destination, aggregator is used to transform and ship data collected by other agents (in this case, Vector).

The installation of this tool requires a Helm repository in the local machine to fetch the source code. Hence the below commands were run in a sequential pattern before installing Vector in a K8s cluster:

helm repo add vector https://helm.vector.dev (Adding vector repo to helm list)

helm repo update (Updating the helm repos)

helm fetch –untar vector/vector . (command to clone the repository to local machine)

Configuration:-

data_dir: /vector-data-dir

sources:

<Custom source id>:

type: kubernetes_logs (because we are using kubernetes as our source)

exclude_paths_glob_patterns: <Array of directories which has be excluded when collecting the logs from the nodes> (Optional)

sinks:

<Custom sink id>:

type: <Destination cloud storage>

inputs: <Array of source id’s from where log has to be pushed>

bucket: <Bucket name of cloud storage>

key_prefix: <Path inside the bucket where the logs has to collected> (Optional) encoding:

codec: <Encoding of the log file> (Optional) Command to install the Vector:-

helm install vector . –namespace vector

After deploying Vector in the development environment and testing it, the efficiency was ~100% with negligible loss. The switch was then made to Vector and deployed in the production environment. Vector can ship up 100,000 events or logs/sec, which is a very high throughput rate compared to other tools for log aggregation performance. Vector was able to achieve 99.98-100% efficiency even in the Kubernetes production cluster.

To learn more about how DataOps can enable highly performant data pipelines with real-time logging and monitoring, watch this video.

November 23, 2022
Avoiding Developer Burnout With Clean Code Best Practices

Employee burnout is common in the tech industry, especially for developers. In fact, 83% of software developers feel burnt out from their work and of those, nearly half of them feel overworked. Often, this is because the processes currently in place at their companies are not effective and too much time is spent doing mundane work.

Resolving coding errors is a necessary part of software development, but this isn’t always an easy chore. Given the complexity of code today, developers do tend to spend a large percentage of their bandwidth on maintenance issues, refactoring, debugging or just modifying code that they never wrote in the first place. That said, 44% of developers stated that fixing software defects and problems is their biggest frustration.

Every developer wants to create good code, but repetitively dealing with issues that aren’t part of the code they write steals away time from doing interesting work. More than a third of developers claim that they would prefer spending more time with their families, participating in more physical activity and even getting a full night’s sleep than spending the majority of their time fixing code. Developers also stated that they could use the time at work to focus on new features and capabilities.

By implementing clean code best practices across the organization, development teams can create an environment that prioritizes new and interesting projects while organically addressing errors. Clean code practices help keep developers engaged and interested.

What is Clean Code?

Every organization relies heavily on software and the source code that powers it to run successfully. Companies are aware that the source, or DNA, of their software is what counts as its most important asset. The behavior and functionality are both governed by the source code and if it’s not kept in good condition, the source code can turn from an asset into a liability.

Clean code refers to a codebase that has reached a flawless and problem-free state. It is code that is maintainable, reliable, secure, readable and embodies other such attributes of code. While developers write clean code for their new projects from the beginning, they simultaneously address existing issues through set quality standards. The overall workload of security and development teams also is greatly reduced when clean coding practices are considered and integrated into the development workflow.

With this method, fewer issues reach the final stage of inspections. It also avoids pricey post-production rework or drawn-out feedback processes. The outcome is a simplified and effective method for managing the state of your code.

How Clean Code Helps Developers

By giving your developers access to the power of clean code practices, organizations can make sure that their development teams are working on projects that are impactful and exciting rather than dealing with mundane repetitive work. This means that they have more time to research, put new ideas into action, and also be part of new initiatives that have a direct impact on their organization’s success.

When using the clean-as-you-code approach, you create a precedent that, moving forward, new code that is added or updated won’t result in new problems. As new updates are made to old code, the overall quality of that code progressively improves. This is a much more manageable process and a huge step toward improving the overall quality of code.

Additionally, the clean-as-you-code approach empowers developers to take charge of the quality of the code they deliver. The feedback loop to detect and fix issues is naturally shortened with the proper tooling they can use in their current DevOps workflow. And contextualized education helps them quickly understand the root cause and get guidance to fix them. When their everyday routines automatically incorporate this technique, the possibility of recurring or exposed weaknesses is greatly reduced. In the long run, this helps save development time and effort, making development so much more enjoyable.

There are many things that are a necesssary part of a developer’s everyday responsibilities. Making it simple and efficient is a win-win for everyone. No matter the state of your codebase, the maturity of your development team, or the size of your organization, clean code best practices are easy to embrace and will make your developers happier and more satisfied at work.

November 18, 2022
Data Centers IN SPAAACE | Discord GDPR Fine | AWS Fires Dead Wood

Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

This week: The EU wants to put servers in orbit, a GDPR penalty for Discord, and Amazon has the hatchet out. (more…)

November 17, 2022
5 Tips for Securing DevOps: What You Wish You Knew Sooner

Foundations and frameworks, concrete and steel—not exciting. But that’s the foundation and framing of pretty much every modern building. Everything else that is part of a building–flooring, wiring, lighting, room placement and so on—is made possible by the foundation and the framework. If you want DevOps to succeed in your organization, you have to start by building a solid foundation and a framework, including security. What’s below is a suggested approach to the concrete and steel of DevOps.

Three Foundational Needs

1. Leadership and Governance
Someone has to make the decisions, be responsible for leading the team(s), give direction and provide operational governance. Securing DevOps is not just about compliance with a set of security concepts; it’s also about setting the right tone for the workplace. Leadership and governance are key enablers of development success.

The pressure to deploy software faster and more often is real—but so are the security, regulatory and many other concerns around automation that may uncover bugs or defects in software before it gets pushed to production. In an environment where there’s pressure to release code faster, how does one balance these competing priorities? What’s more important: Speed or quality? This tension between speed and quality remains one of the main challenges faced by organizations adopting DevOps practices today.

Organizational leadership and governance regarding people and data must be in place to keep people working on the right things, keep up morale, provide direction about project prioritization and budget the right tools for getting the job done.

An important subcategory of governance is data classification. Where does the data reside? Over which channels does it flow? What kind of data is it? Classifying the data has long-term implications for resource procurement such as what storage to use and if it needs to be encrypted, how stringent access control for the environments needs to be and interoperability of the various technologies.

2. Regulatory Compliance
It wasn’t too long ago that if a business remained only national, then there weren’t necessarily many regulatory requirements. Presently, if a business has a website (and who doesn’t?), then international and cross-border data transfer is a given unless one takes the course of geo-restriction, which in itself can be a major undertaking with repercussions.

Determine regulatory needs. From where do people or other services access the data? What cross-border transfer is involved? To paraphrase the late 1960s-1980s public service announcement, “It’s 10:00 p.m. – do you know where your data is?” Design and implement accordingly.

3. Risk Management
One cornerstone of technology management is that it’s all about risk. If there’s a serious incident such as data intrusion or theft, what will the cost be to the organization and was the company part of the problem (e.g., negligence)?

What are the technical vulnerabilities, threats and risks presented by the product or service? What kinds of testing (e.g., regression, penetration, vulnerability, CTI) would it take to assess and address the risks? What does the patch cadence look like?

On the personnel side, what are the risks of key players leaving? What’s the morale of the team regarding aspects such as culture and pay? What’s being done to address those risks? Included are third-party, contractual, and privacy risks (remember data classification above?).

NPM packages are a growing trend as an avenue of exploitation; uptime is a common requirement for meeting obligations, and privacy is considered by many to be more than a privilege but a human right, and not something to take lightly.

Two Framing Activities

The foundation activities undergird the entire structure of DevOps, but also consider what shapes the structure.

1. Software Development Life Cycle (SDLC)
The SDLC is vital. Without it, where do all personnel (team, contractors, consultants) know how to proceed with designing and implementing the product? What model is used, and who’s responsible for what? It’s all here in the SDLC. Of course, things like references for API architecture specifics and corollary application documentation would make one feel like wandering in the Library of Congress (and make developers want to burn it down like the ancient Egyptian Library of Alexandria!).

Speaking of APIs: Due to the enormous rise of API use in all industries, including health care and retail, any use of APIs must be part of the SDLC.

Not only are APIs on the rise for public use and interaction, but microservices use within an org has increased drastically. APIs are necessary internally and externally for businesses to reach, maintain and accelerate the speed of innovation, so including them as critical resources is crucial. The average API usage per company increased by 221%, so attending to that ecosystem by ensuring proper activities, especially security, are included is needed for developers.

The SDLC is there to make workflow and production consistent (and concomitantly easier) by codifying what needs to be done for both company and customer, not to make work more difficult by presenting nauseating tedium. Like any corporate policy, it needs to remain useful, while not watered down.

Here are a few specific aspects to include:

● Threat modeling in some form needs to be included. It doesn’t have to be a magnanimous effort, but the realities of threats will play a major part in development.
● Testing. Regression, stress, pentest, security – these and others will help guide what kind of testing environment one will need and lead to any need to increase items such as capacity and scalability.
● Secure coding. If it’s being done, take steps to improve. If it’s not being done, just get started. It’s always hard to go back and fix insecure code, but it’s a necessity due to the many risks and regulations involved in providing software to the public.

2. Training
Training is required because businesses have to adapt as technological changes create new opportunities that lead to continuously changing customer demands.

Training is everywhere! There’s no one right place or right approach – just get training and change as needed (specifics should be handed down by leadership).

On a strict and lowball budget? One option is OWASP membership (which costs $50 per person per year) and includes several types of AppSec and DevSecOps training. That small investment can help raise the level of clean and secure coding, threat and security consciousness and professional reputation.

Setting the Course and Always in Motion

An issue with setting any course in business is that it’s semi-permanent. A DevOps model put in place becomes part of the foundation, which can make one reticent to commit. The only way to prove the chosen methods is to set them in stone and do them consistently for a long time. Aspects of the model may change over time, but once that foundation is put in place, it takes an enormous amount of work, time and money to change it. The change wouldn’t be a remodel, but a reconstruction, so the model takes proper consideration at the outset to chart the right course.

While DevOps needs a solid foundation, it’s an always-in-motion model. There are new technologies, new people, business changes, customer demands and societal and cultural shifts. These concepts can be included in calculations, but what those changes will be is incalculable. Hire and train the right people and the necessary steps and adjustments will be possible.

November 16, 2022