devopsy.kinsta.cloud

Category: DevOps Practice

The Advent of Intelligent Software Engineering
Today, to maintain competitiveness, every company needs to behave like a software company and deliver new solutions with high velocity, quality and reliability. New features need to be introduced at a rapid pace, but with high-quality expectations. At the same time, deployment releases and operations are getting complex, debugging and traceability in production are increasingly difficult, security is more prominent, uptime and reliability needs have scaled up dramatically … the list goes on.

In short, traditional software engineering is changing radically. Some trends causing this disruption include:
- Cloud is eating software. And that has not only changed the infrastructure where software is hosted but also how applications are architected, developed, packaged and deployed. Cloud is driving the adoption of cloud-native technologies; serverless components and containerization have changed how we build applications. The hard line between development and infrastructure team is getting fuzzy.
- Cybersecurity is now a boardroom concern. Rising cyberattacks and malware threats have left everyone vulnerable. Traditionally, software was released every few months, leaving enough time for security testing by specialists. Today, with the advent of very rapid release cycles, security checks need to start early in the development process.
- AI/ML is the new electricity but is causing shocks. AI/ML-powered use cases are finding their way across platforms, and across industries. Traditional software engineers are not familiar with AI/ML and ML developers are not familiar with engineering disciplines and best practices, leaving a big divide in engineering maturity.
- When one company raises the bar, others are expected to follow. Consumer expectations are constantly changing. They expect the same experience that they get from Amazon, Uber and Google from every application. It is expected that you provide a seamless, frictionless experience that your platform is available anytime, on any device and that it is fast and secure.
- APIs are fueling innovation: Innovation at speed and scale is not easy, but APIs can enable exactly that by tapping into the collective power of the crowd. Today, any developer can use platform-provided APIs to build a new, innovative solution. An API-first mindset is emerging, and the app store model (e.g., Apple App Store, Salesforce AppExchange, AWS Marketplace) has changed how we think of innovation and creating new platform extensions.
- Low-code/no-code platforms are quickly evolving, making it easy to develop small, situational applications without writing any code. Citizen developers are emerging.
- Modern architecture paradigms drive new ways of working. Modern platform architecture has evolved significantly in recent years from monolithic applications to SOA to microservices- and API-based distributed platforms. Event-driven architectures have gained prominence. Streaming architecture has become increasingly popular to support massive scalability. This has a significant impact on how engineering teams are organized and how they build, test, deploy and manage new platforms.
- Legacy platforms are slowing down growth. Legacy applications built 20+ years ago needs to be modernized for today’s context. Modernization is a massive exercise depending on the age, technology and the size of the application. But modernization is easier said than done. The legacy and the new worlds need to coexist for a while and need a bridge strategy from both a technology and cultural perspective.
To meet these changing demands, engineering needs to become a lot more intelligent.

The Advent of Intelligent Engineering

We can bring intelligence to engineering by using more data to manage projects, leverage AI/ML analytics to bring improve engineering productivity and predictability, leverage more automation and apply the right process to the right problem.

Data-driven Decision-Making in Engineering

Engineering teams use numerous tools in the development environment for code repository, bug tracking, code scanning, build, test, deploy, etc. Popular tools include Azure DevOps, GitHub, Atlassian and Jenkins, among others. These tools generate lot of granular data. All these data silos can be aggregated and used to track interesting metrics around:
- Developer productivity: How much time is spent on writing how much code versus other activity; story points delivered versus committed
- Software quality: Number of bugs generated; rate at which bugs are being fixed; number of bugs that slip into production
Additionally, on top of this data, one can build an intelligence layer using machine learning to draw hidden insights, highlight co-relations and conduct better root cause analysis. These insights can bring greater predictability to engineering outcomes. For example, whether or not developers can meet a release schedule, the level of quality, etc.

Data provides evidence of how your teams are performing. It helps in removing potential roadblocks and enables better communication with all stakeholders. It can be used to better manage project performance and be leveraged for objective decision-making. Despite the availability of this data, engineering leaders are still operating in the dark. Decisions are made based on intuition and gut feel rather than on hard data. It’s time to become more intelligent by using data in project performance tracking and decision-making.

Elevate Productivity and Predictability of Engineering Services With AI

AI/ML is disrupting every industry and is now finding its way into software engineering. AI/ML-enabled developer tools are emerging that can be leveraged across the software engineering life cycle. These tools can not only save time for repetitive activities but also improve software quality in large and complex tasks.

AI/ML-enabled tools can enhance a developer’s daily life from coding to testing. Today, AI-powered tools can scan and analyze code to provide intelligent code completion suggestions, flag any deviation from coding best practices (naming convention compliance, variable misuse, etc.), perform peer review, convert code from one language to another, find security vulnerabilities, etc. Dynamic application security testing (DAST) solutions use AI to discover potential attack vectors in milliseconds which would otherwise take a few days. AI/ML can help auto-generate HTML code from hand-drawn UI sketches. AI/ML also can help developers automatically generate unit tests from existing code and provide suggestions about improving tests.

In fact, testing is the biggest category that can benefit from AI/ML-based intelligence, specifically in large complex applications. QA teams who rely on manual testing are not able to keep pace with the rate at which code changes and releases are done today. AI/ML-enabled tools can help testing teams in a number of specific ways.
- Change impact analysis: AI/ML can help with coverage analysis and identify which tests need to run based on what has changed in the application.
- Test creation: AI/ML can help test teams create test cases from plain English descriptions and can also learn how to improve tests over time; AI/M also can automatically heal broken tests. AI-powered tools can automatically convert manual UI tests to API tests.
- Visual testing: AI-powered image comparison technology can enable visual testing to analyze the UI screen differences detected across tests.
- Test analysis: AI can analyze test cases and defect metrics to increase test coverage while reducing the number of tests.
It’s time to become more intelligent by using AI-powered tools to augment engineering teams, improve their productivity and bring predictability.

Intelligent Automation: Beyond Development Processes

Today, there are a lot of automation options in the software development life cycle. DevOps enables automation of build, test and deployment of new versions of your software. You can also automate other activities such as test automation, code scanning, performance test, etc. DevSecOps enables automation of security processes (e.g., code scanning for vulnerabilities) into the engineering life cycle. AI/ML-powered tools can further drive automation improvements (e.g., improve flow in your CI/CD pipelines.) Overall, automation enables higher team productivity and results in the rapid or on-demand release cycles required for business agility.

Automation is not just limited to development activities, it also applies to other parts of the engineering life cycle, for example:
- Infrastructure-as-code (IaC) tools can automatically provision and configure cloud infrastructure environments (storage, network, etc.).
- Monitoring tools can automate monitoring for availability, production load metrics and security problems and generate alerts.
- Log management: The number of logs generated in today’s ecosystem is vast and it is impractical to collect them manually. Log management tools can automatically aggregate and analyze information from logs.
Leveraging automation aggressively and using it throughout the engineering life cycle can make software engineering much more intelligent. You should be automating repetitive tasks and low-value processes and use the saved time for higher-value activities.

Making Agile Smarter

There are several popular software engineering methodologies available such as Agile, Scaled Agile, the Spotify model and more. Regardless of the methodology or process you use, intelligent application of the right methods and tools to the right problems can make them more effective, efficient and intelligent. For example, traditional Agile processes were originally designed in the manufacturing space and were then adapted to develop mature software. But in today’s fast-changing market, development teams need to rapidly and continuously try new ideas either to demonstrate new business features or for technical validation of emerging technology. This is true not only in the early stages of new product development but also on existing platforms where you need to keep pace with changing market dynamics. Be ready to experiment.

We need to get smarter about selecting the right process and tailoring it to your needs rather than let the process dictate your outcome.

If Software is Eating the World, Complexity is Eating Software

While product engineering has been around for a long time, modern software engineering is a lot more complex than just writing code and following Agile processes. Intelligent engineering practices can get solutions to market faster with better quality and reliability and gain more time for innovation to keep up with today’s speed of business.
August 22, 2022
Unstructured Data Management: Avoiding Insider Knowledge Gaps
Most IT leaders know that managing an unstructured data environment can be quite an undertaking. Before beginning an unstructured data management project, there are a lot of factors to consider including the number of storage devices, the size of a particular dataset, the number of files in the dataset and, perhaps the most intimidating, the unique environment the unstructured data inhabits.

One-size-fits-all solutions can often inadvertently sweep important characteristics of unstructured data environments under the rug. Granted, this short-term, simple solution is easier than putting in the work to customize the best-fit solution for each organization’s data set. However, anyone undertaking an unstructured data management project should be forewarned—this methodology can create a longer-term detrimental effect known as “insider knowledge.”

Insider knowledge is what is left when a group of people spend time setting up data management systems within an organization and then move on from their roles without passing along critical information about the data management system to those who take over. It makes things incredibly difficult for those who inherit the agglomerated system and, chances are, the organizations who currently have this problem don’t yet realize they have it.

The Beginning of the End: Creating Insider Knowledge

Today’s storage administrators have a lot on their plate and are perpetually spread thin. Their daily responsibilities typically include:
- Creating shares and exports
- Designing a file system layout
- Assigning devices for each app or dataset
- Hunting down unused data
- Organizing and pairing workflows to datasets
Often, with their busy day-to-day environment, the IT team’s decisions and actions are being made with a siloed approach. And as the project progresses, a unique language known only to the IT professionals overseeing the project is created. They may create their own tags and place files by a certain organizational system. However, most of this is not written down, meaning it is left to be deciphered like a code without an answer key. This language is the “insider knowledge.”

Insider Knowledge and the Great Resignation

Why is this an issue? As long as someone knows what is going on in the overall picture of the data, it’s fine, right? Well …

Solely depending on the team of IT professionals who created the workflow means that they must stick with the company forever for their systems to make sense. After all, once the people with that knowledge leave, it is largely lost forever to the company.

It is not a bet worth taking that the employees of a company will merely stick around to see the data management project to completion. The threat that insider knowledge brings to organizations is that only the people with an idea of how the system works can properly onboard the next hire. This is a great position to be in for the IT professionals who know everything about the system as there is a sense of job security, but it’s not great for the organization.

For organizations, 44% of employees in the workforce are active job seekers. The Great Resignation has showcased that job seekers have the upper hand in the market today, and that good talent can be bought and poached from companies without a moment’s notice. When—not if—an organization’s IT team members move on, it is imperative to have a well-documented, organized foundation for the unstructured data management system so that the next team is able to pick up where they left off.

Otherwise, when team members leave, they will inevitably leave knowledge gaps and questions will arise such as:
- Are you hosting 20 TB of dead application data on your NAS?
- Is this 20 TB of data something the storage administrator was planning on deleting, or is it important?
- Are you going to pay to keep that data on Tier 1 disks?
- Do you have old employee files that were archived to other storage devices but were never deleted?
The implications of insider knowledge are important to remember before beginning to tackle a data management project. It truly is as simple as future-casting and deciding not to give a few people control over an entire enterprise’s unstructured data stores.

Leave it to the Technology: What to Ask When Finding a Solution

To manage unstructured data, there are now many tools on the market built with insider knowledge (and the pre-existing codes within organizations) in mind. By automating the process as much as possible, organizations can avoid a headache when their IT team moves on. Some questions to ask to aid in finding the proper tool include:
- Location of data: How many storage devices are there? How many data centers are there? And how many countries are the data centers spread across? Do any live in the cloud?
- Scanning capabilities: Do the aforementioned locations of the data affect the tool’s ability to scan it? What functions can be performed, from just scanning to analyzing with big data abilities?
- Presentation: How do you receive the information from the tool? Is it digestible and usable? Will it easily highlight the forgotten data sets?
- Aftermath: When you have the information at your disposal and have the insider knowledge in front of you, how can it help you move forward to promote a healthier unstructured data environment?
- Time: How long will it take? How much data can the engine scan? And will this affect business-as-usual?
These are inquiries IT leaders must find answers to before deciding on the best solution for their organization’s long-term success. Regardless of how understaffed and overworked IT teams may be, in today’s modern data center, the last thing that should be done is give a few IT professionals complete control over an enterprise’s unstructured data management system.

Staff turnover is never fun. In IT, it can severely impact an entire organization if IT teams are forced to start from scratch on projects because no one is left to translate the system’s language that was built on insider’s knowledge.

With the amount of data created each day, the time to clean up and automate the process as much as possible is now, as the time to decide on a solution that works for today was yesterday. IT leaders who take time to evaluate the new technologies and comprehensive data management solutions available will be poised to spend their budgets wisely to reduce the risk of setbacks, delays, cost overruns and even IT team turnover.
August 19, 2022
Agile Sucks (Redux) | Plus: DevOps on Mars

Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

This week: Agile is bad, but “Wagile” is worse.
Plus: This prod is even worse than yours. (more…)

August 18, 2022
Time-Series Database Basics
There’s been a lot of buzz about time-series data for the past few years. No matter what you’re monitoring—financial data, server performance, social media activity or something else entirely—time-series data can give you valuable insights into trends and patterns.

But what exactly is a time-series database, and why do you need one? This article will look closely at time-series databases and how they can help you make sense of your data.

What is a Time-Series Database?

A time-series database (TSDB) is a database management system optimized for storing and querying data that changes over time. Time-series data is often used in monitoring applications where it’s important to quickly retrieve information about a system’s current state and trends and patterns over time.

In most definitions, time-series data, often referred to as time-stamped data, is a sequence of values indexed in time order. Time stamping refers to data collected at various times, where each value is time stamped. These data points are usually gathered from the same source and are used to measure progress over time.

Important Notes:

You can use relational or NoSQL databases to crunch time-series data, but purpose-built time-series databases are tailored to exploit the unique features of time-series data. This implies that time-series databases ingest at a faster rate, query more quickly and compress data more efficiently. Furthermore, time-series databases include special analytical capabilities and management features that are not found in most relational or NoSQL databases.

Some Advantages

This list is not exhaustive, but here are some of the advantages that you might get from using a time-series database:

1.) Time-series databases can handle high-velocity data very well.

2.) Time-series databases are purpose-built for storing time-series data, making them more efficient in storage and querying.

3.) Time-series databases often have built-in analytics and management features designed specifically for time-series data.

4.) Many time-series databases are open source, which means they’re free to use.

Some Disadvantages

There are also some disadvantages that you should be aware of before using a time-series database:

1.) Time-series databases are often more complex to set up and manage than relational or NoSQL databases.

2.) There are relatively few time-series databases to choose from, so you might not have as much flexibility in choosing a platform that meets your specific needs.

3.) Time-series data can be very large, so you’ll need enough storage capacity to accommodate your data.

What Are the Characteristics of Time-Series Data?

There are several characteristics of time-series data that make it unique and require special handling. From a database standpoint, the most essential ones are as follows:
1. Timestamp: Every data point has its timestamp. The timestamp is crucial for calculating or analyzing the information.
2. Structure: Metrics from devices or monitoring are almost always structured, unlike those produced by Internet applications. They have predetermined data types and lengths, and the structure will not alter until the device’s firmware is updated.
3. Stream-like: Data sources, such as audio or video programs generate data at a set or constant rate. These data streams are completely unrelated to one another.
4. Stable flow: Time-series data traffic is constant over time, and it may be calculated and predicted if enough data sources are used within a certain sampling period.
5. Immutability: The source of this data is a time-series data store. Each data point is created only once and never updated or corrected. Time-series data, like log data, is typically append-only.
How Is Time-Series Data Used?

Many of you may be wondering why time-series data is so important. The answer is that it allows us to track changes over time, which can be incredibly valuable for monitoring and troubleshooting purposes.

For example, let’s say you’re a system administrator tasked with keeping an eye on server performance. By monitoring various metrics—such as CPU usage, memory usage and network traffic—you can quickly identify when a server is starting to experience issues.

Time-series data can also be used for predictive purposes. For example, if you notice that CPU usage spikes at certain times of day, you can use that information to plan for future capacity needs.

Of course, time-series data isn’t just limited to server performance. It can be used for any application where it’s important to track changes over time. This includes everything from weather and financial data to social media and website analytics.

Let’s Get Specific

Here’s what you should know about time-series data: It’s typically used to seek insights into operations, create alerts based on real-time analysis and forecast future trends.

The following characteristics are found in time-series data applications:
1. High write-read ratio: Twitter and LinkedIn are internet applications with single articles that millions of people read, but raw time-series data is primarily scanned and evaluated by apps and algorithms.
2. Retention policy: In general, time-series data is not kept permanently. Organizations have a retention policy that specifies when and how their data is destroyed.
3. Real-time analytics and computing: Time-series data must be calculated in real-time to identify out-of-the-ordinary activity and sound alarms based on the acquired information or aggregate findings.
4. Query scope: Time-series data is generally requested over a period or a set of data sources, and filters are used to prevent all historical information from being requested. Furthermore, all or a portion of the data sources with a filter condition are always aggregated.
5. Trends: Single data points are typically not significant in time-series data. The emphasis is on how data evolves, such as fluctuations in the previous hour or day.
Popular time-series solutions like TDengine or InfluxDB, for example, create more efficient processing of time-series data and greater performance than general databases by taking advantage of these qualities.

Do Time-Series Databases Need Specialized Databases?

We live in a modern world where data is constantly being generated at an unprecedented rate. To keep up with the demand, businesses need to be able to store and process large amounts of data quickly and efficiently. This is where specialized time-series databases come in.

Everything is online, including meters, automobiles, lifts, assembly lines and even bicycles. And these items are sending out a never-ending stream of numbers and events that IoT and the cloud have unleashed an explosion in time-series data generation.

With this in mind, it’s not surprising that several specialized time-series databases have been created to deal with this type of data influx.

Time-series data sets are enormous and pose a significant problem for general database management systems, such as relational and NoSQL databases. Non-specialized databases have trouble with the following elements of time-series data:
1. Data ingestion rate: In many time-series data scenarios, millions of data points are generated every second and must be ingested in real-time. Relational databases are not built to handle this quantity of data, and while NoSQL databases may be scaled to do so, the amount of resources required soon becomes prohibitive.
2. Query latency: It is often the case that a time-series system must scan a massive number of data points to get an aggregate result, which might lead to sluggish performance. For example, it would take days for a basic database to compute the average response time of all Amazon.com clicks, by which time the overall conclusion may be incorrect.
3. Storage cost: Data generated by internet-connected devices and apps is nonstop 24/7, with a single day producing as much as a terabyte of data. Because relational and NoSQL databases cannot compress this data effectively, storage costs can quickly skyrocket.
These problems are generally linked to efficiency in processing big data sets, although there are some places where general databases frequently do not meet even the fundamental demands of time-series applications:
1. Data life cycle management: Time-series data is generally removed in bulk, not one data point at a time, as it ages out.
2. Roll-up: In most cases, time-series data is collected and rolled up over a set period before being stored in the new table. Raw data and rolled-up data can have distinct life cycles and retention policies.
3. Special analytic functions: Time-series applications need more specialized features than general databases, such as time-weighted average, moving average, cumulative sum, rate of change, elapsed time for a specific state and the delta between two consecutive data points.
4. Interpolation: The database management system must be able to interpolate data based on the adjacent data points and rules to regularize data sets when applications or algorithms require.
5. Continuous query: Therefore, if you hear a user or customer say they want to know when their data is updated, it’s reasonable to assume they want this functionality.
6. Session and state windows: Aggregations and analytical procedures may be performed on a session or state window, not just time—for example, consider one that only calculates average power consumption when a machine is switched on.
With general databases, developers must write custom code to implement features specific to their data set. Different data workloads require different database solutions; one size does not fit all. For time-series data, no matter the size of your data set, a purpose-built time-series database is the best tool for the job.

Why Are Time-Series Databases Becoming Popular?

Two years ago, time-series databases were the most popular database type in the business, owing to its expanding number of use cases. It is especially useful if you’re executing sophisticated transactions like advertising, e-commerce, supply chain management and so on.

However, because of the rapid expansion of the IoT, they are becoming even more popular. As more devices become internet-connected and send data—time-series, of course—to the cloud, many industries are interested in purpose-built time-series databases.

Time-series data is proving to be an important tool, not just for decision-making and optimization but also for industrial applications. Finally, IT infrastructure has been growing rapidly, with everything from servers, containers, network devices, apps, and microservices being monitored to create massive quantities of time-series data.

Older time-series databases, on the other hand, are often closed systems employing antiquated structures that do not scale to handle the increasing amount of data. A million time-series data points used to seem like a lot, but now millions and even billions of data points are commonplace.

Finally, integrating old (previous) time-series solutions with popular data analysis tools like artificial intelligence and machine learning platforms is difficult, if not impossible. These legacy systems cannot be transferred to the cloud without a lot of work and their licensing terms are no longer adequate for current apps.

Final Thoughts

The expanding market and the constraints of previous time-series databases are making room for a new generation. These new time-series databases are built from the ground up to take advantage of the cloud, big data and AI/ML technologies.

If your business or application deals with time-series data, you need a purpose-built time-series database. The advantages of using such a database are too great to ignore, and the disadvantages of using an older, less capable database are becoming more and more apparent.
August 18, 2022
Busting 5 Common Database Modernization Myths
Modernizing databases is one of the best ways to improve database and application performance. However, many companies are hesitant to make major changes, even if they could lead to better outcomes. Why? Because there are prevailing myths about how database modernization projects fail to deliver on expectations. Five myths, in particular, seem to hold leaders back from pursuing modernization opportunities. We cover each of these myths in detail below:

Myth One: There is little performance benefit to modernizing databases in the cloud

Myth Two: Cloud databases won’t reduce my IT costs

Myth Three: I’m better off leaving my data on-premises for security and availability reasons

Myth Four: There’s not much I can do in cloud databases that I can’t do on-premises

Myth Five: Commercial databases aren’t compatible with cloud databases

Fortunately, we’ve debunked these database modernization myths and can speak to the incredible benefits of modernizing databases on the cloud. We’ll use Amazon Web Services (AWS) cloud, specifically, as an example here, as it is a commonly used option for boosting database performance and innovation.

Myth: There is Little Performance Benefit to Modernization of Databases in the Cloud

Modern cloud computing technology has advanced tremendously in recent history. With this advancement comes loftier expectations around database performance. And the truth is commercial databases are falling behind.

Commercial databases are less scalable overall. They can struggle with latency issues, especially when supporting modern applications. Plus, finding enough processing power can be a challenge. Companies are still dealing with chip and RAM shortages related to ongoing supply chain disruption.

Cloud providers, on the other hand, haven’t suffered as much. They buy components in bulk and have an overwhelming amount of equipment, infrastructure and memory for running cloud databases. Who benefits from this accessibility? Companies that have traded their on-premises infrastructure and commercial databases for the cloud.

One of the best features of cloud databases is their on-demand scalability. Organizations can easily scale with customer needs. In contrast, on-premises or private cloud solutions bind engineering teams to their hardware. These constraints don’t exist for cloud users.

Companies on the cloud can choose from a wide variety of plug-and-play solutions to optimize performance across decoupled services (AKA microservices). These microservices can each have their own database engines that scale as needed.
Myth: Cloud Databases Won’t Reduce my IT Costs

The second myth that holds companies back is the belief that IT costs won’t come down in the cloud. Theoretically, on-premises and cloud databases could be equally costly, but this is unlikely in real-world scenarios when you consider all factors.

With on-premises databases, companies pay in full whether they are buying direct or through a private cloud. They also typically have to buy licenses with fixed pricing. Even if organizations don’t use the maximum resources available to them, they are still on the hook for the full cost. AWS doesn’t operate this way.

On AWS, organizations only pay for resources consumed if they choose to go the on-demand route. IT teams can otherwise select from different savings plans if they are willing to make minimum volume commitments. Compared to fixed-price contracts, these approaches ensure that costs scale more closely with processing power. This level of elasticity is essential for modern companies that want to deploy modern applications.

Myth: I’m Better Off Leaving my Data On-Premises for Security and Availability

Outside of performance and pricing concerns, some still think that commercial databases are better from a security and availability standpoint. Yet, the cloud wins out here, as well. Cloud databases minimize manual work related to:

Encrypting data at rest

Encrypting data in transit

Defining port allowances

Preparing for compliance audits

This is crucial because manual work opens the door for human errors. On-premises databases put these tasks on IT’s shoulders and make administrators responsible for ensuring data integrity. There is no room for mistakes, which is scary when managing complex applications. The good news is that cloud databases alleviate this pressure.

For instance, Amazon Aurora maintains six copies of data across three availability zones. The solution boasts 99.99%+ durability with <30-second instance failover. Similarly, Amazon DynamoDB guarantees a reliability of up to 99.999%. AWS’ managed databases also offer 256-bit encryption with SSL certifications and rotating access keys.
Moreover, the cloud provider offers an identity and access management (IAM) solution that simplifies authentication and role assignments for users and resources. AWS also lets users set up security groups, which take away much of the complexity associated with protecting cloud-based infrastructure.

Myth: There’s Not Much I Can Do in Cloud Databases That I Can’t Do On-Premises

The fourth myth that keeps leaders from modernizing databases is the idea that they aren’t missing out on much by staying on-premises. But the truth is that commercial databases fall short in significant ways when trying to elevate legacy applications. Most are monolithic by design, which is antithetical to the agile nature of the cloud.

Organizations that want to deliver modern applications must have modern back-end infrastructure. Today, this means leveraging microservices–and having different databases for different application functions. Taking this approach allows organizations to scale distinct parts of their applications separately according to demand.

Myth: Commercial Databases Aren’t Compatible With Cloud Databases

The final myth that slows database modernization progress is the notion that popular commercial databases, like Oracle and SQL server, aren’t compatible with cloud databases. This isn’t true. Amazon RDS, for example, lets users run their Oracle and Microsoft SQL server workloads in a fully managed database service. Amazon Aurora is also fully compatible with MySQL and PostgreSQL databases.

For the most part, cloud databases allow companies to continue using familiar code, drivers, applications and tools without having to do much refactoring. And should leaders decide to migrate away from commercial solutions altogether, there are import and export tools that help accelerate the process.

Conclusion

To summarize, much of the conventional wisdom around keeping commercial databases is outdated. The cloud offers a much more compelling and exciting path. Cloud-native solutions enable companies to take database performance to the next level without any added complexity.
August 18, 2022
Overcoming Multi-Cloud Management Challenges

We’ve long talked about how the cloud is the future. But with the vast majority of corporate data now stored in the cloud, it’s clear that most businesses now realize this is a must-have in today’s increasingly demanding and agile business environment.

The cloud has been the backbone of the remote restructuring of the workplace over the last few years, enabling organizations to access data flows and critical workloads no matter where their employees are stationed. So much so that there’s been a dramatic rise in the number of enterprises deploying different service providers as part of a multi-cloud strategy. This enabled them to pursue greater agility and flexibility within a hybrid environment.

Recent research from Wanclouds found that most IT professionals use two or more cloud platforms today—and just under half (48%) said they used a mix of public and private clouds. However, the reality is that the more cloud environments your organization use, the more complex it becomes to both manage and secure them. Here are three multi-cloud management challenges impacting IT teams today and how they can overcome them.

Maintaining Uptime in the Face of Rising Security Challenges

Avoiding downtime has never been more important, nor has it been more challenging to achieve. Extreme weather events are becoming more frequent and increasing the risk of natural disasters or power outages, leading to data loss. In fact, 65% of U.S. and UK IT decision-makers said their organization experienced at least one data loss incident last year, while 21% admit they experienced two data loss incidents.

But it is, perhaps, the security challenges that pose the biggest threat to IT departments and the developer teams they support. Since the onset of COVID-19, organizations have been targeted for cyberattacks far more often and have been infiltrated by cybercriminals much more quickly. According to McAfee, there were 3.1 million cloud-based cyberattacks in 2020 alone. Unsurprisingly, these attacks are fueling concern in IT circles. 88% of U.S. and UK decision-makers said they’re at least slightly worried—and 15% are apprehensive—about cloud workload security in 2022.

While multi-cloud environments enable developer flexibility, they also cause inherent configuration and visibility problems, and this is where cyberattackers can take advantage of weaknesses. The complexity of migrating data across different cloud platforms and operating systems can also lead to delays with patching. The key is often ensuring visibility across these platforms and maintaining interoperability. Security platforms for multi-cloud environments must be set up to share threat information across providers and platforms. If not, threats can easily be missed, or even if they are detected, the response across infrastructure will be too slow to minimize damage.

With businesses accelerating their goal of becoming 100% cloud-native in the next few years, a cloud-based disaster recovery strategy must also be embedded within their management plans. Without one, every business risks losing vital data and having their systems, operations or services shut down by natural and man-made disasters, security risks, hardware failures and power outages.

Staying on Top of Hidden Charges

Cloud spending has skyrocketed in the past two years, and all signs suggest that organizations will continue to allocate more of their IT budgets to the cloud over the years to come. But not every business has the same resources as, say, Apple, which reportedly spent over $30 million a month on Amazon’s cloud, according to a 2019 CNBC story. The reality for most IT teams is that they’re constantly battling hidden charges and unexpected cloud costs that threaten to blow up their entire IT budget.

In fact, in Flexera’s 2022 State of the Cloud report, respondents self-estimated that their organizations wasted just under a third (32%) of their cloud spend, up from 30% last year. Meanwhile, Wanclouds research found that 54% of British and American IT professionals say they have been hit with unexpected cloud costs in the past year.

It’s a severe problem throwing IT teams for a loop from the migration stages of the multi-cloud plans right through to management and maintenance. Sure, cloud providers like Amazon provide basic tools for resource tracking and spend visibility. But truthfully, these features generally fall short of providing real-time and enterprise-wide insights on how to optimize costs.

To eliminate the sticker shock of unanticipated cloud costs, organizations need to have a defined plan for cloud cost management. Again, company-wide visibility into their cloud platforms’ billing systems is imperative and a way to track spending across their multiple accounts. It’s vital for companies to properly tag the infrastructure for understanding costs and maintain up-to-date inventory of all the resources.

Meeting Multi-Cloud Compliance Requirements

Every cloud user must comply with specific regulatory standards following industry guidelines and local, national and international laws. Organizations in the health care space, for instance, must comply with HIPAA rules, which are essentially a set of national standards to protect sensitive patient health information from being disclosed without the patient’s consent or knowledge. Meanwhile, manufacturing IT teams must abide by stringent intellectual property security protocols and protocols for other sensitive data like PCI and PII.

Ensuring compliance across multiple cloud environments is necessary, but undoubtedly a challenge. This is especially true in today’s attack-filled security landscape, where internal threats can pose as much risk to data security as external attackers. (32% of the organizations surveyed by Varonis found incorrect access authorizations and privileges assigned to users). Not to mention compliance requirements are constantly changing, making it even more difficult for organizations to meet demands.

This is why every IT and security professional invested in the cloud needs a solution that lets them set up compliance policies against their cloud infrastructure, making it easy for them to ensure compliance requirements in the case of any unforeseen natural disasters or security incidents. We’ve seen the damage that just one data breach can have on a business—costing them millions of dollars in cleanup and leaving them with considerable damage to their reputation. It’s imperative that an organization’s entire multi-cloud strategy prioritizes compliance and the security of sensitive data.

August 17, 2022
Building a Platform for DevOps Evolution, Part One

Winston Churchill once said, “To improve is to change; to be perfect is to change often.” While nearly everyone will confess that change is necessary to grow and succeed, we all know it’s much harder, especially when human behavior is involved. Professionals who work in DevOps environments know this all too well.

Adoption of DevOps principles and behaviors should be viewed as a gradual process. Many organizations make false starts or attempt to implement a change only to see it fail. Sometimes this is due to changes in leadership, market conditions or technology, but then organizations catch their second wind, build momentum and reaccelerate their evolution.

DevOps can Change

With a shared vision, mutual goals and hard work, DevOps can change to keep up with user demands. DevOps professionals will always confront two potent barriers: Cultural and process debt and technical debt.

Cultural and process debt occurs when leaders refuse to delegate authority, distrust their teams and lack a value stream approach to work. Technical debt develops when monolithic systems are tightly coupled, automation is delayed for testing, environments and version control and there is no access to cloud technologies.

Although these obstacles are challenging and complicated, many organizations still meet them head-on. Evolution depends on a habit of making improvements every day through the use of an improvement kata such as PDCA (plan, do, check, act), which involves long-term strategies and objectives and continual experiments between the current and next target states.

An evolving DevOps approach maintains moving pieces across a large organization of hundreds, thousands or even tens of thousands of people who cannot change overnight. Many lack the will and the cognitive capacity to embrace and enact change.

This environment requires release managers to delegate power and transition from centralized commander to an enabler, understanding different value streams have varied skills and requirements. Mark Peters, a cybersecurity specialist and DevOps leader with years of military operations experience, outlined his top release management traits in the context of DevOps ways of working as automated processes, distributed accountability and accelerated delivery.

PDCA Improvement Kata

DevOps proposes that we adopt an improvement kata in our day-to-day efforts as an improvement habit, consisting of four steps that are repeated:

1. Set the long-term vision
2. Understand the current state
3. Identify the next target state
4. PDCA between the states

Experimentation is at the heart of the PDCA cycle. The most practical experiments are empirical, data-driven and date-specific. Using a value stream management platform to give data to support the experiments leads to data-driven discussions on real and measurable success.

In the next part of this series, we will begin exploring different experiments DevOps organizations can conduct to drive change and improve their release management procedures.

August 16, 2022
The Rogers Outage of 2022: Takeaways for SREs

When, eight years from now, folks are creating lists of the top IT incidents of the 2020s, there’s a good chance that they’ll include the Rogers outage of 2022. The failure, which made internet and cellular network service unavailable for more than 12 million users across Canada, was one of the most significant outages in memory, in terms of both the number of affected accounts and the level of service disruption.

For site reliability engineers, there’s a lot to learn from this incident, from both a technical and a crisis-management perspective. Here’s a look at our main takeaways.

The Rogers Outage: What Happened?

Before diving into lessons for SREs, let’s go over what happened during the Rogers outage and what we know about the cause of the incident.

Rogers is a major telecommunications provider that serves around 12 million customers in Canada. Early on July 8, 2022, it suffered an incident with its infrastructure that made internet and wireless phone service unavailable for most or all of its customers. The outage was so complete that users couldn’t even make 911 calls or withdraw money from ATMs–which means the failure brought down not just ordinary services, but also what you might consider resources that are essential for normal daily life.

Connectivity was restored for most users within about 15 hours. However, some customers reported continuing disruptions for several days after the beginning of the outage.

What Caused the Rogers Failure?

At first, Rogers did not reveal much technical detail about what triggered the incident. A statement attributed to the company’s CEO the day after the outage said that a “maintenance update in our core network … caused some of our routers to malfunction.” But that didn’t explain specifically which type of maintenance update the company was trying to apply, why the update didn’t go as planned or which types of routers were affected.

It wasn’t until several weeks later, when Rogers submitted a response to questions from the Canadian Radio-Television and Telecommunications Commission, that more technical detail came to light. Rogers explained to the CRTC that, during an update process, the company ran code that deleted a routing filter. It’s still unclear exactly how the bad code made its way into the update routine.

Lessons for SREs

For SREs, the Rogers outage is an example of a worst-case scenario. It was an extremely disruptive incident that took a relatively long time to mitigate, at least when you consider how severe the outage was. That’s why SREs should use Rogers’ experience to assess what SRE teams can do to avoid issues like this–and manage them more effectively when they do happen.

Test Your Updates

From a technical perspective, one of the biggest takeaways from the Rogers outage for SREs is that you should test updates thoroughly before applying them. If engineers had tested their update routine more carefully, they would presumably have detected the bad code that deleted the routing filter before the deletion actually took place.

To be fair, Rogers is hardly the first company to experience a major outage due to a bad update. Atlassian, for instance, had a similar issue earlier this year, when a buggy migration process caused a long-lasting outage.

If you’re an SRE, you don’t want your company to be the next to make headlines when something that should be a simple change, like an update, triggers a massive outage. Test, test and test again before bringing your changes into production.

Build in Redundancy

It also seems reasonable to conclude that Rogers didn’t have redundancy built into its infrastructure in the form of backup routers that could take over when the primary routers crashed due to the bad update.

This is another reminder to SREs about why it’s so critical to avoid putting all of your eggs in one basket or creating single points of failure. Redundancy may cost more, and it may not totally prevent disruptions. But it can make them less severe and easier to recover from. It’s not hard to imagine the Rogers incident being a lot less severe if even just 20% or 30% of its traffic could have been automatically redirected to backup routers. That might have been enough to keep critical services operational, at least.

Have an External Crisis Communications Plan

Rogers has received more than a little criticism for not communicating well with its customers during the outage. The company waited hours before releasing an initial statement about the failure. It was not very responsive to media queries early on, and, as noted above, it took several weeks before Rogers finally released technical details about what went wrong (and even then, the details emerged only because of a government inquiry into the incident).

The lack of rapid, transparent communication probably wasn’t deliberate. We have to believe that Rogers didn’t want to leave customers in the dark. Instead, the slow communication likely reflects the absence of a pre-planned strategy for external incident communication in a situation where Rogers’ own networks were non-operational. Without a communications plan, Rogers presumably struggled to decide which information to share, or how to share it, in the midst of a major crisis.

If Rogers had a playbook in place for communicating with the public and media about this type of issue before its network went down, it probably would have been able to share information more effectively–and it may not have faced as much of a backlash from customers and regulators.

Internal communications, at least, presumably were smoother. The company did manage to get service back up for most customers within about fifteen hours, which probably would not have happened if its engineers lacked a means of communicating with each other as they worked to resolve the incident.

Conclusion

For SREs, the Rogers outage of 2022 is a lesson in the importance of testing, designing redundant infrastructure and preparing for transparent communications before crisis strikes. While more careful planning in these respects may not have totally prevented the incident, it would likely have at least reduced its severity and mitigated the backlash that Rogers now faces from angry customers and regulators.

August 15, 2022
5 Ways to Prevent an Outage

In today’s always-on, ever-connected world, we all expect 100% availability.

What gets in the way of this? The devil is in the details. Over time, everything breaks: Disks, nodes, containers, networks, DNS servers and configuration issues can all lead to major outages.

Amazon’s 13 minutes of downtime in August 2021 translated to almost $5 million in lost revenue and Amazon Web Services’ three outages in December of last year cost millions more. And it was widely reported that the October 2021 Facebook (including related products: Messenger, Instagram and WhatsApp) outage resulted in an estimated $100 million in lost revenue due to a network configuration error.

There have been many outages of varying scope and duration—some attributed to technology breaking, others traced back to people making mistakes. In fact, a 2017 Amazon outage was attributed to “human error”–or more specifically, one employee’s typo. The employee intended to take a small number of servers offline to fix a billing system issue but the incorrect command led to several servers being affected and a major outage.

So, what can your company do to reduce the risk, duration and impact of a potential outage? For one, companies can’t pretend that if they only try harder, they can prevent humans from making mistakes. Do you have dozens of people manually keying in hundreds of commands every day? If so, a mistake is inevitable. Companies should instead investigate how and why one small blunder in a command line could do widespread damage. Guardrails and redundancies should be in place to protect against and minimize these types of incidents.

As the enterprise continues its digital transformation toward a multi-cloud, hybrid and distributed world, here are five guidelines to consider—guidelines that can help you preempt and prevent your next big outage.

1. Always Consider Your Blast Radius

Blast radius measures how one error, typo or malfunction can impact other parts of a system.

What is the chain of components or services that is impacted by one error?

What are the hidden dependencies?

How many customers would be affected and for how long?

More is connected than you think; discover how it’s all related in advance to eliminate surprises.

If too many systems in a network are mutually dependent and interconnected, the result is deadlock. In other words, when one breaks, the other also fails. We have seen this in past Facebook outages. A single erroneous command sparked a domino effect that shut down the backbone connecting all of Facebook’s data centers globally.

How can you prevent this scenario? Plan for the worst possible scenario. When it comes to mission-critical systems, take an intentional approach to situation failover and backup. What if A, B and C all happen simultaneously? Also, think about your cost versus reliability trade-offs. Maybe you can’t afford two full sets of servers but can afford a full set of archived code.

2. Smooth Out Rough Edges

Consider that core infrastructure and code get used (and tested) a lot more than edge code/infrastructure. Basically, the core is hardened while edge infrastructure is more vulnerable. You should be more proactive at the edge around identifying vulnerabilities and doing explicit testing to counter this risk.

The 2014 Google disk erase/satellite outage is a cautionary tale involving the edge. In Google’s case, the edge networks offered a key advantage: A better connection for the customer due to lower latency since data is accessed from a source closer to the user as opposed to the core network. Google had a CDN/edge network for TCP connection termination, caching for various apps, DNS regulation, etc. Teams needed to routinely upgrade a satellite rack and run decommission automation—and this automation failed part of the way through. Error message. They retried the decommission process again to debug the failure. The manual re-execution of the decommission automation meant thousands of satellite machines were decommissioned and, within minutes, the disks of all satellite machines globally were erased. The result: Customers experienced an increase in latency. Behind the scenes, it required two days of all-hands-on-deck to reinstall these machines.

3. Think Escalator, Not Elevator

A good system has both defense and depth. Considering that it takes three or four things to (really) go wrong, think about failure cascades. Bluntly put, limit dependencies. Measure the longest dependency failure path and mean-time-to-recovery (MTTR). When you do this, think escalator not elevator. What do I mean? Well, when an elevator fails, you are left with a shaft; an escalator that fails still works as a staircase.

In 2016, severe weather in Sydney, Australia resulted in an AWS EC2/EBS outage. First, the utility provider suffered a loss of power at a regional substation. This failure resulted in a total loss of utility power to multiple AWS facilities. In one of the facilities, AWS power redundancy didn’t work as designed and power was lost to a significant number of instances in that availability zone. To make matters worse, the instances that lost power lost access to both their primary and secondary power as several power delivery lineups failed to transfer load to their generators. Also, a latent bug in AWS instance management software resulted in a slower-than-expected recovery of the remaining instances.

The lesson? This correlated power delivery lineup failure illustrated that failures are not independent. You must design for the largest correlated failure in combination with the background noise of ongoing independent failure. Recovery code needs to be scale-tested against the largest expected failure.

4. Stay Ahead of Digital Certificate Rotations

Don’t underestimate the importance of digital certificates—and their expiration dates. A few years ago, LinkedIn allowed one of its SSL certificates to expire and it knocked out LinkedIn sites in the U.S., UK and Canada. In 2017, Equifax missed a breach for 76 days because of an expired certificate. The average annual cost of unplanned outages due to certificate expirations is $11.1 million.

For example, you could have a load balancer sitting in front of several servers. The load balancer could have an up-to-date certificate while the servers behind it are expiring. It is important to check for expiring servers in multiple ways. Ideally, you check the boxes in two ways: Ping the HTTP endpoints and also have scripts that check the certificate files themselves.

Expired certificates (TLS for example) can lead to encryption and authentication risks as well as lost internet connectivity and larger outages involving websites and applications. To avoid this, you continuously check, renew and deploy certificates—and this can be achieved via automation. Think about automatically rotating expiring certificates.

5. Tackle Toil to Empower Your Teams

What’s the biggest headache for site reliability engineers (SREs) and those who manage production operations? Toil.

We can define toil as manual, tedious tasks that engineers perform within production environments. Basically, toil wastes time and slows down operations. As a result, most SREs and engineering teams strive to minimize toil within their workflows.

In production operations, you should work to limit toil. Make sure your team is regularly investing time in proactive work; the reactive work shouldn’t take most of their time. Toil reduction gives developers, DevOps engineers and SREs more time to focus on projects that create net-new value for the business (new application features). Google stated that it has an “advertised goal” of keeping toil “below 50% of each SRE’s time.” Basically, no engineer should spend more than half of their time on manual, repetitive work. But the ultimate goal should be to get toil as close to zero as possible. Automating incident remediation can help SREs spend less time on toil and less time on-call.

Also, when you implement fixes—especially people and process fixes—ask yourself if this is a systemic fix or one that is relying on best intentions. Look for tools to safely empower your team (with guardrails). As you work to identify major vulnerabilities and dependencies (as discussed above), you should automate away the most common issues while planning for the “corner cases” (or the 1% of 1%) that may lead to major outages (a.k.a. the worst-case scenarios).

As you consider these guidelines, make sure you focus on putting actual mechanisms in place—not one-off, “hopeful” directives. Do not rely on people to remember best practices from a training session—these are not mechanisms. Instead, design processes that can be systematically implemented. When you do a post-mortem, look for concrete programs you can put into place that will prevent the same mistake from happening again. Automation can help you achieve this level of confidence, management and quality control—and it may just keep you one step ahead of a potential future outage.

August 15, 2022
Next-Level Tech: DevOps Meets CSOps

Operations are some of the most important aspects of an organization. In tech, DevOps has dominated the conversation due to its necessity in keeping up with the rate of building products, applications and infrastructure with code. DevOps encompasses much more than a team of developers; it is a culture, a set of practices and tools that allow organizations to deliver applications with speed and scale and then effectively operate. This philosophy has been incredibly useful over the last decade but has lacked a connection with other parts of the organization. Developer teams are connected, but within their silo, their work is lacking integration with the end user, diluting the opportunity to deliver the best solutions for their customers. More immediately, developers, DevOps and ITops cannot see the breadth of impact when working in an autonomous environment, leading to poor prioritization and remediation.

Enter customer service. Another pillar of organizational strength, customer service is not only a necessity but has risen to become a strategic function for the business. They are the first responders of a business when digital services don’t work as expected or their customer expectations are not met. Customer service agents are highly trained, well-funded and armed with the latest technology so they can be the most effective first point of contact for customers. Customer service teams can learn something from developers in their organization and work within the same kind of ‘operations’ model that dictates philosophies and practices: in other words, CSOps or customer service operations.

Why CSOps?

Customer service is an incredibly powerful element of business operations, as it is tied to user experience. Consumers are not shy about announcing when they have experienced poor customer service, and the opposite is also true. End users also loudly trumpet the moments that they have received strong customer service; that goes a long way in creating loyalty, repeat business and retention. In a study from Bain & Company, it was noted that customers were four times more likely to buy from a competitor if they experienced a service-related problem rather than an issue based on price or product. Leading with customer service, like many e-commerce platforms have done, is a new standard of operating and is critical to meeting the expectations of customers. In a B2B setting, customer service is no different. Customers or users are drawn to organizations that can quickly remediate their concerns in a frictionless and efficient manner.

Customer service is now at the forefront of corporate initiatives and isn’t going anywhere. In a recent article from the Wall Street Journal, analysts and executives weighed in on CIO budgets and forecasted priorities amidst growing recession concerns. With cuts coming to some areas, customer service was not one of them. The overall sentiment is that “businesses see more value than ever in tech that will enhance the customer experience and give them an edge in an increasingly tight market.”

In today’s organizations, you cannot compete with DevOps if you do not have CSOps. They are symbiotic in their relationship to the bottom line—if products or applications do not meet the customer’s desired outcome, communication from customer service teams must connect with developers and convey what’s wrong or missing. Then, developers can work to provide the proper solutions. Customer service operations feed the DevOps pipeline to create products and services that address customer needs. In other words, you cannot succeed with DevOps if you don’t have CSOps.

Creating the DevOps and CSOps Relationship

The truth is that most CSOps and back office teams like SREs or ITOps don’t have a good relationship, nor do they have effective processes to communicate with each other. This disconnect not only creates problems with customers, but the lack of internal communication creates fault lines across teams and an inability to produce results efficiently. With the trajectory of technology organizations over the last decade ignited by the SaaS model, flexible work and the rise of applications, it is important to look at best practices within an organization and understand how teams can work better together.

While developer teams continue to iterate and produce code, customer service teams are constantly fielding customer feedback, concerns and complaints. Customer service agents are often the first to learn if something is broken and where—if organizations are not using their front-line workers as a tool in remediation and incident response, they are missing out on valuable information that can help them find a solution.

Beyond just fielding complaints and reports about product and service issues, customer service teams are able to aggregate and correlate reports, as well as gather additional information from reporting users. When something goes wrong, mobilizing the required rapid response can often be difficult. If there is a lack of communication between the customer service teams and the developer or engineering teams, a customer complaint about a product or service could take too long to make it through remediation. Creating a rapid connection from the customer service team to the developer team allows customer service agents to have visibility into the problem and the solution, offering feedback to the customer in real-time as the solution is being implemented. It is this ‘Ops’ culture on both the customer service and development teams that supports the organization from end to end.

The symbiotic relationship of DevOps and CSOps can benefit any organization, but adoption requires visibility and bi-directional communication. Development teams that are in close contact with their customer service teams benefit from having better, real-world user information that can then be incorporated into new features and improvements. When customer service teams have a clear path to escalate customer-reported issues—with the right context to the appropriate teams—time to resolution decreases by eliminating silos and breaking down the walls between CSOps and DevOps.

It is time that every organization integrates customer service teams as part of delivering on customer satisfaction instead of just reporting on it. The integration with engineering and back office teams creates a culture of service-first operations.

August 12, 2022