devopsy.kinsta.cloud

Category: IT as Code

  • Federal Agencies Share DevSecOps Guidelines

    Federal Agencies Share DevSecOps Guidelines

    The National Security Agency (NSA), Cybersecurity and Infrastructure Security Agency (CISA), and the Office of the Director of National Intelligence (ODNI) have published a set of DevSecOps best practices based on the Enduring Security Framework (ESF).

    Developed by a public-private cross-sector working group led by NSA and CISA, the ESF framework was created in the wake of the breach of the SolarWinds software supply chain that led to malware being distributed downstream to a number of organizations that relied on the company’s IT management platform.

    The Securing the Software Supply Chain for Developers set of best practices published by NSA, CISA and ODNI takes the ESF a step further. The guidance walks organizations through the processes required to develop secure code, verify third-party components, harden build environments and deliver code.

    The best practices provided are the latest dividend of an executive order issued by the Biden administration that requires every federal agency to review the security of its software supply chains. The expectation is that enterprise IT organizations will conduct similar reviews using the best practices and frameworks defined by the federal agencies that specialize in cybersecurity.

    The challenge, of course, is embedding those best practices within existing DevOps workflows. In theory, more responsibility for cybersecurity is shifting left toward application development teams. However, the level of cybersecurity expertise that exists today among application developers is limited at best. Cybersecurity was an elective in most developers’ training and education programs, so it should come as no surprise that few of them ever attended a cybersecurity class.

    There’s a clear need to improve cybersecurity training for developers, but the real work involves automating cybersecurity processes within the workflows managed by the tools developers employ to write code and the platforms that DevOps teams use to deploy applications, said Mitch Ashley, principal analyst for Techstrong Research, an arm of Techstrong Group that is also the parent company of devopsy.kinsta.cloud. “Security needs to be part of the toolchain, workflows and how the software is designed,” he said.

    Ideally, organizations should appoint a security champion that will work with development teams to consistently implement a set of DevSecOps best practices, added Ashley.

    Of course, the security champion doesn’t eliminate the need for cybersecurity professionals. Rather, it reduces the pressure when there is a chronic shortage of cybersecurity professionals. In the absence of any formal cybersecurity review during the application development process, it’s simply too easy for developers to inadvertently introduce malware into a software component or misconfigure a cloud infrastructure service. In either case, cybercriminals are increasingly finding it easier to compromise software supply chains at multiple points.

    It’s not clear how much the federal government’s guidance might impact the overall state of application security. However, one way or another the level of security scrutiny being applied to how applications are built and delivered is only going to increase in the months ahead. The challenge is finding a way to deliver secure code without necessarily slowing down the rate at which applications are currently being built and deployed.

  • How to Go Multi-Cloud: A Simple Three-Step Approach

    How to Go Multi-Cloud: A Simple Three-Step Approach

    Odds are your business already uses cloud technology for automation, cybersecurity, IT infrastructure updates and other purposes. But you don’t have to stick to a single cloud server or provider. In fact, you can take advantage of multi-cloud strategies for your organization and reap the benefits for years to come.

    That said, multi-cloud adoption and optimization can be tricky, especially if you don’t have a plan in mind from the get-go. Today, let’s break down a simplistic yet effective three-step approach to go multi-cloud without running into major problems.

    Step One: Map Your Cloud Zoning Policy

    Your first major step should be to map out your cloud zoning policy and plan. In a nutshell, the cloud zoning choices you make can impact your commitments, costs, and even the eventual functionality of the multi-cloud setup.

    By mapping out your clouds, you also map out which processes and applications will run on each individual cloud server or provider. In essence, you decide what gets locked into one cloud, what data has to be ported between clouds and what applications must run on multiple clouds simultaneously.

    Need an example? A cloud zoning policy might cover whether you keep your web surfing or data analytics on separate cloud providers or on the same cloud servers. 

    You should map your cloud zoning policy regardless of whether you do the setup or hire a service. In the latter case, offering a read-to-go zoning map will make the service’s job easier and minimize the possibility of mistakes and/or hiccups.

    How to Determine Optimal Cloud Zoning

    Naturally, figuring out how to optimize cloud zoning can be difficult. The best way to do this is to determine your special focus areas. Alternatively, consider how you want your multi-cloud strategy to best benefit your organization.

    Say that you want to guarantee 100% uptime for your customers or visitors, even if your service goes down or if you experience a data breach. In this case, you can set up your cloud zoning strategy to balance the data load across multiple clouds at the same time.

    Or say that you want to make sure your users have global availability 24/7/365. In that case, you can set your cloud zoning policies to ensure that users can access your data or websites from anywhere around the globe at any point.

    Essentially, figure out what matters most to your organization–and what you want the most from multi-cloud optimization–and zone your cloud applications and policies accordingly.

    Step Two: Architect the Multi-Cloud Environment

    The next major step is to architect your multi-cloud environment. This involves looking at the high-level design of the environment and creating a rock-solid foundation for multi-cloud servers.

    At this stage, you should have at least some idea of how your business will grow and how the multi-cloud environment will assist with its scalability and resource needs. You need to know:

    • Where your data science and machine learning applications need to be
    • Where your product application serves from
    • Where your data warehousing takes place
    • Where cloud security is run from
    • How each of those functions grows alongside the others

    In the architecture phase of a multi-cloud setup, you need to figure out which projects and apps are so-called “cloud agnostic.” Cloud agnostic projects and applications don’t have to be portable; because of this, they can lean on managed services or proprietary IT infrastructure from your organization.

    How to Set Up an Ideal Multi-Cloud Environment

    To reap the most benefits out of a multi-cloud environment, you should adopt a containerized and flexible approach. Not only is this cost advantageous, it also allows you to set up your multi-cloud environment with as much versatility as possible.

    If you architect or plan your multi-cloud environment so it is containerized and flexible, you can work with practically any infrastructure-as-a-service (IaaS) provider. Therefore, you can transition to different cloud hosts or service providers whenever necessary according to your budget or other factors.

    To do this, be sure to perform heavy forecasting. You need to know how much data storage your computing will require, the database utilization your organization will need, the number of computer nodes you’ll likely require and so on.

    As a side bonus, containerization in your multi-cloud setup means that if one server or process goes down, the others are less likely to fail from a cascade effect as well.

    Step Three: Prep for Contracts and Forecast Costs

    The last step in going multi-cloud for your organization is handling the financial aspect of the process. You need to prepare for contracts and commitments alongside forecasting costs.

    Forecasting is critical in this phase because you’ll prepare for contracts and select different cloud services. Because of this, you need to understand how flexible your budget is relative to your infrastructure requirements.

    Specifically, you’ll need to match the costs to every multi-cloud forecast and build your overall resource and money consumption budget. Basically, you need to know:

    • How much you’ll likely spend on multi-cloud processing and services
    • Whether you need certain features, such as website monitoring tools like Visualping
    • Whether your budget can take that forecasted strain

    If the answer to the third question is “no,” you may need to go with a more affordable service or adjust your multi-cloud environment architecture and zoning policies. By forecasting costs ahead of time, you won’t run into a disaster scenario where you already have your multi-caught environment up and running but can’t pay for it, forcing you to scramble as a result.

    Minimize Commitment Risk

    Luckily, there are ways to minimize commitment risk and avoid budgetary disasters. For example, you can use commitment buy-back guarantees, or you can use flexible commitment options like that from AWS. These include computing savings plans, which consume cloud resources in any region and which result in very low savings rates.

    Of course, you can and should also practice very careful budgeting and accounting. Once you ensure that your commitment costs and savings are attributed to the right services, server resources, applications, etc., you’ll have a better idea of how much money and other resources you really need and avoid over-committing to a provider that is too expensive.

    Then you can further avoid commitment risk when planning your migrations of applications between different vendors carefully. Budgetary costs can skyrocket if migrating applications and data between vendors take longer or result in unexpected complications. Therefore, you should make sure that your migrations go quickly and easily or that a cloud service provider offers assistance during this phase (possibly as part of a deal to get you to sign with them in the first place).

    Wrapping Up

    As you can see, you can transition your organization to multi-cloud in a matter of months. But even if you follow the above steps perfectly, remember that you won’t optimize your commitments, performance and costs up to 100%. Still, you can prep your organization for long-term success and multi-cloud benefits with the right preparation and planning.

    With the right cloud services provider, you’ll gain additional assistance and support throughout this process and be able to take advantage of the extra resources from a multi-cloud setup quickly and easily. 

  • 90% Of Orgs Say Multi-Cloud Helps Achieve Business Goals

    90% Of Orgs Say Multi-Cloud Helps Achieve Business Goals

    Multi-cloud is the new normal. And thankfully, this new normal isn’t just happenstance—it’s driving positive outcomes.

    New findings show that most organizations benefit from using multiple cloud providers. Multi-cloud adoptions are helping achieve business goals, such as improving reliability and scalability and increasing overall security and governance. However, nearly all organizations (94%) admitted that they have some avoidable cloud spending and skills shortages continue to hold back a fully-realized multiple-cloud success.

    HaschiCorp recently released the 2022 State of Cloud Strategy Survey: Making Multi-Cloud Work. In addition to showcasing the positive outcomes of multi-cloud adoption, the report found a trend toward automation frameworks that help operationalize multi-cloud. Below, I’ll explore the report’s key takeaways to see what CloudOps and platform teams can garner from current multi-cloud trends.

    Multi-Cloud is Achieving Business Goals

    First off, the use of the public clouds is increasing dramatically—the majority of public cloud workloads will likely double in the next two years. To support this influx, more organizations are moving toward multi-cloud. According to the HashiCorp survey, 60% are already multi-cloud and 21% plan to use multiple clouds within the next year.

    The good news is that a multi-cloud strategy appears to be working. A full 90% of respondents said multi-cloud is helping their organizations achieve their business goals. This figure is an impressive jump from HashiCorp’s 2021 survey, in which only 53% of organizations said multi-cloud was helping them achieve their business goals.

    So, what exactly are companies extracting from multi-cloud adoption? Well, the topmost business goal driving multi-cloud is reliability, at 46%. This is followed by digital transformation (43%), scalability (42%), security and governance (41%) and cost reduction (39%). As we can see, multi-cloud equally benefits various areas.

    Centralized Automation Supports Multi-Cloud

    With the rise of multi-cloud, CloudOps is becoming its own discipline. As such, it’s common for an organization to build a dedicated Cloud Center of Excellence (CCoE) or strategy team to manage cloud operations—86% said they relied on a cloud platform team. Such a team is often in charge of standardizing cloud services, developing cloud best practices, creating operational policies and related duties.

    To aid cloud platform teams, cloud automation is becoming increasingly important. The survey found that 99% see automation as important for operationalizing multi-cloud infrastructure. Multi-cloud automation could help solve inconsistencies across varying cloud workflows. Respondents also found it can be applied to deliver more flexible infrastructure, improve security and governance and better utilize cloud resources. Improving security is especially important here, as 89% said security is a key driver of cloud success.

    So how are these automations built? Well, folks are rarely developing cloud automation tools from scratch—only 27% do so. It’s far more common to either use open source or commercial tools for cloud automation. It’s pretty equal across the board whether these tools are run ‘as-a-service’ or are self-hosted.

    Regardless of the tooling type, companies are experiencing a broad set of benefits from using middleware to manage their multi-cloud operation strategy. 49% rank scalability as a benefit. This is followed by infrastructure provisioning (42%), cost optimizations (40%) and more flexible deployment options (39%).

    (Nearly) Everyone is Overspending on Multi-Cloud

    Unsurprisingly to some, the study found that 94% of organizations are wasting money in the cloud. The following were the top cited areas that contribute to cloud overspending and waste: Idle or underused resources (66%), overprovisioning resources (59%), a lack of needed skills (47%) and manual containerization (37%).

    The cloud overspending dilemma necessitates forethought into optimizations and reduction of idle resources. As I’ve previously covered, some organizations are even tinkering with employing a cloud economist, or Cloud CFO, to keep escalating cloud fees in check. To help this mission, 43% say consistent and automated tooling has or would lead to better utilization of cloud resources.

    Although wasteful spending is pervasive, the majority of companies are forecasting their cloud spending correctly. More than half (57%) said that in 2021 their organization’s projected cloud spend aligned with their actual cloud spend.

    Getting the Most Out of Multi-Cloud

    Multi-cloud is becoming more common across all IT divisions—as is middleware to operationalize and standardize logic between multiple clouds. And, widespread business success with multi-cloud signals increased maturity in this area.

    Yet, multi-cloud doesn’t come without its challenges. Skill shortages rank as the highest factor impeding an organization’s ability to operationalize multi-cloud. Businesses must face other challenges, such as siloed teams, compliance and risk management and lack of training in order to fully realize multi-cloud. Standardized operating models, along with shared workflows and policies, are possible ways to combat these roadblocks.

    The HashiCorp study surveyed 1,000 technology practitioners and decision-makers from around the world. HashiCorp also partnered with Forrester to provide additional analysis of the findings. To read the results for yourself, check out the full report here.

  • VMware Embraces Graph Technology to Manage Multiple Clouds

    VMware Embraces Graph Technology to Manage Multiple Clouds

    At the VMware Explore 2022 conference today, VMware launched a portfolio of multi-cloud management tools, dubbed VMware Aria, based on a graph-based data store that captures the resources spanning multiple clouds and their relationship to one another.

    Accessed via a VMware Aria Hub platform previously known as Project Ensemble, VMware Aria Graph provides a common definition of applications, resources, roles and accounts in near-real-time across a multi-cloud environment that can consist of millions of nodes.

    VMware CEO Raghu Raghuram told conference attendees the Aria Graph is at the core of an effort to enable IT organizations to move beyond “cloud chaos” to embrace a next-generation “cloud smart” approach to managing distributed computing environments that span multiple platforms.

    Based on an event-driven platform, the VMware Aria Graph aggregates data pulled from public application programming interfaces (APIs) into a data store residing on that cloud. The data store is based on a federated model so there is no need to move data into a central repository. Instead, VMware is making use of GraphQL to launch queries across a federated data model. In addition, IT teams can discover dependencies and analyze historical configuration data to discover the root cause of any issue. All operational data is now aggregated in a way that creates a single source of truth.

    VMware has also provided a set of API services that can be used to integrate the Aria Graph data store with third-party platforms.
    In general, the company is committed to accelerating “the great replatforming” of IT environments spurred by digital business transformation initiatives, said Raghuram. The challenges IT organizations are encountering are roadblocks that involve everything from a lack of skills, the weight of existing enterprise applications and fragmented operations and cybersecurity models, he noted.

    The VMware Aria graph data store will enable the company to automate a wide range of management tasks, Raghuram said. That includes using the right cloud for the right application and governing and accessing them at the lowest cost possible, added Raghuram. The ultimate goal is to enable a unified developer experience that provides a frictionless multi-cloud experience across a consistent set of enterprise infrastructure, he said.

    VMware offerings that leverage the VMware Aria Graph include VMware Aria Guardrails for enforcing IT policies, a VMware Aria Migration tool for automating assessment, planning and execution using the VMware HCX platform and a VMware Aria Business Insights tool that surfaces business insights using machine learning algorithms. The rest of the VMware Aria portfolio is anchored around extensions to existing platforms in the form of VMware Aria Cost powered by CloudHealth, VMware Aria Operations and VMware Aria Automation.

    VMware also announced today that its existing VMware Cross-Cloud Service offering for deploying its stack of software across multiple clouds has been extended to add support for the Microsoft Azure cloud.

    Overall, VMware claims there are now five million developers in its ecosystem that collectively launch 350,000 projects every week. That equates to roughly 35 new applications a minute, according to Raghuram. All told, VMware claims there are 85 million workloads running globally on its platform.

    The bulk of those workloads today still run in on-premises IT environments. In fact, Raghuram noted that cloud platforms still only account for $413 billion in IT spending each year, which is only 10% of the total IT spending that reaches $4.4 trillion each year.

    VMware, which is expecting to complete its acquisition by Broadcom in the next few months, is betting that as more of those workloads move to the cloud, the need to centrally manage heterogenous IT environments will play to its strengths.

  • TidalScale Enables Hot-Swapping of Failed Servers

    TidalScale Enables Hot-Swapping of Failed Servers

    TidalScale has updated its namesake software for automating server management to include the ability to detect and prevent impending failures and enable IT teams to hot-swap another server.

    Version 4.1 of TidalScale also adds server wear leveling and preventative health checks that identify servers that should be replaced based on their remaining estimated life span.

    TidalScale CEO Gary Smerdon said that capability is especially critical given the high rate of DRAM memory failure that results in a high rate of downtime caused by server failures. The latest update to the company’s software-defined server management software reduces the failure rate in on-premises or cloud computing environments by a factor of 100 to ensure 99.999% uptime, he said.

    The company also announced that its software is now available on the Amazon Web Services (AWS) Marketplace.

    TidalScale employs machine learning algorithms to aggregate and virtualize CPU cores, memory and I/O across servers to present operating systems with a single software-defined server. That makes it possible to hot-swap individual servers to ensure availability.

    The single biggest cause of downtime in IT environments is memory failures; they are now 100 times more likely than they were a decade ago, noted Smerdon. In most cases, the root cause of the issues is the demands of modern applications running on legacy infrastructure that was not intended to support the latency requirements of those applications, he added.

    That issue can be especially acute in cloud computing environments where infrastructure is often shared by multiple applications, said Smerdon.

    It’s not clear at what rate cloud service providers are replacing legacy server infrastructure, but the ability to determine when to replace infrastructure gives IT teams an added measure of control over servers regardless of where they are deployed. The biggest issue, of course, is not so much replacing a server as it is determining which one actually failed. It often takes longer to identify a failed server than it does to bring a new one online.

    TidalScale is made available by both cloud service providers and manufacturers of servers used in on-premises IT environments, but it needs to be configured and deployed by an IT team. The overall goal is to not only increase system availability but also reduce the total cost of IT by managing infrastructure more efficiently across the life span of a server, said Smerdon. Some IT teams routinely replace servers at specified intervals, regardless of actual usage, to avoid downtime. TidalScale enables IT teams to determine which servers to replace based on the actual wear within the context of a self-healing server farm, he noted.

    Given all the things that might potentially go wrong in an application environment, not having to worry about server failures should enable DevOps teams to spend more time optimizing application performance.

    In theory, of course, modern applications based on microservices should be able to reroute calls to application programming interfaces (APIs) anytime a server becomes unavailable. However, over time it’s not uncommon for single points of failure to emerge as applications are updated. In the meantime, the majority of applications running today are based on legacy monolithic architectures that are much more susceptible to server failures that, unfortunately, are still far too common.

  • OpenText to Create Software Powerhouse – Acquires Micro Focus for $6B

    OpenText to Create Software Powerhouse – Acquires Micro Focus for $6B

    OpenText this week announced it will acquire Micro Focus for $6 billion. The acquisition is part of an effort to reinvigorate a broad portfolio of information management, application development and testing, integration and cybersecurity platforms by accelerating the rate at which they are transitioned to OpenText’s managed private cloud.

    The acquisition is expected to close in the first quarter of 2023, and the combined company expects to reduce costs by an additional $100 million on top of the $300 million Micro Focus previously committed to cutting as part of an ongoing effort to return to profitability and growth.

    OpenText CEO Mark J. Barrenechea told analysts the ultimate goal is to enable organizations to accelerate their digital business transformation initiatives using a set of application programming interface (API)-centric offerings that reside on a private cloud.

    Digital transformation represents the largest opportunity in the history of computing, he said, but Barrenechea also acknowledged that a lot of companies are still struggling with making that transition. “There has to be a better way to get digital transformation right,” he said.

    The Micro Focus portfolio includes everything from a robotic process automation (RPA) offering, a recently-launched IT service management (ITSM) offering—dubbed Service Automation Management X—along with the Vertica database and LoadRunner testing tools. There is also a slew of cybersecurity platforms that include ArcSight, Fortify, IDOL Zenworks and Sentinel along with legacy COBOL application development tools and even the remnants of Novell.

    In effect, OpenText now has one of the broadest software portfolios in the industry after previously acquiring Documentum, Carbonite, Hummingbird and Zix. “The strategy appears to be to simply gather up all the pieces on the chess board,” said Mitch Ashley, principal analyst for Techstrong Research, an arm of Techstrong Group that is also the parent company of devopsy.kinsta.cloud.

    Despite the disjointed Micro Focus portfolio, OpenText has no intention of selling off any of those offerings, said Barrenechea. Going forward, the strategy is to not only accelerate the transition to the cloud but also expand application license renewal services, he added. OpenText also sees an opportunity to expand the appeal of the Micro Focus portfolio into the small-to-medium business (SMB) market, noted Barrenechea.

    All told, it will take six quarters once the deal is closed to align the Micro Focus business model with the one OpenText currently employs, which Barrenechea said should result in growth starting eight quarters after the deal closes.

    There’s no doubt that, once the deal closes, IT organizations that rely on Micro Focus will be closely watching the level of ongoing investment in the portfolio for any signs of one platform being deemphasized in favor of another. It’s also probable that many customers will be relieved to see the stewardship of those platforms assumed by a company that is on firmer financial footing.

    Regardless of how the OpenText portfolio evolves in 2023, however, one thing that is certain is IT organizations have no shortage of alternatives to consider.

  • Time-Series Database Basics

    Time-Series Database Basics

    There’s been a lot of buzz about time-series data for the past few years. No matter what you’re monitoring—financial data, server performance, social media activity or something else entirely—time-series data can give you valuable insights into trends and patterns.

    But what exactly is a time-series database, and why do you need one? This article will look closely at time-series databases and how they can help you make sense of your data.

    What is a Time-Series Database?

    A time-series database (TSDB) is a database management system optimized for storing and querying data that changes over time. Time-series data is often used in monitoring applications where it’s important to quickly retrieve information about a system’s current state and trends and patterns over time.

    In most definitions, time-series data, often referred to as time-stamped data, is a sequence of values indexed in time order. Time stamping refers to data collected at various times, where each value is time stamped. These data points are usually gathered from the same source and are used to measure progress over time.

    Important Notes:

    You can use relational or NoSQL databases to crunch time-series data, but purpose-built time-series databases are tailored to exploit the unique features of time-series data. This implies that time-series databases ingest at a faster rate, query more quickly and compress data more efficiently. Furthermore, time-series databases include special analytical capabilities and management features that are not found in most relational or NoSQL databases.

    Some Advantages

    This list is not exhaustive, but here are some of the advantages that you might get from using a time-series database:

    1.) Time-series databases can handle high-velocity data very well.

    2.) Time-series databases are purpose-built for storing time-series data, making them more efficient in storage and querying.

    3.) Time-series databases often have built-in analytics and management features designed specifically for time-series data.

    4.) Many time-series databases are open source, which means they’re free to use.

    Some Disadvantages

    There are also some disadvantages that you should be aware of before using a time-series database:

    1.) Time-series databases are often more complex to set up and manage than relational or NoSQL databases.

    2.) There are relatively few time-series databases to choose from, so you might not have as much flexibility in choosing a platform that meets your specific needs.

    3.) Time-series data can be very large, so you’ll need enough storage capacity to accommodate your data.

    What Are the Characteristics of Time-Series Data?

    There are several characteristics of time-series data that make it unique and require special handling. From a database standpoint, the most essential ones are as follows:

    1. Timestamp: Every data point has its timestamp. The timestamp is crucial for calculating or analyzing the information.
    2. Structure: Metrics from devices or monitoring are almost always structured, unlike those produced by Internet applications. They have predetermined data types and lengths, and the structure will not alter until the device’s firmware is updated.
    3. Stream-like: Data sources, such as audio or video programs generate data at a set or constant rate. These data streams are completely unrelated to one another.
    4. Stable flow: Time-series data traffic is constant over time, and it may be calculated and predicted if enough data sources are used within a certain sampling period.
    5. Immutability: The source of this data is a time-series data store. Each data point is created only once and never updated or corrected. Time-series data, like log data, is typically append-only.

    How Is Time-Series Data Used?

    Many of you may be wondering why time-series data is so important. The answer is that it allows us to track changes over time, which can be incredibly valuable for monitoring and troubleshooting purposes.

    For example, let’s say you’re a system administrator tasked with keeping an eye on server performance. By monitoring various metrics—such as CPU usage, memory usage and network traffic—you can quickly identify when a server is starting to experience issues.

    Time-series data can also be used for predictive purposes. For example, if you notice that CPU usage spikes at certain times of day, you can use that information to plan for future capacity needs.

    Of course, time-series data isn’t just limited to server performance. It can be used for any application where it’s important to track changes over time. This includes everything from weather and financial data to social media and website analytics.

    Let’s Get Specific

    Here’s what you should know about time-series data: It’s typically used to seek insights into operations, create alerts based on real-time analysis and forecast future trends.

    The following characteristics are found in time-series data applications:

    1. High write-read ratio: Twitter and LinkedIn are internet applications with single articles that millions of people read, but raw time-series data is primarily scanned and evaluated by apps and algorithms.
    2. Retention policy: In general, time-series data is not kept permanently. Organizations have a retention policy that specifies when and how their data is destroyed.
    3. Real-time analytics and computing: Time-series data must be calculated in real-time to identify out-of-the-ordinary activity and sound alarms based on the acquired information or aggregate findings.
    4. Query scope: Time-series data is generally requested over a period or a set of data sources, and filters are used to prevent all historical information from being requested. Furthermore, all or a portion of the data sources with a filter condition are always aggregated.
    5. Trends: Single data points are typically not significant in time-series data. The emphasis is on how data evolves, such as fluctuations in the previous hour or day.

    Popular time-series solutions like TDengine or InfluxDB, for example, create more efficient processing of time-series data and greater performance than general databases by taking advantage of these qualities.

    Do Time-Series Databases Need Specialized Databases?

    We live in a modern world where data is constantly being generated at an unprecedented rate. To keep up with the demand, businesses need to be able to store and process large amounts of data quickly and efficiently. This is where specialized time-series databases come in.

    Everything is online, including meters, automobiles, lifts, assembly lines and even bicycles. And these items are sending out a never-ending stream of numbers and events that IoT and the cloud have unleashed an explosion in time-series data generation.

    With this in mind, it’s not surprising that several specialized time-series databases have been created to deal with this type of data influx.

    Time-series data sets are enormous and pose a significant problem for general database management systems, such as relational and NoSQL databases. Non-specialized databases have trouble with the following elements of time-series data:

    1. Data ingestion rate: In many time-series data scenarios, millions of data points are generated every second and must be ingested in real-time. Relational databases are not built to handle this quantity of data, and while NoSQL databases may be scaled to do so, the amount of resources required soon becomes prohibitive.
    2. Query latency: It is often the case that a time-series system must scan a massive number of data points to get an aggregate result, which might lead to sluggish performance. For example, it would take days for a basic database to compute the average response time of all Amazon.com clicks, by which time the overall conclusion may be incorrect.
    3. Storage cost: Data generated by internet-connected devices and apps is nonstop 24/7, with a single day producing as much as a terabyte of data. Because relational and NoSQL databases cannot compress this data effectively, storage costs can quickly skyrocket.

    These problems are generally linked to efficiency in processing big data sets, although there are some places where general databases frequently do not meet even the fundamental demands of time-series applications:

    1. Data life cycle management: Time-series data is generally removed in bulk, not one data point at a time, as it ages out.
    2. Roll-up: In most cases, time-series data is collected and rolled up over a set period before being stored in the new table. Raw data and rolled-up data can have distinct life cycles and retention policies.
    3. Special analytic functions: Time-series applications need more specialized features than general databases, such as time-weighted average, moving average, cumulative sum, rate of change, elapsed time for a specific state and the delta between two consecutive data points.
    4. Interpolation: The database management system must be able to interpolate data based on the adjacent data points and rules to regularize data sets when applications or algorithms require.
    5. Continuous query: Therefore, if you hear a user or customer say they want to know when their data is updated, it’s reasonable to assume they want this functionality.
    6. Session and state windows: Aggregations and analytical procedures may be performed on a session or state window, not just time—for example, consider one that only calculates average power consumption when a machine is switched on.

    With general databases, developers must write custom code to implement features specific to their data set. Different data workloads require different database solutions; one size does not fit all. For time-series data, no matter the size of your data set, a purpose-built time-series database is the best tool for the job.

    Why Are Time-Series Databases Becoming Popular?

    Two years ago, time-series databases were the most popular database type in the business, owing to its expanding number of use cases. It is especially useful if you’re executing sophisticated transactions like advertising, e-commerce, supply chain management and so on.

    However, because of the rapid expansion of the IoT, they are becoming even more popular. As more devices become internet-connected and send data—time-series, of course—to the cloud, many industries are interested in purpose-built time-series databases.

    Time-series data is proving to be an important tool, not just for decision-making and optimization but also for industrial applications. Finally, IT infrastructure has been growing rapidly, with everything from servers, containers, network devices, apps, and microservices being monitored to create massive quantities of time-series data.

    Older time-series databases, on the other hand, are often closed systems employing antiquated structures that do not scale to handle the increasing amount of data. A million time-series data points used to seem like a lot, but now millions and even billions of data points are commonplace.

    Finally, integrating old (previous) time-series solutions with popular data analysis tools like artificial intelligence and machine learning platforms is difficult, if not impossible. These legacy systems cannot be transferred to the cloud without a lot of work and their licensing terms are no longer adequate for current apps.

    Final Thoughts

    The expanding market and the constraints of previous time-series databases are making room for a new generation. These new time-series databases are built from the ground up to take advantage of the cloud, big data and AI/ML technologies.

    If your business or application deals with time-series data, you need a purpose-built time-series database. The advantages of using such a database are too great to ignore, and the disadvantages of using an older, less capable database are becoming more and more apparent.

  • Overcoming Multi-Cloud Management Challenges

    Overcoming Multi-Cloud Management Challenges

    We’ve long talked about how the cloud is the future. But with the vast majority of corporate data now stored in the cloud, it’s clear that most businesses now realize this is a must-have in today’s increasingly demanding and agile business environment. 

    The cloud has been the backbone of the remote restructuring of the workplace over the last few years, enabling organizations to access data flows and critical workloads no matter where their employees are stationed. So much so that there’s been a dramatic rise in the number of enterprises deploying different service providers as part of a multi-cloud strategy. This enabled them to pursue greater agility and flexibility within a hybrid environment. 

    Recent research from Wanclouds found that most IT professionals use two or more cloud platforms today—and just under half (48%) said they used a mix of public and private clouds. However, the reality is that the more cloud environments your organization use, the more complex it becomes to both manage and secure them. Here are three multi-cloud management challenges impacting IT teams today and how they can overcome them.

    Maintaining Uptime in the Face of Rising Security Challenges

    Avoiding downtime has never been more important, nor has it been more challenging to achieve. Extreme weather events are becoming more frequent and increasing the risk of natural disasters or power outages, leading to data loss. In fact, 65% of U.S. and UK IT decision-makers said their organization experienced at least one data loss incident last year, while 21% admit they experienced two data loss incidents.

    But it is, perhaps, the security challenges that pose the biggest threat to IT departments and the developer teams they support. Since the onset of COVID-19, organizations have been targeted for cyberattacks far more often and have been infiltrated by cybercriminals much more quickly. According to McAfee, there were 3.1 million cloud-based cyberattacks in 2020 alone. Unsurprisingly, these attacks are fueling concern in IT circles. 88% of U.S. and UK decision-makers said they’re at least slightly worried—and 15% are apprehensive—about cloud workload security in 2022. 

    While multi-cloud environments enable developer flexibility, they also cause inherent configuration and visibility problems, and this is where cyberattackers can take advantage of weaknesses. The complexity of migrating data across different cloud platforms and operating systems can also lead to delays with patching. The key is often ensuring visibility across these platforms and maintaining interoperability. Security platforms for multi-cloud environments must be set up to share threat information across providers and platforms. If not, threats can easily be missed, or even if they are detected, the response across infrastructure will be too slow to minimize damage. 

    With businesses accelerating their goal of becoming 100% cloud-native in the next few years, a cloud-based disaster recovery strategy must also be embedded within their management plans. Without one, every business risks losing vital data and having their systems, operations or services shut down by natural and man-made disasters, security risks, hardware failures and power outages.

    Staying on Top of Hidden Charges

    Cloud spending has skyrocketed in the past two years, and all signs suggest that organizations will continue to allocate more of their IT budgets to the cloud over the years to come. But not every business has the same resources as, say, Apple, which reportedly spent over $30 million a month on Amazon’s cloud, according to a 2019 CNBC story. The reality for most IT teams is that they’re constantly battling hidden charges and unexpected cloud costs that threaten to blow up their entire IT budget. 

    In fact, in Flexera’s 2022 State of the Cloud report, respondents self-estimated that their organizations wasted just under a third (32%) of their cloud spend, up from 30% last year. Meanwhile, Wanclouds research found that 54% of British and American IT professionals say they have been hit with unexpected cloud costs in the past year.

    It’s a severe problem throwing IT teams for a loop from the migration stages of the multi-cloud plans right through to management and maintenance. Sure, cloud providers like Amazon provide basic tools for resource tracking and spend visibility. But truthfully, these features generally fall short of providing real-time and enterprise-wide insights on how to optimize costs.

    To eliminate the sticker shock of unanticipated cloud costs, organizations need to have a defined plan for cloud cost management. Again, company-wide visibility into their cloud platforms’ billing systems is imperative and a way to track spending across their multiple accounts. It’s vital for companies to properly tag the infrastructure for understanding costs and maintain up-to-date inventory of all the resources.

    Meeting Multi-Cloud Compliance Requirements

    Every cloud user must comply with specific regulatory standards following industry guidelines and local, national and international laws. Organizations in the health care space, for instance, must comply with HIPAA rules, which are essentially a set of national standards to protect sensitive patient health information from being disclosed without the patient’s consent or knowledge. Meanwhile, manufacturing IT teams must abide by stringent intellectual property security protocols and protocols for other sensitive data like PCI and PII. 

    Ensuring compliance across multiple cloud environments is necessary, but undoubtedly a challenge. This is especially true in today’s attack-filled security landscape, where internal threats can pose as much risk to data security as external attackers. (32% of the organizations surveyed by Varonis found incorrect access authorizations and privileges assigned to users). Not to mention compliance requirements are constantly changing, making it even more difficult for organizations to meet demands.  

    This is why every IT and security professional invested in the cloud needs a solution that lets them set up compliance policies against their cloud infrastructure, making it easy for them to ensure compliance requirements in the case of any unforeseen natural disasters or security incidents. We’ve seen the damage that just one data breach can have on a business—costing them millions of dollars in cleanup and leaving them with considerable damage to their reputation. It’s imperative that an organization’s entire multi-cloud strategy prioritizes compliance and the security of sensitive data. 

  • Avoiding Security Review Delays

    Avoiding Security Review Delays

    In the summer of 2021, I had lunch with a senior security developer at one of Seattle’s leading tech firms. Even though we were relaxed in the sunny and cool afternoon of the Pacific Northwest, there was no doubt my friend was frustrated. Part of his job was running code through a popular scanning product and then evaluating the results. It was a tough role because he sat outside the team that was developing the code and he was often under pressure to run his analysis within the constraints of their deadlines. 

    All too often, he had to deliver the bad news that the scan had found security issues. But that alone wasn’t why he was frustrated. His issue was that the tool that he was using was taking several hours to complete scans. He’d taken to starting the scan at the end of the week and hoping that it would be complete by Monday morning. The very tool that was supposed to make his job easier was actually a headache.

    This is one of the fundamental challenges of analyzing software systems for security vulnerabilities. All too often, developers and stakeholders focus their project timelines on the functional specifications. Teams work feverishly to rapidly iterate from prototype to finished product. 

    Security concerns are never disregarded, but the culture of “Let’s see if we can get it working the way we want first and button it up later” ends up costing more in the long run because the evaluating and testing cycles take longer. Not to mention that if substantial security problems are found, it can be difficult to remediate them without breaking things that were supposed to already have been completed. This problem is becoming acute in the SaaS world where solutions are being built faster than ever before with little to no security oversight.

    The Right Tools and Software Are Critical

    First, managers and executives need to mandate that the entire software development lifecycle and every environment should have security. Particularly for web-based SaaS tools, any vulnerability that exists is a potential attack vector. Attackers know that going after a dev version might not yield a jackpot, but it might give them access that exposes credentials that work in later versions or identifies a vulnerability that flies under the radar into production.

    Second, automation and tooling are simply the name of the game. Using tools with code repositories, pipeline management and testing frameworks help ensure the elimination of as many potential vectors of error as possible. Security checks should be the same. We can’t always rely on the skills of a developer or an administrator to follow security best practices. 

    We need tools that provide reliable and rigorous checks. Moreover, these tools need to run frequently. SAST scanning should be fast enough that a developer can run it with every save, or at least every commit. Coupling SAST and IAST scanning together is the gold standard for secure development. Every time code is compiled into a dev or test environment, it should be tested to make sure that vulnerabilities haven’t crept in and that the permissions and architecture of the system don’t work at cross purposes to expose vulnerabilities.

    Finally, there is a fundamental issue that must be addressed in every manager’s office. Code quality is not the same as security. Sure, there might be some overlap, but it doesn’t make sense to have 200 rules for code quality and 50 for security. The two things are not equal, and it’s a fallacy to expect that having high-quality code without frequent security checks will ultimately lead to less time spent in security review. Don’t miss the forest for the trees. Make sure security is the team’s top priority.

    Advocating for Security in SaaS

    My friend didn’t have a problem holding teams accountable for the code they’d written. In fact, he relished being able to insist on high standards and was proud that his organization empowered him to do so. That’s exactly what we need in the software industry: Individuals who are unafraid to demand the best from their peers. But it’s imperative that we provide effective tools they can rely on to succeed.

    The problem of security testing delays are not new to software development. It has been well-documented that it can take 640x the money to fix bugs in production, but due to a lack of effective tools this problem has been amplified in SaaS development. A SaaS development environment presents unique challenges that cannot be met by just employing code analysis or configuration testing for SSPM vendors. It requires a holistic view of the environment and the ability to use customization to be effective in the identification and remediation of vulnerabilities. 

    With a comprehensive view, those responsible for security and those implementing it will see security as a necessity to innovate quickly rather than a necessary evil that is put off until last in hopes it can be avoided. Maybe then my friend won’t be as frustrated next time we meet.

  • 5 Ways to Prevent an Outage

    5 Ways to Prevent an Outage

    In today’s always-on, ever-connected world, we all expect 100% availability.

    What gets in the way of this? The devil is in the details. Over time, everything breaks:  Disks, nodes, containers, networks, DNS servers and configuration issues can all lead to major outages.

    Amazon’s 13 minutes of downtime in August 2021 translated to almost $5 million in lost revenue and Amazon Web Services’ three outages in December of last year cost millions more. And it was widely reported that the October 2021 Facebook (including related products: Messenger, Instagram and WhatsApp) outage resulted in an estimated $100 million in lost revenue due to a network configuration error.

    There have been many outages of varying scope and duration—some attributed to technology breaking, others traced back to people making mistakes. In fact, a 2017 Amazon outage was attributed to “human error”–or more specifically, one employee’s typo. The employee intended to take a small number of servers offline to fix a billing system issue but the incorrect command led to several servers being affected and a major outage.

    So, what can your company do to reduce the risk, duration and impact of a potential outage? For one, companies can’t pretend that if they only try harder, they can prevent humans from making mistakes. Do you have dozens of people manually keying in hundreds of commands every day? If so, a mistake is inevitable. Companies should instead investigate how and why one small blunder in a command line could do widespread damage. Guardrails and redundancies should be in place to protect against and minimize these types of incidents.  

    As the enterprise continues its digital transformation toward a multi-cloud, hybrid and distributed world, here are five guidelines to consider—guidelines that can help you preempt and prevent your next big outage.

    1.  Always Consider Your Blast Radius

    Blast radius measures how one error, typo or malfunction can impact other parts of a system. 

    What is the chain of components or services that is impacted by one error? 

    What are the hidden dependencies? 

    How many customers would be affected and for how long?

    More is connected than you think; discover how it’s all related in advance to eliminate surprises.

    If too many systems in a network are mutually dependent and interconnected, the result is deadlock. In other words, when one breaks, the other also fails. We have seen this in past Facebook outages. A single erroneous command sparked a domino effect that shut down the backbone connecting all of Facebook’s data centers globally.

    How can you prevent this scenario? Plan for the worst possible scenario. When it comes to mission-critical systems, take an intentional approach to situation failover and backup. What if A, B and C all happen simultaneously? Also, think about your cost versus reliability trade-offs. Maybe you can’t afford two full sets of servers but can afford a full set of archived code. 

    2.   Smooth Out Rough Edges

    Consider that core infrastructure and code get used (and tested) a lot more than edge code/infrastructure. Basically, the core is hardened while edge infrastructure is more vulnerable. You should be more proactive at the edge around identifying vulnerabilities and doing explicit testing to counter this risk.

    The 2014 Google disk erase/satellite outage is a cautionary tale involving the edge. In Google’s case, the edge networks offered a key advantage: A better connection for the customer due to lower latency since data is accessed from a source closer to the user as opposed to the core network. Google had a CDN/edge network for TCP connection termination, caching for various apps, DNS regulation, etc. Teams needed to routinely upgrade a satellite rack and run decommission automation—and this automation failed part of the way through. Error message. They retried the decommission process again to debug the failure. The manual re-execution of the decommission automation meant thousands of satellite machines were decommissioned and, within minutes, the disks of all satellite machines globally were erased. The result: Customers experienced an increase in latency. Behind the scenes, it required two days of all-hands-on-deck to reinstall these machines. 

    3.   Think Escalator, Not Elevator

    A good system has both defense and depth. Considering that it takes three or four things to (really) go wrong, think about failure cascades. Bluntly put, limit dependencies. Measure the longest dependency failure path and mean-time-to-recovery (MTTR). When you do this, think escalator not elevator. What do I mean? Well, when an elevator fails, you are left with a shaft; an escalator that fails still works as a staircase.

    In 2016, severe weather in Sydney, Australia resulted in an AWS EC2/EBS outage. First, the utility provider suffered a loss of power at a regional substation. This failure resulted in a total loss of utility power to multiple AWS facilities. In one of the facilities, AWS power redundancy didn’t work as designed and power was lost to a significant number of instances in that availability zone. To make matters worse, the instances that lost power lost access to both their primary and secondary power as several power delivery lineups failed to transfer load to their generators. Also, a latent bug in AWS instance management software resulted in a slower-than-expected recovery of the remaining instances.

    The lesson? This correlated power delivery lineup failure illustrated that failures are not independent. You must design for the largest correlated failure in combination with the background noise of ongoing independent failure. Recovery code needs to be scale-tested against the largest expected failure. 

    4. Stay Ahead of Digital Certificate Rotations

    Don’t underestimate the importance of digital certificates—and their expiration dates. A few years ago, LinkedIn allowed one of its SSL certificates to expire and it knocked out LinkedIn sites in the U.S., UK and Canada. In 2017, Equifax missed a breach for 76 days because of an expired certificate. The average annual cost of unplanned outages due to certificate expirations is $11.1 million.

    For example, you could have a load balancer sitting in front of several servers. The load balancer could have an up-to-date certificate while the servers behind it are expiring. It is important to check for expiring servers in multiple ways. Ideally, you check the boxes in two ways: Ping the HTTP endpoints and also have scripts that check the certificate files themselves.

    Expired certificates (TLS for example) can lead to encryption and authentication risks as well as lost internet connectivity and larger outages involving websites and applications. To avoid this, you continuously check, renew and deploy certificates—and this can be achieved via automation. ​​Think about automatically rotating expiring certificates.  

    5.   Tackle Toil to Empower Your Teams

    What’s the biggest headache for site reliability engineers (SREs) and those who manage production operations? Toil. 

    We can define toil as manual, tedious tasks that engineers perform within production environments. Basically, toil wastes time and slows down operations. As a result, most SREs and engineering teams strive to minimize toil within their workflows.

    In production operations, you should work to limit toil. Make sure your team is regularly investing time in proactive work; the reactive work shouldn’t take most of their time. Toil reduction gives developers, DevOps engineers and SREs more time to focus on projects that create net-new value for the business (new application features). Google stated that it has an “advertised goal” of keeping toil “below 50% of each SRE’s time.” Basically, no engineer should spend more than half of their time on manual, repetitive work. But the ultimate goal should be to get toil as close to zero as possible. Automating incident remediation can help SREs spend less time on toil and less time on-call.

    Also, when you implement fixes—especially people and process fixes—ask yourself if this is a systemic fix or one that is relying on best intentions. Look for tools to safely empower your team (with guardrails). As you work to identify major vulnerabilities and dependencies (as discussed above), you should automate away the most common issues while planning for the “corner cases” (or the 1% of 1%) that may lead to major outages (a.k.a. the worst-case scenarios).

    As you consider these guidelines, make sure you focus on putting actual mechanisms in place—not one-off, “hopeful” directives. Do not rely on people to remember best practices from a training session—these are not mechanisms. Instead, design processes that can be systematically implemented. When you do a post-mortem, look for concrete programs you can put into place that will prevent the same mistake from happening again. Automation can help you achieve this level of confidence, management and quality control—and it may just keep you one step ahead of a potential future outage.