devopsy.kinsta.cloud

Tag: DataOps

  • 5 GraphQL Trends to Watch in 2023

    5 GraphQL Trends to Watch in 2023

    The GraphQL query language had a big year in 2022. We witnessed its increase in production use cases, solving overfetching and underfetching concerns that plagued traditional API integrations. GraphQL can significantly improve usability, aggregate multiple services and optimize how data is ported from system to system.

    Yet, GraphQL is also prone to new security concerns. And as attacks on APIs continue to increase and grow in sophistication, an increased security focus will undoubtedly be required to avoid vulnerabilities. The coming year will likely also see architects grappling with their GraphQL maturity as they consider how best to scale its use across an organization and its various stakeholders.

    Below, we’ll consider where the tides are flowing for GraphQL in 2023. These forecasts are informed by my coverage in 2022 and reports from within the web development community.

    1. Increased Production Use

    A full 62.6% of developers reported relying on APIs more in 2022 than they did in 2021, according to RapidAPI’s State of APIs report. As the number of APIs in use expands, we will undoubtedly see more GraphQL examples. GraphQL is popular due to its usability and flexibility which outweighs traditional integration points. Studies showed it is primarily used for exposing internal data, although partner and public use cases are not uncommon.

    Representational state transfer (REST) remains a dominant style for modern APIs. According to Postman’s 2022 State of the API report, 89% of developers use REST. GraphQL is in fourth place at 38%, superseded by webhooks (35%) and SOAP (34%). Although usage remains small in comparison, it’s steadily gaining traction and will continue to expand.

    2. Unifying Disparate APIs

    GraphQL helps expose fields in an easily consumable manner. But it’s also good for aggregating disparate microservices into a unified schema. As such, it is often used as a meta layer that combines multiple REST services, databases and even GraphQL schemas. For the time being, it appears that REST has staying power. And it’s unnecessary to migrate all REST services to GraphQL. Looking to the future, it is more likely to be adopted as a layer on top of existing REST APIs.

    Combining multiple backend services under the hood and exposing them in a unified schema creates a more usable, localized interface that frontend developers can work from. In the future, GraphQL might become a meta layer for more organizations, increasing the discovery of internal services and helping teams reuse internal microservices.

    3. Supergraphs and Subgraphs

    As mentioned above, GraphQL is used not only to combine legacy REST services but also to combine multiple GraphQL schemas. Some call this a graph of graphs—or a supergraph. A supergraph that combines multiple GraphQL schemas would offer unparalleled developer experience in that engineers can access many services from the same endpoint.

    However, there’s a downside to centralizing with a unified graph. Different internal teams or partners might only need access to a particular set of functions. Giving the entire schema to every person who walks through the door would be TMI, not to mention a potential security risk that breaks the rule of least privilege. So, the idea of a subgraph has gained traction. A subgraph is an Apollo concept, but the theory could certainly be applied in other implementations. In the future, we’ll likely witness more use cases that use subgraphs to help federate a supergraph across an organization, helping segment a graph to specific stakeholders’ needs.

    4. Turning on GraphQL Security Controls

    As of 2022, less than half (43.2%) of GraphQL implementations have disabled query introspection. Disabling field suggestions and introspection are means to reduce the risks of public GraphQL exposures. As GraphQL security best practices become more apparent, these numbers will likely become more commonplace. Other security controls, such as implementing query timeouts, query rate limiting and query cost analysis, can also help curb potential malicious behaviors.

    As we’ve covered before, security by obscurity is not enough to protect GraphQL. Yet, many native GraphQL security and performance directives go unused. Besides turning on these advanced features, new tooling is coming to market that could harden GraphQL. Security vendors continue to add support for GraphQL, making its protection more accessible for non-security engineers.

    5. Managing GraphQL at Scale

    GraphQL has arisen in some large production use cases in recent years, helping developer usability at GitHub, boosting productivity at Netflix and aiding Shopify’s API strategy. These case studies proved its usefulness in large enterprises. But how do we effectively scale GraphQL across a massive organization?

    The concepts of supergraphs and subgraphs are ways to help scale it across a company. But retaining high performance will also be vital to maintaining an effective, scalable footprint. As such, we will likely continue to see companies optimize how GraphQL is used as it matures. This includes optimizing queries and query depth and applying filtering and rate limiting to fit application demands. Taking measurements of GraphQL response times will be necessary to establish benchmarks to improve over time.

    We’re Just Querying the Surface

    Above, we’ve only scratched—or queried—the surface of the exciting developments on the horizon for GraphQL in the coming months. There is much more to be said regarding updates to the query language and improvements within the surrounding tooling landscape. Where do you think GraphQL will go in 2023? Let us know in the comments below!

  • How Embedded Analytics Can Transform Future Businesses

    How Embedded Analytics Can Transform Future Businesses

    It’s no secret that big tech companies like Google, Amazon and Netflix are not only leveraging data but also delivering embedded analytics and insights as a standard functionality of their software, services, applications and toolkits. The analytics appear to consumers as recommendations, usage habits and other helpful information. Such insights not only help consumers extract the most value out of the product they are using but also help providers understand how they can provide better services. Businesses that embed analytics to provide a differentiated user experience also have a direct impact on engagement and revenue.

    Have you considered how embedded analytics could differentiate your company’s own software and hardware products? Let’s understand how embedded analytics can help transform technology businesses:

    1. Analytics are expected and welcomed in your products

    By embedding analytics into existing enterprise software and applications, users can drill down into real-time and context-aware data (for example: How busy a location will be at various times of the day) and can benefit from faster data-driven decision-making without worrying about application compatibility. Commonly, software applications provide in-app analytics to delight and serve their customers, while hardware companies can provide helpful stats on a web portal or app.

    2. Embedded analytics is easier than ever today

    Cloud technology has completely disrupted tech-enabled industries. It’s easier than ever for software, apps and hardware to connect to a data repository to save logs that are a rich source of analytics. The invention of the S3 bucket and object store has made it fairly cheap to keep log data. Finally, the transition to platform-as-a-service (PaaS) and software-as-a-service (SaaS) has fundamentally added an “easy button” to tracking consumption patterns and resource utilization.

    3. Embedded analytics helps design better products and services

    Embedded analytics can give businesses more insight into application telemetry (how customers use the data, how they engage with the product) and provide insights into the challenges they face from a usability standpoint. Providers can use advanced analytics to quickly iterate, draw patterns and make predictions, troubleshoot customer challenges, discover new avenues to monetize, detect and mitigate security threats or create entirely new features and applications that put customers on a faster path to improved application experience and better decision-making.

    4. Embedded analytics increases operational efficiencies

    The tech industry is transitioning from traditional monolithic architectures where applications are self-contained to microservices architectures where different components are spun up as needed via API calls. By embedding analytics directly into developer tools, businesses can manage resources without sacrificing availability or performance. Companies can allocate and optimize resources dynamically and understand peaks and valleys in consumption. Businesses must be able to leverage historical and real-time data to build predictive models to proactively identify and mitigate issues with individual components of the application before the customer experiences a problem.

    Embedding Analytics in Your Application

    Before you start embedding analytics in your application, it’s important to choose the right database foundation, as this will support current needs and future goals. Below are the top capabilities that should be standard features in analytics databases:

    1. Handles massive data volumes: The database must scale easily and efficiently with minimal cost using massive parallel processing. Today’s scale might be terabytes, tomorrow it might be petabytes and so on.

    2. Enables speedy data analytics: Speed is a key factor in a positive user experience. Users want instant insights because they don’t like waiting for results. The database should be capable of optimizing load and query performance so that analytics is both timely and relevant.

    3. Embeds machine learning: Machine learning algorithms are critical for predictive analytics. A platform that can embed ML and iteratively learn from new data can make a big difference and add a whole new level of intelligence.

    4. Features native integrations: The more time developers spend figuring out tools and integrations, the less time is spent innovating. The platform must ideally have out-of-the-box integrations with popular applications, data sources (on-premises, cloud, multi-cloud or hybrid), ETL, and visualization tools (e.g., Microstrategy, Looker, Power BI, Qlik, Tableau). It should be flexible enough to support user-defined functions for things that don’t have native integration.

    5. Encourages self-service: The database should make it easy for data scientists to leverage tools and programming languages they already know such as R, Java, C/C++, Python and SQL. Data scientists will also appreciate a platform that has built-in functions for a wide range of analytics (such as geospatial, time-series, event-series, real-time, machine learning, text analytics, pattern matching and regression) and delivers the tools to spot trends, uncover anomalies and anticipate the unexpected.

    6. Supports highly advanced analytics: Many platforms can analyze single tables or do a simple look-up but few have the power to analyze dozens of tables, hundreds of data types, dimensions and attributes from numerous sources over several years of history.

    7. Unified: The database should be able to streamline everything right from data ingestion to exploration to production until delivery. Users should be able to analyze data in its place to always have a single source of truth.

    It’s not hard to visualize that the most disruptive companies of the future will be the ones that harness data as a core asset and embed analytics in their products and services. Embedding analytics requires the right database foundation; to get to the right foundation, IT teams need to cut through the noise and study the fine print. With the proper database, the future is yours to create.

  • What the Convergence of Observability and Security Means for Devs

    What the Convergence of Observability and Security Means for Devs

    There is a scene in the movie Apollo 13 when the mission control flight director asks why the carbon dioxide scrubber in the command module was a different shape than the one used in the lunar module (and therefore incompatible). The engineer simply replied, “This just isn’t a contingency we’ve even remotely looked at.”

    A similar scenario plays out at many organizations when they examine security and development practices. When it’s time for code to go into production, every developer hopes their code will run as planned–smoothly and error-free. Unfortunately, something will be missed, performance could degrade or new, unanticipated problems can emerge after an application is deployed to production.

    Developers fear the “unknown unknowns.” And it keeps them up at night.

    When an “unknown unknown” problem occurs, it takes time for teams to do the forensics to identify the problem. It can take days or even weeks for teams to complete that initial investigation. If a mission-critical application is offline for a significant period of time due to a forensics investigation, then it can negatively affect your customer’s experience and, ultimately, your organization’s bottom line.

    The “unknown unknowns” problem is another challenge facing developers in today’s unpredictable macro environment. That includes the deluge of data, increasing cyberthreats, an increasingly distributed infrastructure and a remote and hybrid workforce as well as a mix of modern cloud-native and monolithic apps.

    To help developers build for the future and address security concerns, it is imperative they rethink the way they are collecting, operationalizing and storing different types of data from a variety of sources.

    Security and Observability: Better Together

    Today, security and observability have recently begun to overlap, driven by the growing need for organizations to better understand the activity inside their environments. Many forward-thinking organizations are operationalizing the massive amounts of log and event data currently being generated to understand and assess issues in their infrastructure and apps and using these actionable insights to optimize workload, app resource availability, security and uptime.

    So how can developers take advantage of the security and observability convergence trend happening right now?

    Currently, it’s common for developers to have two agents as part of their tech stack. One agent for observability, which is collecting data such as logs and events to get relevant insights into the health and performance of their application in real-time. The other agent is for security to collect data in their devtest, integration and production environments.

    Since these two agents often collect the same data, there is an opportunity to consolidate it into a single location, such as a unified security platform with a single agent, where this data can be overlaid with time series analytics to deliver relevant insights to users. Using this approach of having both security and non-security data in one place will significantly speed up the process of getting these valuable insights into the developer’s hands.

    In addition to getting insights faster, another benefit of having this data in a single location is the ability to access historical data. Giving developers the ability to store and manage data for long periods of time allows them to get insights from this historical data, so they can better understand alerts, incidents and adversaries. This can help them correlate in terms of understanding the risk to an application, such as potential “unknown unknown” threats, which they can proactively address before it causes any damage.

    Adopting a converged security and observability approach is the most mature step that organizations can take to help developers deliver the best outcomes and best experiences for their customers and users. As more organizations rely on the cloud and the speed of business rapidly increases, developers must transform their view of the health, performance and security of applications and infrastructure. Giving developers this perspective will dramatically help them take rapid action against an adverse event before it impacts their company’s bottom line.

  • What Should Business Owners Know About SSDs?

    What Should Business Owners Know About SSDs?

    Without a doubt, SSDs have become the latest trend in hardware storage, and you can expect their use to increase over time. Implementing SSDs in your business can improve productivity and make it more efficient by increasing the speed of your PC long enough to extend its useful life. Here is all you need to know about using an SSD for your business.

    What is an SSD Storage Device?

    An SSD is a storage device that functions similarly to an HDD. But instead of having moving parts, an SSD stores data on interconnected flash memory microchips that continue to store information even when there is no power flow. These microchips are known as NAND and are more reliable and faster than the HDD.

    SSDs are fast because they use the Non-Volatile Memory Express (NVMe), which allows high data exchange speeds of up to 2,600 MB/s, almost five times faster than a standard SSD. The NVMe SSDs are more recent than SATA SSDs and use the peripheral component interconnect express (PCIe).

    SSDs Vs. HDDs

    SSDs and HDDs differ in several ways, and you should understand the fundamental differences between the two. HDDs are mechanical and use magnetic heads to read and write data while spinning the platters. The number of rotations per minute determines the speed of an HDD.

    Meanwhile, SSDs are electronic and use flash memory chips, which makes them faster than HDDs. They also use NOR or NAND flash memory as their storage medium. The NAND is smaller and faster than NOR in read/write operations and can store data when off.

    What is the Lifespan of an SSD?

    Cells wear out faster when they write more data, and a standard SSD cell has a lifespan of around 3,000 write cycles, which may seem small at first glance. But the SSD uniformly distributes the write operations across all cells, reducing the chances of the cells dying and increasing the drive’s longevity.

    SSDs also have backup cells that regenerate dead cells. For this reason, large SSDs have an extended lifespan.

    How SSDs can Help Your Business: Speed, Longevity, Energy

    SSD is ideal for businesses that handle and store vast amounts of data due to its high performance, speed, and reliability. Here are more reasons why you should use an SSD for your business:

    1. Faster Read/Write Speeds
    SSDs are faster at accessing your business data, reducing the startup time for programs and operating systems. Compared to a traditional HDD, an SSD runs up to 30 times faster, allowing you to boot your PC in seconds instead of minutes.
    2. Longevity
    An SSD may be suitable for upgrading drives in a portable PC. It has no spinning drive, which means better durability and a lower likelihood of damage or losing data when you drop your business laptop.
    3. Energy Saving
    When it comes to going green and reducing energy consumption, SSD storage devices should be at the top of your list. SSDs usually generate less heat and energy since they don’t have moving parts. Therefore, you will save substantial power when using your PC, and more battery life means more productivity.

    Challenges of an SSD Storage Device

    It has always been expensive to store enterprise-level data on servers, with SSDs being slightly more expensive. As business storage capacity increases, SSDs become more expensive, and an HDD may be suitable if you want to archive a significant amount of data. Apart from archiving data, SSDs are generally the best option for storing and transferring data.

    Bottom Line: Speed, Reliability, Improved System Cooling

    Modern businesses cannot function without storage devices, and an SSD offers a unique solution that significantly increases productivity by reducing loading times. An SSD promises faster system performance, increased reliability, and improved system cooling. Due to these advantages, your business will save both time and money.

  • How to Select the Right Volume Type/Storage for Your Workload

    How to Select the Right Volume Type/Storage for Your Workload

    AWS services are at the top of the cloud services food chain, and a hot topic for nearly every business considering migrating to the cloud. Along any cloud migration journey, many hidden traps await the unprepared. The volumes you’ve already provisioned might seem like a perfect match—but are you sure they suit your workload and carry the best price for performance delivered? Today, we’ll make the difficult task of choosing a storage class for your workload a bit less complicated.

    In the World of EBS

    AWS EBS currently provides three categories of considerable storage: Hard disk drives, general purpose SSD, provisioned IOPS, and one that you will most likely never need to use—magnetic. Although I’ll provide brief descriptions of each, I’ll focus on the most widely used volume types—general purpose.

    GP Storage – the Common Choice

    This is the staple storage for pretty much all AWS users. It gives you the best bang for your buck and is commonly used for tasks that don’t require heavy lifting from a drive perspective. GP storage ranges from 1 gibibyte (GiB) to 16 tebibytes (TiB), providing from 128 to 1,000 mebibytes per second (MiB/s) of throughput, which will most probably fit your use case, unless the workload is significantly specific.

    There are multiple types of GP storage, each with different pricing and characteristics that I’ll provide a few details on below.

    GP2 – Old-Gen SSDs for the Masses

    GP2 is the typical solution for boot volumes of EC2 instances, development environments, and lighter/less important databases. This is one of the most popular options out there, although its popularity comes from old habits rather than actually being a perfect shot. GP2 volume performance scales with its size—the bigger they are, the faster they go. Still, these might seem a good option, but to be honest, they aren’t so perfect. And the biggest elephant in the room is the term you’ll have to remember – “burst credits.”

    Burst Credits – Math Homework for Your Business Work

    Provisioning a GP2 volume, you get a one-time grant of 5.4 million burst credits, which you can use to make it handle input/output better during spikes in demand. More of those tokens means you’ll be able to sustain bigger performance demand for longer, boosting your storage up to 3,000 IOPS as long as you have enough of them in your pocket.
    While depleted when you go “turbo,” they accumulate when you’re not using your disks at full speed—saving up unused performance for when you need it most. Burn them all, and your IOPS performance drops to base value and throughput falls to the baseline IOPS multiplied by max I/O size.

    The performance of GP2 storage volume can be calculated with this formula:
    Throughput in MiB/s = ((Volume size in GiB) × (3 IOPS per GiB) × (I/O size in KiB))

    Until you reach the point where your volume is so big that its base performance exceeds burstable, you’d have to learn them by heart. Where is this point? Quite high, at about 1.1 TB (1024 GiB)—a lot of disk, even for a medium-sized business, especially since those volumes can’t be mounted to multiple targets.

    Figure 1: Graph of Max IOPS available for certain volume sizes (Source: AWS documentation)

    Unless you’re deploying a disk-devouring monster such as an EC2 host instance for dozens or hundreds of Docker containers, or you plan to run an enormous Elasticsearch node for the next few years with no clean-up routines in place, you’re not likely to need this much space. Also, in either of the two cases mentioned, you’d probably run out of CPU and RAM first; there are better alternatives for these cases as well, not to mention that the lack of maintenance is definitely not a good practice.

    In addition, for practically every use case—web serving, web applications, databases, casual EC2 instances for use with Docker, and more—pretty much anything is better than GP2.

    Luckily, in 2014, the introduction of GP3 brought some great improvements.

    GP3 – The “Newcomer”

    While not really a “new” player, GP3 volumes are a welcome improvement after experiencing the inner workings of GP2. They perform better and cost less in comparison, allowing you to easily scale your operations according to your needs much easier. Gone are the days of burst credit balances.

    The golden rule of this storage class is “mix n’ match.” Baseline performance here is the maximum burst performance of a GP2 volume, 3,000 IOPS, and 125 MiB/s. If your business needs more, you can purchase additional amounts of either, paying extra only for what you’re interested in.

    The limits are quite generous. You can purchase up to 500 IOPS per GiB (for at least 32 GiB volumes, capping at 16,000 IOPS), or up to 0.25 MiB/s per GiB (for volumes of at least 8 GiB, 4,000 IOPS or more, capping at 1,000 MiB/s).

    If you were previously happy with your GP2 volume, it’s almost guaranteed your company will be just as happy with GP3, and you’ll save a decent percentage of the storage cost, even up to ~20%.

    Assuming that you don’t need more than what GP2 provided you with, make the switch. To make it even easier to cut those costs down, AWS is now automatically setting the IOPS and throughput of your new GP3 volume to the exact same as your GP2. Less guessing, less messing around, more good stuff. Just click a few buttons and save some cash, no strings attached.

    This is the storage class I would recommend for every “generic” or “typical” workload. It’s a good starting point for almost every business use case that doesn’t expect huge amounts of disk operations, but is ready to scale up when the dreaded flood of demand for disk power comes.

    GP3 can easily handle the role of boot volume for EC2 instances. It can host both simple and a bit more complex applications, databases such as MySQL and PostgreSQL, or a blend of those in a containerized environment, especially in a multi-node configuration or as part of a Kubernetes cluster. Zesty Disk for Kubernetes handles this switch automatically if you have the CSI driver installed

    When the time comes, it can scale in tandem with demand. In certain cases, it might even be cheaper to provision high-speed storage with GP3 than the IO1/IO2 types, which are dedicated to speed and next on our list.

    IO Storage Class – Need for Speed

    The IO storage class was introduced to cater to the needs of businesses based on extreme durability, high read/write speeds, and the lowest latency available. It requires certain criteria to be met to show its true potential and costs significantly more. But, in turn, it provides unmatched speeds and reliability, as well as some features you won’t find in any other storage type.

    This volume type is great if you seek a volume to provision for your high-speed disk-based database, for example, or to create a single MySQL database host for multiple large-scale e-commerce instances, although there are multiple options available for this from RDS to multi-instance DBs (e.g., MySQL Galera). It’s also a fit if you need to accept a lot of data fast to process with an ML algorithm.

    IO1/IO2 Vs. IO2 Block Express – Which Type of “Fastest” do you Need?

    IO2 Block Express is pretty much IO2 on extremely potent steroids. You get limits that are four times as high for IOPS and throughput, latency of less than milliseconds, and surprisingly, no additional fee. Yes, you read that right, they’re priced the same. Plus, provisioning additional IOPS/throughput above the limit of a standard IO2 instance actually costs less.

    If you decide to use IO2 and meet the requirements (a certain type of EC2 instance), you should definitely consider Block Express—if it’s not already running by default.

    And when would you run IO2 anyway? Well, with its power and hefty price tag, this class is dedicated to the most disk-dependent, throughput-hungry, and IOPS-dependent monstrosities. For example, a data-crunching machine for a stock exchange, a global log-ingestion mainframe, or a multi-pod Kubernetes node prepared for very demanding workloads—the titans of storage, no doubt. It’s a go-to when you need a single machine that’s going to pack as much punch as you can get.

    How about IO1 then? It’s a compromise between low latency, high IOPS, and price. A good choice when you need a bit more than general purpose, but still not enough to go for the fastest storage out there. Keep in mind there’s also one additional drawback—less durability. A 0.2% failure rate annually does not sound like a lot, but if you’re thinking about dozens or hundreds of drives, absolutely critical workloads, or both, it definitely makes a huge difference.

    So, Which Should I Choose for my Workloads?

    As you can see, the world of EBS is quite complex, to say the least. So, I made a chart that should ease the selection process for your use case:

    Fantastic Disks and How to Tame Them

    Choosing the right storage class is a tough decision to make—but the benefits are worth it. With just one shot, you can both improve the performance of your instances and save on costs as soon as your next invoice.

  • Key Findings From the 2022 State of GraphQL Report

    Key Findings From the 2022 State of GraphQL Report

    GraphQL, the open source query language, has been steadily growing in popularity over the past few years. The language is excellent for frontend development, as it creates a usable interface to fetch precisely the data required for the specific client at hand. You can also use it to aggregate multiple backend services and data sources.

    Studies show that the ecosystem around GraphQL is maturing—it has a very global user base and a growing number of tools exist to aid in its adoption. The 2022 State of GraphQL Report found that 61.8% of developers are happy with the overall state of the ecosystem. Yet many native GraphQL features go unused by developers, which suggests there is a wide range of experience levels, from novice to advanced users, working with the technology. This indicates there is plenty of room for growth and knowledge sharing.

    Below, I’ll review the key takeaways from the 2022 State of GraphQL Report, considering overall usage data and focusing on the popularity of specific features. We’ll also highlight the state of standard tools used in conjunction with GraphQL to get the job done.

    Analyzing Overall Usage Statistics

    First off, there is a wide variance in the types of GraphQL APIs in production today, ranging from private to public endpoints. The report found 47.9% of developers use GraphQL for exposing an API intended for a personal website or app, 40.4% use it for private unexposed APIs for internal usage and 19.5% use GraphQL for publicly available APIs intended for third-party developer consumption. This demonstrates its flexibility to cater to multiple scenarios. And while the majority of clients that connect to GraphQL APIs include browsers (62.3%), it’s commonly consumed by other client types such as native mobile apps, other servers and desktop apps.

    Previously, we discussed using GraphQL as a composition layer to combine underlying APIs and services. And the report backs up this idea that developers commonly use it to aggregate various data sources: 55.4% of GraphQL APIs hit databases, while 35% consume REST APIs and 14.6% eat other GraphQL endpoints.

    In terms of benefits, one strong point is the ability to enforce and validate types for every object in the API. Users find other top benefits to be avoiding overfetching and aggregating requests. Yet, some hurdles still present themselves. These top drawbacks include error handling, performance issues, client-side caching and security woes.

    Regarding user base statistics, most users have been working with GraphQL for less than five years. If we look at the type of sector utilizing GraphQL, we see it primarily leveraged in the programming and technology sector. This is followed by e-commerce and retail (16.4%), finance (8.9%) and news, media and blogging (7.2%).

    Many Advanced Features Go Unused

    Many of the more advanced GraphQL features still go unnoticed by the majority of users. For example, live queries is a feature that helps obtain live data from a GraphQL server, yet only 25% who know about it have used it. Over half of respondents had never heard of @skip, a directive that skips a field based on a value passed to it. Similarly, other directives like @specifiedBy, @defer and @stream also share little-to-no awareness or use.

    Analyzing feature usage becomes a bit more serious if we consider the unadopted security and performance features. For example, only 43.2% of respondents have disabled query introspection. Disabling query introspection is a suggested security best practice to limit a hacker’s ability to perform surveillance. But as we’ve previously covered, security by obscurity isn’t enough to protect GraphQL, and an additional security emphasis is often needed.

    As most GraphQL APIs are intended for personal or private scenarios, you would think that more security features would be leveraged to limit the threat of introspection and denial-of-service attacks. Yet, the report discovered low usage and awareness of security and performance features like query timeouts, query rate limiting and query cost analysis, all of which could be turned on to curb malicious behaviors.

    Tools Used In Conjunction With GraphQL

    Developers often seek to combine, aggregate or merge GraphQL schemas from different APIs. To do so, Apollo Federation is the most common solution, used by 22.5% of respondents. This is followed by Schema Stitching (16.9%), GraphQL Modules (4.4%), and GraphQL Mesh (2.9%).

    Looking at other technologies used in connection, Next.js is the most popular framework and PostgreSQL is the most common database used. TypeScript and JavaScript comprise the most common languages used to write a GraphQL backend. Developers also use a balanced mix of IDEs to query and test endpoints, including GraphiQL, GraphQL Playground, Postman, Apollo Studio, and Insomnia.

    The report also found relatively high usage and retention for tools like the GraphQL Code Generator schema builder, Apollo Client and Apollo Server. The data shows a crowded solution market with Apollo as an industry-leading outlier—many other clients, servers, API generators and schema builders have medium-to-high retention rates yet experience far less use.

    Final Thoughts

    The 2022 State of GraphQL Report sheds some interesting light on the state of GraphQL, validating a lot of the interest and trends we’ve covered lately on the blog. Of course, there is more to be said about many areas, such as smoothing the transition to GraphQL.

    One key finding is the low number of security features being leveraged by developers. A greater emphasis on API security is required to avoid incidents and retain a zero-trust approach. This emphasis is part and parcel of responding to ongoing software supply chain threats and will require more awareness and thought leadership around GraphQL security best practices.

    The 2022 State of GraphQL survey ran from June 15, 2022, to July 15, 2022, and collected 3,094 responses. The above analysis only scratched the surface and there are many other insights. For more findings, you can read the full results here.

  • Zilliz Unfurls Managed Vector Database Service

    Zilliz Unfurls Managed Vector Database Service

    Zilliz this week made generally available a managed cloud service that can now be spun up in less than five minutes. The service is based on the company’s open source Milvus vector database.

    Frank Liu, director of operations for Zilliz, said the managed Zilliz Cloud service running on the Amazon Web Services (AWS) cloud will make vector databases much more accessible to a wider range of application developers.

    Vector databases are gaining traction because they provide artificial intelligence (AI) application developers with a more efficient method for managing the unstructured data need to train AI models. Currently, many of those teams are relying on labels and tags to classify and manage data. The Milvus vector database employs deep learning algorithms and its own AI model to create and store, index and manage vectors to more efficiently classify data, said Liu.

    Based on a cloud-native architecture that, for example, employs Kubernetes to scale consumption of cloud resources more efficiently, Zilliz Cloud can also scale to process tens of billions of high-dimensional vector data for data-intensive applications, he added. The company claims Zilliz Cloud delivers up to 10 times greater throughput and five times lower latency for read and write operations for data-intensive workloads.

    Other attributes of the managed Zilliz Cloud include support for multi-language software development kits (SDKs), a web-based graphical user interface to search and explore data, SOC 2 compliance, network isolation using IP whitelists and full data encryption in transit (TLS).

    In the future, Zilliz also plans to add support for Microsoft Azure Cloud and Google Cloud Platform in addition to graphical processor units (GPUs) to optimize the performance of its deep learning algorithms.

    Potential use cases for a vector database span everything from image retrieval and video analysis to fraud detection to drug discovery. Organizations that already use Milvus, which is being advanced under the auspices of the LF AI & Data Foundation, include eBay, Shopee and SmartNews.

    In general, the rise of data-intensive workloads is driving more organizations to embrace DataOps best practices to optimize the management of massive amounts of data. Organizations that build AI models are typically at the forefront of those efforts given the massive amounts of data typically required to train AI models. The challenge organizations have historically faced is not many have the internal expertise required to manage a vector database. The Zilliz Cloud eliminates the requirement by relying on the expertise of a team of database administrators trained by Zilliz to operate a managed service.

    It’s not clear at what rate organizations are embracing vector databases to build AI models. As the next era of applications are constructed, it’s only a matter of time before the machine learning operations (MLOps) practices used to build AI models converge with DevOps practices used to build the applications in which those models will be embedded. The challenge is the data science teams that build AI models tend to also have a distinct DataOps culture all their own; DevOps teams will need to find a way to embrace and extend that.

    In the meantime, the number of types of databases being used to build applications will only continue to expand so it will be up to each DevOps team to know when to manage each one themselves versus relying on a managed service.

  • Switching From FluentD to Vector Log Aggregation Tool

    Switching From FluentD to Vector Log Aggregation Tool

    Log files are extremely important to the data analysis process as they contain essential information about usage patterns, activities and operations within an operating system, application, server or device. This data is relevant to a number of use cases across an organization from resource management, application troubleshooting, regulation compliance and SIEM and business analytics and marketing insights. To manage logs created by these use cases and make use of this wealth of data, log aggregation tools enable organizations to systematically collect and standardize log files. However, choosing the right tool can be quite challenging.

    This blog will detail and compare the popular open source fluentD and Vector tools for log aggregation.

    fluentD Configuration and Efficiency Calculation

    When using orchestration tools like Kubernetes to deploy containers or other API resources, there is a need for a log aggregator to store the pod or node logs in a cloud platform. For a particular requirement, fluentD was used as a log aggregator tool to push K8s pod logs to cloud storage buckets with a sample configuration as shown below:

    <match kubernetes.**>

    @type <cloud platform name>

    project <project name in cloud platform>

    keyfile <credential json to access the cloud storage> bucket <cloud storage bucket name> object_key_format <name for the file to be used>

    path <file prefix/path where the file have to be stored>

    <buffer tag,time>

    @type file

    path /var/log/fluent/gcs timekey 1m timekey_wait 30 timekey_use_utc true flush_thread_count 16 flush_at_shutdown true flush_mode interval flush_interval 1 chunk_limit_size 10MB retry_max_interval 30 retry_wait 60

    </buffer> <format>

    @type json </format>

    </match>

    Using this system, fluentD was pushing only 47.62% of total logs to cloud storage. Since there was a loss of more than 50%, changes were made to the configuration. In most of the changes, the efficiency was somewhere between 40% and 50%, with a maximum efficiency achieved at an average 67% for an entire day. Below are some of the changes made along with the percentage of logs that were pushed to cloud storage:

    <buffer tag,time> @type file

    path /var/log/fluent/gcs timekey 1m timekey_wait 30 timekey_use_utc true flush_thread_count 16 flush_at_shutdown true retry_max_interval 60 retry_wait 30

    </buffer> Efficiency:- 46.32%

    <buffer tag,time> @type file

    path /var/log/fluent/gcs timekey 1m timekey_wait 30 timekey_use_utc true flush_thread_count 16 flush_at_shutdown true

    </buffer> Efficiency:- 49.89%

    <buffer tag,time>

    @type file

    path /var/log/fluent/gcs timekey 10m timekey_wait 0 timekey_use_utc true flush_at_shutdown true

    </buffer> Efficiency:- 37%

    <buffer tag,time> @type file

    path /var/log/fluent/gcs timekey 30 timekey_wait 0 timekey_use_utc true flush_thread_count 15 flush_at_shutdown true

    </buffer> Efficiency:- 60.88%

    <buffer tag,time> @type file

    path /var/log/fluent/gcs timekey 1 timekey_wait 0 timekey_use_utc true flush_thread_count 16 flush_at_shutdown true flush_mode immediate

    </buffer> Efficiency:- 66.77%

    Vector Deployment, Configuration and Resultant Efficiency

    To improve this further, the open source Vector tool by Datadog was also considered. This tool was suitable for K8s setup with a similar configuration as fluentD and was installed in the nodes.

    A Helm command was used to clone the official repository in its VMs; the configuration was changed as described below and installed it as an agent. Vector comes in two working modes: Agent and aggregator. While agent is the plain mode that pushes logs/events from source to destination, aggregator is used to transform and ship data collected by other agents (in this case, Vector).

    The installation of this tool requires a Helm repository in the local machine to fetch the source code. Hence the below commands were run in a sequential pattern before installing Vector in a K8s cluster:

    helm repo add vector https://helm.vector.dev (Adding vector repo to helm list)

    helm repo update (Updating the helm repos)

    helm fetch –untar vector/vector . (command to clone the repository to local machine)

    Configuration:-

    data_dir: /vector-data-dir

    sources:

    <Custom source id>:

    type: kubernetes_logs (because we are using kubernetes as our source)

    exclude_paths_glob_patterns: <Array of directories which has be excluded when collecting the logs from the nodes> (Optional)

    sinks:

    <Custom sink id>:

    type: <Destination cloud storage>

    inputs: <Array of source id’s from where log has to be pushed>

    bucket: <Bucket name of cloud storage>

    key_prefix: <Path inside the bucket where the logs has to collected> (Optional) encoding:

    codec: <Encoding of the log file> (Optional) Command to install the Vector:-

    helm install vector . –namespace vector

    After deploying Vector in the development environment and testing it, the efficiency was ~100% with negligible loss. The switch was then made to Vector and deployed in the production environment. Vector can ship up 100,000 events or logs/sec, which is a very high throughput rate compared to other tools for log aggregation performance. Vector was able to achieve 99.98-100% efficiency even in the Kubernetes production cluster.

    To learn more about how DataOps can enable highly performant data pipelines with real-time logging and monitoring, watch this video. 

  • VMware Backup Guide for Hyper-V Admins

    VMware Backup Guide for Hyper-V Admins

    At first glance, the Hyper-V and VMware virtual environments may seem similar, but upon closer inspection, a number of important differences between these two virtualization platforms become apparent. Initially, performing administration for VMware vSphere could present some challenges for administrators who are familiar with Hyper-V. There are several reasons a business might be switching from Hyper-V to VMware. The company could be deploying a multi-hypervisor environment – e.g., using Hyper-V for Windows-based virtual machines (VMs) while introducing VMware vSphere for VMs running Linux, Free BSD, Solaris, etc. This blog post is intended to help Hyper-V administrators understand the differences and similarities between these two major virtualization platforms, making the transition to VMware easier.

    Comparing Hypervisor Design

    The Hyper-V Role for Servers and Workstations

    Hyper-V users can set up the Hyper-V role not only on the Windows Server operating system (OS) but also on desktop versions of Windows, such as Windows 7, 8 or 10. The installation is similar for both server and desktop solutions. Hyper-V is classified as a type 1 hypervisor, meaning that it is run directly on hardware. There is also a lighter-weight alternative to installing the Hyper-V role on Windows Server: Microsoft Hyper-V Server. Hyper-V Server is a standalone product that is installed on bare metal.

    VMware Workstation for Workstations and VMware ESXi for Servers

    VMware provides VMware Workstation for installation on desktops with Windows and Linux-based operating systems. VMware Workstation is a type 2 hypervisor. A type 2 hypervisor is installed on the underlying OS. As you can see, both Hyper-V and VMware support installation on users’ PCs. If you want to install a type 1 hypervisor on bare metal, you should use VMware ESXi Server. VMs running on VMware Workstation can be migrated to ESXi Server with the VMware vCenter Converter.

    Management Tools

    vCenter as the Analog to SCVMM

    System Center Virtual Machine Manager (SCVMM) is a special universal product created by Microsoft for centralized virtualization management of Hyper-V hosts, clusters, virtual networks and storage. SCVMM is useful for the management of complex multi-site and multi-domain environments.

    Hyper-V Manager is a common tool that is available by default upon installation of the Hyper-V role. With Hyper-V Manager, you can connect to an available Hyper-V host. You can manage the host, the VMs residing on that host, the virtual devices of those VMs and VM replication. You can also create VM checkpoints, start or stop VMs and connect to VMs in order to interact with the user interface of a guest VM.

    Failover Cluster Manager is a tool for managing failover clusters and VMs within a cluster. With this tool, you can create a failover cluster. Failover Cluster Manager is not a substitute for Hyper-V Manager; you still need the latter to manage non-clustered VMs, Hyper-V hosts, virtual switches, etc.

    The equivalent of SCVMM in VMware vSphere is vCenter Server, which can perform the tasks that Failover Cluster Manager and Hyper-V Manager handle in Hyper-V environments. With vCenter Server, you can manage almost all components of vSphere – hosts, clusters, virtual networks (including standard vSwitches and distributed vSwitches), datastores, VM virtual devices, etc. vCenter is installed on a virtual machine, and the vCenter web interface can then be accessed through a browser. You can use vSphere Web Client (which comes in HTML5-based or Flash-based versions) to access vCenter via a web browser. To do so, simply enter the IP address of your vCenter server in the address bar of your browser.

    If you have a standalone ESXi host, you can similarly access that host using a browser with the vSphere client. Enter the IP address of your ESXi server in the address bar of your browser. Note that you cannot create data centers, configure distributed virtual switches or perform VM migrations when accessing a standalone ESXi host with the vSphere client.

    Below, you can see vCenter v.6.5 accessed via browser with vSphere Web Client. If you want to access the VM’s user interface, click Launch Web Console.

    A new tab is opened and you can interact with the guest OS.

     Another way to access the user interface of a VM is using VMware Workstation. For connection to a VM running on the ESXi host from VMware Workstation, click File > Connect to Server.

    Input the IP address of vCenter or ESXi Server, as well as the user name and password required to access that server. Click Connect.

    In the left pane of the VMware Workstation interface, you can see a list of VMs residing on the ESXi host to which you are connected. If you connected to vCenter, you would see the objects such as Datacenters, ESXi hosts and VMs that are managed by vCenter Server.

    PowerShell Vs. PowerCLI

    With the Hyper-V platform, the alternative to using a GUI for management of the virtual environment is using PowerShell. PowerShell is command-line shell that is available on Windows-based systems, allowing users to manage system components. PowerShell provides automation using combinations of cmdlets and scripts.

    In VMware vSphere, you should use PowerCLI as the alternative to a GUI. PowerCLI is a scripting tool built on PowerShell. It extends PowerShell and includes a set of more than 600 cmdlets. You can automate installation and configuration of any features, virtual machines, system patching, etc.

    Clusters

    VMware HA and DRS Clusters Vs. Hyper-V Failover Cluster With PRO

    The Hyper-V virtual environment offers failover cluster functionality. A failover cluster provides automatic failover of VMs from failed hosts to healthy hosts within a cluster. Upon host failure, VMs from the failed host are restarted on other Hyper-V hosts in the cluster. The Performance and Resource Optimization (PRO) feature collects data about resource usage in the virtual environment and generates PRO tips, recommending migration or reconfiguration of VMs for more rational usage of hardware resources. Since Windows Server 2012, PRO was replaced with Dynamic Optimization.

    There are two types of clusters commonly used in VMware vSphere – High Availability (HA) clusters and Distributed Resource Scheduler (DRS) clusters. The HA cluster provides high availability of VMs and carries out automatic VM failover in case of host failure within a cluster. The VMs that were running on the failed ESXi host are restarted on healthy ESXi hosts. The DRS cluster is used for load balancing. vCenter server monitors the usage of hardware resources such as CPU and memory for VMs and ESXi hosts within a cluster. If, after calculations and analysis of these metrics, the DRS decides that some of VMs can migrate from one ESXi host to another (e.g., from an overloaded host to a host with free resources that is almost idle) for more rational resource usage, then migration recommendations are generated. These recommendations can be applied manually or automatically depending on your DRS configuration.

    Power Optimization Vs. Distributed Power Management

    Power Optimization is a feature of Dynamic Optimization in Hyper-V. Power Optimization allows the migration of VMs from Hyper-V hosts with low workloads to other hosts that have enough resources within a cluster. The unused hosts are then powered off in order to save energy. The powered-off hosts are powered back on as soon as their hardware resources are needed for the cluster.

    Distributed Power Management (DPM) is a component of the DRS cluster in the VMware vSphere virtual environment. Similarly to Dynamic Optimization, the DPM executes VM migration from less loaded ESXi hosts to ESXi hosts that have sufficient resources to run the migrated VMs. The freed-up ESXi hosts are powered off (put into a standby state) to save power. When load is increased and the cluster needs more hardware resources again, those ESXi hosts are powered back on.

    Fault Tolerance in VMware HA Clusters

    VMware also provides a useful feature for HA clusters: Fault tolerance. If you use an HA cluster, in the event of ESXi host failure, there is a period of VM downtime between the VM’s failure and the VM being loaded on another healthy host. Using the fault tolerance feature in the HA cluster helps avoid any downtime. This is because a second VM (ghost replica), which is an exact copy of the VM protected with fault tolerance, is already running on another ESXi host within the cluster. The ghost replica is in an inactive state (its network connections and output are disabled) until the primary VM fails. When the host on which the primary VM was running fails, the ghost replica becomes active in an instant, and the network connections of the replica become enabled. Switching to the ghost replica is smooth; you would only notice a slightly higher latency for one packet if you were to ping the IP address of the protected virtual machine. There is no data loss or service interruption if you are using fault tolerance.

    Migration

    Live Migration Vs. vMotion

    There are two types of VM migration for Hyper-V: Live migration and quick migration. Quick migration puts the VM into a saved state, which is similar to hibernation. The VM’s memory is saved to the disk where the other VM files are located. There is a small period of VM downtime once the VM is put into a saved state until it is registered on the new host. Live migration is used for VM migration between Hyper-V hosts (including failover clustering) without losing service availability. The migration procedure consists of different stages. The most important stage of live migration is the iterative transfer of VM memory pages.

    The analog of Hyper-V Live Migration in VMware vSphere is vMotion. With vMotion, running VMs can be migrated from one ESXi host to another without interrupting the VMs and services that are running on the hosts. When the VM migration process from a source ESXi host to a destination host is initiated, the following operations are performed.

    First, compatibility is checked to ensure that the VM can be run on the target host. Hardware resources are reserved on the target ESXi host and the VM process is started. A VM memory checkpoint is created; all further VM memory changes are written to a special area. The data stored in memory before creating a checkpoint is copied to the target VM. The source VM is running and some memory pages are changed (these pages are called dirty pages). They are written to change buffers, which are copied to the target VM and integrated into the memory of the target VM. The process of copying dirty pages is repeated over and over (an iterative process) until there are no memory changes in the source VM’s memory compared to the destination VM’s memory. vMotion then stops the CPU of the source VM, copies the last change buffer, and integrates the memory changes stored in that buffer to the virtual RAM of the target VM. Disk access is discontinued on the source ESXi host and is started on the target ESXi host. vMotion sends a reverse ARP (Address Resolution Protocol) to the physical switch for the purposes of updating the MAC and IP address mappings. The source VM is then shut down and the VM process is deleted on the source ESXi host.

    Storage Migration Vs. Storage vMotion

    Hyper-V Storage Migration can move VM configuration files, checkpoint files, virtual hard disks and mounted ISO files to a new location. Storage migration can be used when the VM must continue running on the same Hyper-V host but must migrate to another storage location. If the VM must be run on another host and placed on another storage location, then live or quick migration must be used along with storage migration.

    Storage vMotion is the vSphere analog of Hyper-V’s Storage Migration. Storage vMotion can perform live migration of VM components such as VM virtual disks (VMDK files) and VM configuration files to another storage (e.g., ESXi local datastore, shared SAN datastore, etc.). First, the VM metadata (including configuration, swap and log files) stored in the VM’s home directory is copied, then the virtual disks are copied.

    Changed block tracking is used to track the changes after the first replication of the source VM’s virtual disk. While the source VM is running, the disk changes are copied in several iterations to the target virtual disk until there are no differences between the source and target virtual disks.

    The virtual disks can be transformed from thick-provisioned disks to thin-provisioned disks or vice versa. During migration with Storage vMotion, the ESXi host on which the VM is running does not change. Storage vMotion is used together with vMotion when the VM must change hosts and datastores at once.

    Storage

    Cluster Shared Volumes (CSV) Vs. VMFS Cluster File System

    Cluster shared volumes are developed to provide simultaneous access to the same disk by multiple Hyper-V hosts in a cluster for reading and writing data. Disk volumes with NTFS or ReFS file systems must be used for CSV.

    VMFS is a virtual machine file system that is optimized for virtual machines and is suitable for clusters in vSphere. A VMFS can be shared by multiple ESXi hosts in a cluster for reading and writing data. This high-performance file system holds files with metadata and has specific locking mechanisms to prevent data corruption caused by writing metadata by multiple hosts concurrently.

    SMB 3.0 Vs. NFS

    SMB 3.0 is a network-sharing protocol developed by Microsoft that can be used for accessing file-level shared storage by Hyper-V hosts in Hyper-V environments as well as Windows environments in general.

    Network File System (NFS) is a sharing protocol developed by Sun Microsystems that provides access to files in file-level shared storage. NFS can be used by different vendors, by different operating systems, and in networks that have different architectures. NFS can be used by ESXi hosts in vSphere for accessing VM files located on shared storage.

    VHDX Vs. VMDK (VM virtual disks)

    VHDX (VHD) is a file format of Hyper-V VM virtual disks. Gen1 VMs can use either VHD or VHDX virtual disk files. Gen2 VMs can use only the VHDX virtual disk file format.

    VMDK is an open file format of VMware VM virtual disks developed by VMware.

    Dynamic Disks vs. Thin-Provisioned Disks

    Dynamic disks are dynamically expanding virtual disks for Hyper-V VMs. The main purpose of using dynamic disks is to save storage space. When Dynamic Disk is created, not all provisioned space is occupied by the virtual disk file. Initially, the virtual disk is a small file. It grows as data is written to the disk.

    Thin-provisioned disks in the VMware virtual environment are similar to Hyper-V’s Dynamic disks. A thin-provisioned disk has a minimal size upon creation and grows as needed as data is written to the virtual disk.

    Checkpoints Vs. Snapshots

    A Hyper-V checkpoint is used to save the state of a virtual machine at a given point in time. You can create a checkpoint, work with a VM, and then revert the VM back to the state it was in when the checkpoint was created.

    The VMware snapshot is the analog to the Hyper-V checkpoint; it, too, is used to save the current state of a VM. You can take a snapshot at any time, e.g., before updating software configuration, updating installations, testing new software, etc. A snapshot can be taken even while a virtual machine is running. The snapshot captures the following VM data: Virtual disk states, virtual memory content, and VM settings. The following files are created and stored in the VM directory when a snapshot is taken: .vmdk, -delta.vmdk, .vmsn, and .vmsd.

    Pass-Through Disks Vs. Raw Device Mapping

    A pass-through disk is a physical disk that is mounted to a VM and used instead of (VHD/VHDX) virtual disks in Hyper-V environments.

    Raw device mapping (RDM) is an alternative to the use of virtual disks for accessing the VM disks in vSphere. Depending on your settings, you can let virtual machines access the physical storage device directly when RDM is used. RDM can be used in virtual compatibility mode or physical compatibility mode.

    Virtual compatibility mode provides full virtualization of the mapped device; snapshots can be used.

    In physical compatibility mode, the guest OS can access the hardware directly; minimal SCSIvirtualization is provided. If a VM snapshot were taken, the RDM disk would not be included in that snapshot.

    For example:

    Suppose DiskName.vmdk is a disk heading file.

    DiskName-rdmp.vmdk is the mapping file (as evidenced by the -rdmp suffix).

    A mapping file resembles a symbolic link to a mapped device.

    RCT Vs. CBT

    Resilient Change Tracking (RCT) makes it possible to copy only those data blocks that have changed since the last backup when an incremental backup of Hyper-V VMs is performed. RCT technology was released with Windows Server 2016. Files with .RCT and .MRT extensions are created when RCT is enabled.

    Changed Block Tracking (CBT) is the vSphere analog of RCT for Hyper-V, helping perform incremental backups. CBT allows backup software to track the blocks that were changed since the last backup and copy only the changed blocks. Files with .CTK extensions are created for each VMDK virtual disk and snapshot file. Block changes are tracked on the virtualization layer. The VMware vSphere API for Data Protection allows third-party software to access the CBT feature for the purposes of performing backups and replications more efficiently. Changed Block Tracking was introduced by VMware in 2009 with ESX 4.0.

    Storage Spaces Vs. vSAN

    Windows Storage Spaces is a technology that abstracts storage from physical disks, used for storage virtualization and provisioning. With Storage Spaces, you can create cost-effective, scalable, highly-available software-defined storage for Hyper-V virtual environments by using servers with HDD devices.

    VMware vSAN (Virtual Storage Area Network) is the equivalent of Windows Storage Spaces. vSAN provides a distributed concept of data storage that is integrated with vSphere at the cluster level and abstracted from hardware disks. ESXi hosts managed with vCenter must be used to deploy vSAN. This approach allows disks of ESXi hosts to be used for vSAN storage while ESXi hosts are used to run VMs. SSDs can be used for caching data, while HDDs are used to store data. Disk groups are aggregated into pools that are accessible for the entire cluster. The data can be mirrored on one or more hosts, which ensures protection against failure. This is defined by the number of failures setting. Using vSAN, you can achieve reliable and scalable enterprise-grade storage.

    Dynamic Memory Vs. Virtual Memory Overcommit

    Dynamic memory allows a VM to use the amount of memory that it currently needs, rather than all virtual memory provisioned for that VM. The hardware RAM on Hyper-V servers can be used more rationally with this feature.

    VMware Virtual Memory Overcommit is the equivalent feature in VMware. Both features use dynamic memory allocation methods, but the methods differ somewhat. With VMware, you can allocate more memory than the ESXi host has for VMs, thereby increasing the density of VMs on the host. The idea is to optimize physical memory consumption – thus, more VMs can run on the ESXi host, but there is no guarantee that all the memory provisioned for it can be provided to a particular VM at a given point in time. Using Memory Overcommit presumes that a memory can be rearranged between the VMs depending on demands. 25% of unused memory is reserved on an ESXi host in case the memory requirements of its VM suddenly increases.

    Hyper-V Integration Services Vs. VMware Tools

    Hyper-V “Integration Services” is a set of tools and drivers for providing proper functionality of child VMs and time synchronization. When Integration Services are installed, the standard drivers for mouse, keyboard, disk controllers, and network controllers are replaced by drivers included as part of the Integration Services suite for better performance and user experience.

    VMware Tools is a collection of drivers, modules, and utilities that is the VMware equivalent of Hyper-V Integration Services. VMware Tools offers advanced functionality to improve performance, user interaction with guest VMs, and integration. When VMware Tools is installed, you can shut down the guest OS directly from menu in the vSphere client or VMware Workstation. Video resolution and graphics performance are increased. Other useful features like Drag & Drop, shared folders, time synchronization, copying/pasting between VMs and hosts, and grabbing/releasing of the mouse cursor also become available. VMware Tools is usually provided as an ISO image or Operating System Specific Package (OSP).

    Migration from Microsoft Hyper-V to the VMware vSphere Platform

    Transitioning from Hyper-V to vSphere can be useful if the vSphere virtualization platform better meets your company’s needs. First, you should prepare the hardware and software. Check VMware’s hardware compatibility list to ensure that your hardware is officially supported. Install ESXi servers, vCenter, and other software that you need. Download and install VMware vCenter Converter. Run VMware vCenter Converter and click Convert Machine.

    Select Hyper-V Server as a source type. Select the IP address or server name of your Hyper-V Server. In this walkthrough, vCenter Converter is installed on the Hyper-V Server and localhost is input as the server name.

    Next, input the username and password of an account that has sufficient permissions to perform the operation. The Windows Administrator account is used in this example. Click Next.

    Select the virtual machine to be converted. Click Next.

    Select VMware Infrastructure virtual machine as the destination type. Type the IP address of your vCenter Server or ESXi host, then enter the username and password. Click Next.

     

    Select the folder and the name that should be applied to the destination VM after conversion. Click Next.

    Select a destination location, including the ESXi host and datastore. Click Next.

    Configure the options for the destination VM. You can see how to change the type of virtual disk provisioning in the screenshot provided below. Click Edit in the Data to copy section.

    In the dropdown list, select the disk provisioning type (thick or thin). Click Next.

    Check the summary and, if you are satisfied, click Finish.

    Wait until the conversion task is complete.

    Once the VM conversion is finished, you can open vCenter with vSphere Web client in your browser and find the converted VM. In this walkthrough, the VM name is Server2016-01. As you can see on the preview screenshot, the VM is successfully running. Click Launch Web console to access the user interface of the VM. You can now tune the virtual machine and install VMware Tools.

    You now know how to migrate from Hyper-V to VMware vSphere.

    Conclusion

    Knowing the differences and similarities between features of VMware and Hyper-V environments makes migration from one platform to another easier and is particularly useful for administrators of a multi-hypervisor environment. Understanding the architecture as well as the working principles of both virtualization platforms, along with their feature sets, helps administrators work comfortably and effectively when they are tasked with maintaining the less-familiar platform. There is one important rule that applies to both VMware and Hyper-V environments: The VMs must be backed up regularly. Robust VM backup can protect your business-critical data against natural disasters, technological failures, human errors and virus attacks.

  • Apache Druid’s Role in Modern Data Analytics

    Apache Druid’s Role in Modern Data Analytics

    From Twitter and Netflix to Salesforce and Confluent, modern analytics applications are being built by developers across the world’s leading digital businesses to create digital tools that serve real-time analytics to hundreds or even thousands of concurrent users.

    Looking more closely at what these apps do reveals some critical, strategic technology decision-making. In particular, why do developers increasingly turn to Apache Druid instead of the myriad of other databases out there?

    In general, they look to Druid to build apps that deliver on a range of objectives, including creating operational visibility at scale, customer-facing analytics, rapid-drill down exploration and real-time ML pipelines. Let’s take a look at each of these in turn:

    Operational Visibility at Scale

    Building apps for operational visibility can include a wide range of use cases, such as product analytics, digital operations, observability, fraud detection and IoT. In each situation, developers need to minimize the gap between events created and time-to-insight.

    From a technology standpoint, there are several databases that can analyze events in real-time, including stream processors like Apache Flink or ksqlDB, time-series databases like InfluxDB or TimeScaleDB and also key-value stores such as Redis. The problem is that all of these technologies share the same inherent limitation for analytics: They aren’t able to process OLAP-style queries effectively at any meaningful scale.

    For instance, if the app in question calls for analyzing time and non-time-based dimensions over the past month, assuming ingestion was one million events per second, that would equate to more than 2.6 trillion events. Any database that supports stream ingestion can query the current status (setting aside the challenge to handle ingestion at this scale) and, to an extent, basic aggregation of numbers and counters. Difficulties can arise, however, if the app requires aggregation across all of these events, “group by” on non-time-based attributes or high concurrency. Using the wrong database leaves users looking at the ‘spinning wheel of death’ as they wait for results.

    In these circumstances, devs turn to Apache Druid. A real-world example explains how: Salesforce engineers built an analytics app using Druid to monitor their product experience. The app enables engineers, product owners and customer service reps to query any combination of dimensions, filters and aggregations on real-time logs for performance analysis, trend analysis and troubleshooting. As a result, their query results are returned in seconds with billions or even trillions of streaming events ingested every day.

    Customer-Facing Analytics

    On the face of things, the technologies offered by Twitter, Forescout and Atlassian don’t have a huge amount in common. Look beneath the surface, however, and they all apply analytics to more than just internal decision-making. They have also created analytics apps and processes to provide their customers with powerful insights as part of a value-added service or a core product offering 

    What does that teach other organizations? When building analytics for external users, a whole different set of tech challenges must be addressed. While concurrency is clearly a key consideration, the database should also support current and future user growth with ease. The thing is, concurrency also has to be delivered economically, or it can have a huge impact on infrastructure costs.

    Similarly, delivering a good interactive experience is dependent on sub-second query response performance—it has become a minimum user requirement almost everywhere. Instead of fighting PostgreSQL or Hive or any other analytics database to meet the performance requirements at scale, Druid offers the performance devs and users expect.

    Take Confluent, for example, which is a popular event streaming platform. They built their cloud-based monitoring service, Confluent Health+, with Druid. Having initially used a different database, they discovered that, as their data volume grew, their legacy pipeline struggled to keep up with their data ingestion and query loads. By switching to Druid, they have been able to create the impressive user experience developers everywhere are focused on delivering.

    Rapid Drill-Down Exploration

    Generally speaking, the primary role of reports is to fulfill a simple requirement: Deliver data in a consumable form. What organizations are increasingly looking for, however, is a deeper level of insight and understanding that shows why something happened. They can take this information and apply it to a wide range of possibilities, such as solving, anticipating or preventing future problems or opportunities.

    Addressing these ‘why?’ questions means data has to be analyzed at tremendous speed to reveal root causes. While this is a simple proposition when the data set in question is small, when dealing with data drawn from today’s high-volume sources, such as cloud services, clicks and IoT/telemetry, the underlying database has to correlate a few to hundreds of dimensions across highly cardinal data with billions to trillions of rows. That presents a much more difficult challenge.

    Indeed, when data is analyzed at a huge scale, full table scans are simply too slow. Exacerbating the problem is the typical query-shaping techniques used to speed up performance have expensive tradeoffs:

    Looking at real-world use cases once again helps demonstrate the role of Apache Druid in situations such as these. For example, developers at cybersecurity company HUMAN built an analytics app for internet bot detection using Apache Druid. Previously, their cloud data warehouse was too slow to keep up with their need to drill into massive web traffic quickly. Today, their Apache-powered app gives their data scientists the ability to aggregate and filter across trillions of events instantly, meaning they can classify anomalies and train their machine learning models to detect real-time malicious activity.

    Real-Time ML Pipelines

    For many people, analytics is synonymous with viewing a synthesized data set presented via a UI like a chart, graph, report or dashboard. Increasingly, however, new operational workflows are completely automated without any human intervention or data visualization.  

    Google Maps is an example of this use case, given it optimizes routing based on real-time and historical traffic flow and patterns. Another example is Reddit’s use of Apache Druid to deliver real-time ad placement within milliseconds of a new visitor arriving on the app. In this context, there is no data visualization to interpret; it has been replaced by automated decisions derived by inference. These performance requirements play particularly well to the capabilities offered by Druid.

    Across a growing variety of demanding use cases, devs are adopting Druid to build inference-based capabilities into their apps. From diagnostics, recommendations and automated decisions, today’s organizations need instant query responses of high cardinality, alongside highly dimensional event data at scale and in real-time. Delivering performance levels like this is becoming increasingly important for businesses focused on innovation, quality of service and competitive advantage.