devopsy.kinsta.cloud

Category: DevOps in the Cloud

  • Akamai Makes Case For Next Era of Cloud Computing

    Akamai Makes Case For Next Era of Cloud Computing

    Akamai Technologies today unveiled a cloud computing service that combines its ability to deploy applications at the edge with the cloud platform it gained via its acquisition of Linode.

    Shawn Michels, vice president of product management for compute and apps at Akamai, said Akamai Connected Cloud will provide the foundation for running a new generation of modern applications that require access to both core infrastructure resources in the cloud alongside highly distributed edge computing services.

    As part of the effort to deliver that combination of services, in the second quarter Akamai is adding three sites in the U.S. to the 11 it already provides. In addition, Akamai revealed it had identified 10 additional cloud sites and 50 distributed edge computing sites that it will add throughout the rest of this year.

    As a long-time provider of a content delivery network, Akamai manages a massive distributed computing environment that spans 4,100 locations across 135 countries. The acquisition of Linode added a set of cloud computing services that enable Akamai to offer the same set of services, including CDN services, as cloud service providers.

    Akamai is now signaling it will compete more aggressively by discounting the egress fees charged whenever data is moved off a cloud platform. One of the most expensive and unpredictable costs is the egress fees cloud service providers charge whenever data is moved off their platforms, noted Michels.

    Finally, Akamai also revealed it has achieved ISO, SOC2 and HIPAA standards compliance for its services.

    Akamai is making a case for an alternative approach to cloud computing that it said will be optimized for both existing monolithic applications running on virtual machines and cloud-native applications based on containers and serverless computing frameworks. A recent survey of 458 development professionals, managers and senior leaders conducted by Techstrong Research, a sister entity of DevOps.com, found 43% of respondents are considering adding additional cloud service providers in the next 12 months. Nearly two-thirds said they are at least considering, evaluating or are ready to buy from a trusted alternative cloud vendor, with 20% reporting they have already contracted with one.

    It’s not clear whether IT organizations might actually migrate existing applications from one cloud service provider to another. At the very least, Akamai expects that as new applications are developed, the company will be able to gain a larger share of the overall market.

    There’s no doubt that a new era of cloud computing is dawning as organizations look to deploy distributed applications that process and analyze data closer to the point where it is created and consumed. These applications will then share the results of the data processed at the edge with, for example, centralized data lakes that make it possible to analyze data collected across a distributed computing environment.

    The one thing DevOps teams can count on is that as this next era of cloud computing evolves, the management of IT environments will become much more challenging for all concerned.

  • Data Mobility: Why it’s a Must and How to Achieve It

    Data Mobility: Why it’s a Must and How to Achieve It

    The workplace has undergone a drastic change, and so has the way companies need to move their data. Wondering how those two points are connected? Let’s dig in.

    It’s clear that remote work is here to stay: Just over a quarter (26%) of U.S. employees are currently working remotely, and a staggering 36.2 million Americans will work remotely by 2025. Naturally, this huge influx of remote (and hybrid) workers has necessitated changes in IT infrastructure. It has changed how people work, engage with their corporate databases and applications and what they expect from their corporate technology infrastructure.

    In the past, companies migrated their data every four or five years when they refreshed their storage infrastructure. CIOs and storage experts now think differently about how data centers are built and optimized. Important questions all companies should consider are: Does it make sense to build out storage capacity ahead of the need? Do we really plan to have our databases and applications hosted on-premises over the next five years? One CIO recently told me, “If you aren’t talking about your future in the cloud, you aren’t paying attention.”

    The Need to Migrate Data More Frequently

    The shift toward a more flexible storage architecture, even in a hybrid cloud environment, has created the need to migrate data much more frequently, fueling the demand for data mobility. Robust data mobility is essential in today’s world: Companies need to securely and efficiently move their data to accommodate a strategy that accommodates both on-premises and cloud; even multiples of these. Data mobility constraints can’t dictate the success of an organization’s digital transformation. Moving data to the best location based on an organization’s goals—whether they’re related to cost, manageability, performance or some other factor—is critical.

    Another critical factor to consider regarding data mobility: The data must always be accessible to the end user, regardless of when or where it’s moving. In the age of remote work, people are working based on their own unique schedules, so data must be accessible at all times.

    In fact, data from Microsoft recently revealed that 30% of its employees choose to work at night. Gone are the days when systems could be offline for data migration during “non-working hours.” Remote work demands continuous data access so employees can stay productive and work at their convenience.

    With this in mind, here are three key steps companies can take to achieve data mobility.

    1. Strategy
    When determining where to store data, consider who will need access to the data, what performance level will be required and what the cost will be for each data storage location. Is everything best contained in a single public cloud? Is a multi-cloud strategy or a hybrid cloud storage environment a better approach? We often talk about data mobility and data migration interchangeably, but not all data migration solutions can continuously move data between storage environments.

    Organizations must also determine how they will move their database and applications to the new environment. If this step isn’t addressed during the strategy stage, it can become an afterthought and create problems down the road.

    2. Deployment
    Once an organization has determined which storage environment best fits its needs, it’s time to move the data from its source to its destination. A single, one-time migration involves moving the data from its current location to a new location and stepping away until the next migration. Data mobility solutions are ideal when data needs to move back and forth, requiring data mobility solutions (e.g., moving from on-premises to a multi-cloud environment).

    During this step, organizations must ask the following questions and plan accordingly to minimize any negative impact: How long will this migration take? How secure is the data in transit from source to destination? How disruptive will the transition be to end users? What will the impact on the organization be? How much manual effort is required?

    Sometimes moving data can make applications sluggish, affecting productivity and resulting in workers overloading the help desk with inquiries about what’s happening. Data migration tools can help organizations tackle deployment efficiently, securely and without disruption to end users (again, continuous data access is vital).

    3. Assessment and cutover
    After the data has arrived at its new destination, it’s time for the cutover process (i.e., running the application in its new location and removing the old storage). Cutover is when problems can arise if everything isn’t mapped correctly. Organizations need protections in place to ensure there is no risk to their operations should something go awry during the process. Additionally, minimizing the impact of this transition for end users is vital, so organizations should also be doing whatever they can to reduce downtime.

    Companies can consider leveraging a data mobility tool that can synchronize in the background without impacting users. So, while users are still working on the old site, new infrastructure can be updated simultaneously, requiring just seconds to minutes of cutover. This time can be planned for off hours so that it’s minimally disruptive and doesn’t require notifying employees.

    The technology landscape is constantly changing, and so are storage vendors’ offerings. Companies need solid data mobility to pivot at a moment’s notice to take advantage of newer, better and more cost-effective capabilities. Data mobility gives companies the flexibility to stay competitive, and data migration tools that allow continuous data access keep workers productive.

  • Leveraging Automation for Secure, Cost-Effective Mobile App Delivery

    Leveraging Automation for Secure, Cost-Effective Mobile App Delivery

    As businesses mobilize their customers and employees, they need to develop innovative mobile applications that safeguard user trust. But secure mobile development and manual security testing can be expensive and time-consuming—a bigger challenge in slowing economies. However, the latest innovations in security automation for mobile teams can drive down security costs by half while speeding release time, getting quality mobile apps to market faster to grow revenue faster.

    With mobile dominating all digital time spent online versus the web, businesses cannot compromise the safety of their mobile users and mobile data. Security automation integrated throughout the development process empowers developer, QA and security teams’ efficiency, enabling faster release cycles with built-in security. This approach powers success for digitally enabled businesses that depend on mobile apps to generate revenue, engage customers and tap new markets.

    A single security vulnerability within a mobile app can instantly damage a business’s financial stability, brand reputation and customer trust. Even highly respected brands like UnderArmour and British Airways experienced serious issues after their mobile apps were breached. Business leaders should consider the benefits of adding security automation directly into the development pipeline to drive quality and protect users in a cost-effective manner.

    Faster, Lower Cost Developer Cycles

    Suppose a business creates a 12-week plan to develop, test and release a mobile app software update with a labor budget of $60,000 per week. The development team creates a schedule to write the code in 10 weeks and allocates a two-week period for a manual security pentest. After developers finish writing the code, they must wait for the pentest results to point out any security issues. When testing uncovers security issues, devs, security analysts and QAs must determine which issues need to be remediated. Then the devs can fix the issue, retest to ensure it is properly fixed, then release it. Most of the time, finding any issues delays the release by weeks or months, delaying the expected revenue generation and other business benefits by weeks or months, in turn.

    Instead, businesses can take a cost-effective approach by running security automation continuously through the development life cycle. Dev teams can write code, allow security automation to run during off hours, and have tickets with built-in remediation info sent back to devs the next day. This can remove weeks of testing and remediation delays to ensure teams release on time and on budget. Businesses can cut costs further by establishing common coding, testing and remediation policies in advance and then deploying an automated policy engine to focus on just those security requirements and issues relevant to the policy to operate at the fastest pace possible.

    Integrating automated security testing into development tools enables DevSecOps teams to build, test, fix and release faster with security built in. Not only does this strategy increase the security and quality of the mobile app, it drastically lowers development costs by improving team efficiency.

    Faster, Lower Cost Security Testing Cycles

    Every organization with a mobile app has different security requirements. Those with low-risk mobile apps may outsource their security testing once or twice a year. Those in highly regulated industries like healthcare and financial services may have internal security analysts and test as often as every release and event outsource pen testing quarterly. Whether testing internally or externally, security testing costs can become a major pain point for companies looking to manage their finances efficiently.

    The average cost of a full-scope, two-week outsourced manual pentest costs around $15,000 to $20,000 per test. If an organization tests its mobile app on a quarterly basis, its pentesting costs jump to $60,000 to $80,000 per year. Internal pentesting teams can potentially be cheaper, but organizations still need to pay for analysts’ salaries and the tools and resources they need to conduct tests. Whether a business has an internal security team or relies on outsourced manual pentests, leveraging security automation can significantly reduce costs.

    Businesses with internal security teams can use automation to test builds immediately after devs release code. By continuously assessing code quality from start to finish, security teams can eliminate their testing time and speed up release cycles. Businesses that outsource security testing can use security automation to cut costs by reducing the frequency of full-scope manual pentests. Instead of testing on a biannual or quarterly basis, continuous automation can help reduce the testing frequency to once or twice a year with comparable coverage.

    The most efficient approach possible is the hybrid security automation approach of guided testing that combines daily automated security testing as mobile apps are built with periodic human security analysts for coverage of the parts of the mobile app that automation can’t cover. Guided testing starts below $20,000 per mobile app for unlimited automated security testing integrated into DevSecOps toolchain with four periodic guided tests per year, slashing outsourced quarterly pentesting costs by more than 75%.

    Mobile apps play an important role in helping businesses generate much-needed revenue, but running fast and cutting corners without security built in ultimately places the business and users at risk. So business leaders should take extra steps to keep them secure, and securing mobile apps can be achieved without breaking the bank. By incorporating security automation and innovations like guided testing into the mobile app development process, businesses can maximize the speed of delivery while simultaneously reducing costs and improving quality.

  • Report Affirms Continued AWS Cloud Dominance

    Report Affirms Continued AWS Cloud Dominance

    A report published this week by Wiz, a provider of a cloud security platform, found more than half of organizations (57%) are using multiple clouds, with 22% currently using three or more platforms.

    However, the report also noted that 72% of all workloads are running on the Amazon Web Services (AWS) cloud. Among organizations that have more than one cloud platform, Microsoft Azure (41%) is the most common secondary platform (41%), while Google Cloud Platform (GCP) is the most used tertiary platform among organizations employing two or more clouds (44%).

    The report also noted 9% of AWS customers use managed database servers, compared to 87% in GCP and 82% in Azure. However, over 91% of companies also have non-managed database servers running in a cloud. In total, only 6% of companies use managed database servers exclusively, compared to 90% using a mix of managed and non-managed database servers, the report finds.

    The most widely deployed databases are PostgreSQL, Redis and MySQL, with 90% of organizations running at least one instance of a PostgreSQL server.

    Scott Piper, principal cloud security researcher for Wiz, said the report makes it clear that despite gains by Microsoft and Google, AWS remains the most dominant cloud service provider. Even after a decade, rival cloud service providers have not been able to overcome the first-mover advantage that AWS continues to enjoy. In many cases, a second or third cloud is being employed only because a single development team preferred it or an IT team assumed responsibility for it when their company acquired another organization using it, noted Piper.

    However, while most organizations have multiple accounts per cloud service provider, the report also found that the bulk of the application workloads are managed via only a handful of accounts, he added. That suggests more organizations are subject to higher levels of disruption in the event of an outage than many of them may realize, said Piper.

    One of the reasons so many organizations have a narrow range of accounts, of course, is they simply don’t have enough personnel. In fact, a general skills shortage also tends to limit the number of cloud platforms organizations might otherwise employ.

    Multiple cloud platforms also tend to exacerbate cybersecurity challenges. With the introduction of each additional cloud service, the overall size of the attack surface that needs to be defended expands. The Wiz report found 47% of organizations have at least one database or storage bucket publicly exposed to the internet. It can take less than 13 hours for cybercriminals to discover and access an exposed bucket with a guessable name, noted Piper.

    More challenging still are the number of application programming interfaces (APIs) that need to be secured as cloud service providers continue to add new services, he added.

    Ideally, organizations should be adopting a framework to harden privileges, encryption, authentication and connectivity and improve hygiene, otherwise known as PEACH, said Piper.

    Regardless of the approach to cloud computing, it’s apparent there’s still a long way to go before organizations are ready to embrace hybrid cloud computing spanning multiple clouds.

  • Voice.ai ‘Stole’ Code ¦ AWS Gets Filthier

    Voice.ai ‘Stole’ Code ¦ AWS Gets Filthier

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: Alleged theft of GPL code, and Amazon will run its data centers on gas. (more…)

  • Cisco Bets on OpenTelemetry to Advance Observability

    Cisco Bets on OpenTelemetry to Advance Observability

    Cisco today announced a technology preview of a Cisco Full-Stack Observability Platform based on instances of open source OpenTelemetry agent software being advanced by the Cloud Native Computing Foundation (CNCF) to collect metrics, event, logs and distributed traces. The Cisco Full-Stack Observability platform will be available in June 2023.

    The platform, announced at a Cisco Live EMEA event in Amsterdam, Netherlands, will combine the capabilities of multiple Cisco platforms and technologies into a single offering.

    At the same time, the company also announced the bidirectional integration between its existing Cisco AppDynamics application life cycle management (ALM) tool and the Cisco ThousandEyes network monitoring platform beginning in April.

    Finally, Cisco is making available a Business Risk Observability offering to provide business risk scoring. Business Risk Observability combines capabilities of its Kenna Risk Meter tool—gained when Cisco acquired Kenna Security in 2021—with the ability to monitor business transactions through Cisco AppDynamics to provide better visibility into application security. In addition, Cisco has integrated its Panoptica application programming interface (API) management security tool and the Talos threat intelligence platform within the Business Risk Observability Platform.

    Carlos Pereira, a Cisco fellow and chief architect for strategy, incubation and applications at Cisco, said Cisco is moving to narrow the historic divide between application development and cybersecurity teams to enable organizations to adopt DevSecOps best practices and make it simpler to prioritize the remediation of vulnerabilities that are likely to have the most impact on the business.

    Like most providers of observability platforms, Cisco is betting that more applications will be instrumented as OpenTelemetry software continues to mature. Cisco is now the single largest contributor to this project, noted Pereira.

    OpenTelemetry will also play a critical role in improving application security as more applications become instrumented, he noted. In fact, a recent Cisco AppDynamics survey found more than three-quarters of respondents (76%) believed that DevSecOps is essential for organizations to effectively protect against a multi-stage security attack. However, only 43% reported their organization had started implementing DevSecOps workflows.

    Observability, of course, has always been a core DevOps tenet, but achieving and maintaining it is challenging. Most DevOps teams today aspire to maintain some level of continuous monitoring. However, as it becomes easier and less costly to instrument applications, interest is rising in observability platforms that make it simple to investigate anomalies indicative of an issue that could disrupt an application environment. Most of those observability platforms are infusing those platforms with machine learning algorithms to surface those anomalies as part of an effort to augment the capabilities of a DevOps team.

    Regardless of the approach to observability, many organizations will soon be trying to determine whether to rely on existing monitoring tools that make it possible to track pre-defined metrics. In some cases, there may still be a need for that legacy approach to managing IT. Still, in other instances, it may be possible to rationalize some of those platforms in favor of a more advanced approach that relies on a more advanced approach to achieving observability.

  • 5 Technologies Powering Cloud Optimization

    5 Technologies Powering Cloud Optimization

    Cloud optimization is the process of improving the performance, cost-effectiveness, and reliability of cloud computing resources and services. It involves identifying and addressing inefficiencies in cloud architecture, application design and resource utilization, as well as implementing best practices for security, scalability and disaster recovery. 

    The goal of cloud optimization is to maximize the benefits of cloud computing while minimizing the risks and costs associated with it. It is a multi-disciplinary process that may involve several key elements, such as:

    • Capacity planning: Evaluating the current and future resource requirements of your cloud environment and ensuring that the right amount of resources are available to meet your needs.
    • Resource utilization: Monitoring and managing the utilization of cloud resources to ensure that they are used efficiently and effectively.
    • Cost optimization: Minimizing the cost of cloud resources and services by identifying and addressing areas of inefficiency and waste, such as over-provisioning or underutilized resources.
    • Performance optimization: Improving the performance of cloud applications and services by optimizing network bandwidth, storage performance and database performance.
    • Security optimization: Enhancing the security of cloud resources and data by implementing best practices for access control, encryption and threat mitigation.
    • Scalability optimization: Making sure that your cloud environment can scale up or down as needed to meet changing resource demands.
    • Disaster recovery: Establishing a disaster recovery plan to ensure that your cloud environment can quickly recover from failures or outages.

    Why is Cloud Optimization Important?

    Optimization is important for several reasons, including:

    • Cost savings: By optimizing the use of cloud resources, organizations can reduce their cloud computing costs and ensure that they are only paying for what they actually need.
    • Improved performance: By optimizing cloud applications and services, organizations can improve the performance and responsiveness of their systems, leading to a better user experience and increased productivity.
    • Increased reliability: Optimization helps organizations to improve the reliability and availability of their cloud environment, reducing the risk of downtime and ensuring that their systems are always available when they are needed.
    • Better security: By implementing best practices for security, organizations can better protect their cloud resources and data, reducing the risk of data breaches and other security incidents.
    • Scalability: Cloud optimization enables organizations to scale their environment up or down as needed, ensuring that they always have the resources they need to support their growing business.
    • Improved agility: Organizations can be more agile and responsive to changing business needs, enabling them to quickly and effectively respond to new opportunities and challenges.

    5 Technologies Powering Cloud Optimization

    Cloud Cost Management

    Cloud cost management is a critical component of optimization that helps organizations to monitor and manage their cloud spend. The goal is to ensure that organizations are only paying for the cloud resources they actually need and that they are using those resources efficiently.

    Cloud cost management solutions (such as Azure Cost Management or AWS Trusted Advisor) provide detailed cost reports, usage analytics and cost optimization recommendations to help organizations understand their cloud spend and identify areas of inefficiency. For example, these solutions can:

    • Monitor and report on cloud resource usage: By monitoring the usage of resources, cloud cost management solutions can provide detailed cost reports that show how much an organization is spending on each resource, such as compute instances, storage and network traffic.
    • Automate cost optimization: Many cloud cost management solutions can automate cost optimization tasks, such as turning off unused resources or right-sizing instances to reduce waste.
    • Provide cost analysis and forecasting: These solutions can provide detailed cost analysis and forecasting to help organizations understand the impact of changes to their cloud environment and plan for future spending.

    Autoscaling

    Autoscaling is a technology that enables organizations to automatically scale their cloud resources up or down as needed to meet changing demands. The goal of autoscaling is to ensure that organizations always have the right amount of resources to support their workloads while minimizing costs and ensuring that their systems are always available when they are needed.

    Autoscaling works by monitoring the performance and usage of cloud resources, such as compute instances, storage and network traffic, and automatically adjusting the size of those resources to meet changing demand. For example, autoscaling can:

    • Add or remove instances as needed: If demand for a resource increases, autoscaling can automatically add more instances to meet that demand. If demand decreases, autoscaling can automatically remove instances to reduce waste.
    • Optimize resource utilization: Autoscaling can optimize the utilization of cloud resources by right-sizing instances to meet demand, ensuring that resources are not over-provisioned or underutilized.
    • Improve system availability: By automatically adding or removing resources as needed, autoscaling can ensure that systems are always available and responsive, reducing the risk of downtime and improving the user experience.

    API Gateway

    An API gateway is a server that acts as an intermediary between an application and one or more microservices. The API gateway is responsible for request routing, composition and protocol translation, which enables microservices to communicate with each other securely and efficiently.

    An API gateway can help with cloud optimization in several ways:

    • Load balancing: The API gateway can distribute incoming API requests to multiple microservices, improving the overall performance and reliability of the system. It can also provide features such as automatic failover and load balancing, which helps to ensure that the system remains available and responsive even in the face of high traffic volumes or other performance challenges.
    • Resource optimization: By acting as a single entry point for API requests, the API gateway can help to reduce the overall resource usage of the system. For example, it can cache frequently used data and serve it directly to clients without having to go back to the microservices, reducing the load on the underlying system.
    • Monitoring and management: API gateways provide a centralized management interface for the microservices, making it easier to monitor and manage the system as a whole. This includes features such as logging, monitoring and alerting, which can help to identify and resolve performance issues quickly and effectively.

    GitOps Platforms

    GitOps is a paradigm that helps automate application deployment and infrastructure provisioning. It typically involves using Git, an open source version control system, as a single source of truth for declarative infrastructure and applications. 

    GitOps aims to improve the developer experience by letting teams use their familiar software development tools to manage application and infrastructure changes.

    With GitOps, cloud optimization can be achieved by:

    • Automating the deployment process: GitOps automates the deployment process, reducing the risk of human error and speeding up the deployment process. This helps to ensure that resources are deployed consistently and correctly, improving the reliability and performance of the environment.
    • Managing cloud infrastructure as code: With GitOps, cloud infrastructure can be defined and managed as code, making version control and collaborating on configurations easier. This helps to optimize the use of resources and improve the overall efficiency of the cloud environment.
    • Improving collaboration and communication: GitOps enables teams to collaborate and communicate more effectively, reducing the risk of conflicts and delays. With Git, changes can be easily tracked, discussed and approved, improving the speed and quality of deployments.
    • Enforcing governance and compliance: GitOps provides a centralized and auditable record of all changes to the environment, making it easier to enforce governance and compliance requirements. This helps to improve the security and reliability of the cloud environment, reducing the risk of breaches and other security incidents.

    APM

    Cloud application performance management (cloud APM) is the practice of monitoring, analyzing and optimizing the performance of cloud-based applications. The goal is to ensure that these applications are available, responsive and performing optimally to meet the needs of users.

    These solutions provide a range of tools and capabilities for monitoring and optimizing cloud-based applications, including:

    • Performance monitoring: Cloud APM solutions monitor application performance in real-time, providing visibility into performance metrics such as response time, resource utilization and error rates.
    • Root cause analysis: These solutions can help identify the root cause of performance issues by providing detailed performance data and tracing information.
    • Capacity planning: Cloud APM solutions can provide detailed performance data and analytics to help organizations plan and optimize their application deployments.
    • Performance optimization: These solutions can provide recommendations for performance optimization, such as right-sizing cloud resources, reducing resource utilization and improving application design.
    • Continuous performance monitoring: These solutions can continuously monitor application performance, providing real-time visibility and alerting to help organizations quickly identify and resolve performance issues.

    Conclusion

    In conclusion, cloud optimization is critical for organizations looking to maximize the value and minimize the costs of their cloud deployments. By using technologies such as cloud cost management, autoscaling, API gateways, GitOps platforms and APM, organizations can improve availability, performance and efficiency while reducing costs and ensuring that they are always prepared to meet changing business needs. 

    These technologies provide the tools and capabilities necessary to monitor, analyze and optimize cloud-based applications, helping organizations to ensure that their deployments are always delivering the best possible value. By embracing optimization, organizations can stay ahead of the curve and ensure that their cloud deployments are always aligned with their business goals and objectives.

  • Azure Migration Strategy: Tools, Costs and Best Practices

    Azure Migration Strategy: Tools, Costs and Best Practices

    Azure migration refers to the process of moving applications, data and other resources from on-premises or other cloud environments to Microsoft Azure, which is a cloud computing platform and service offered by Microsoft. 

    Azure migration can include migrating virtual machines, databases and storage to Azure, as well as re-architecting applications to take advantage of Azure services. Typically, the goal of Azure cloud migration is to take advantage of the scalability, security and cost benefits of the Azure platform.

    4 Azure Migration Tools

    Here are several tools provided by the Azure cloud which can help you migrate applications to the Microsoft cloud.

    Azure Migrate

    Azure Migrate is a service offered by Microsoft as part of Azure that helps customers discover, assess, and migrate their on-premises workloads to Azure. It provides a centralized hub for the discovery and assessment of on-premises workloads and helps customers plan and execute their migration to Azure. Azure Migrate includes the following features:

    • Discovery: Azure Migrate discovers on-premises servers and their dependencies and creates an inventory of them.
    • Assessment: Azure Migrate assesses the readiness of on-premises servers for migration to Azure and provides recommendations for target Azure resources.
    • Migration: Azure Migrate helps customers plan and execute their migration to Azure using various migration tools such as Azure Site Recovery, Azure Database Migration Service and Azure Data Box.

    By using Azure Migrate, customers can move their workloads to Azure with confidence, knowing that they have a comprehensive plan and the right tools in place to ensure a successful migration.

    Azure Database Migration Service

    Azure Database Migration Service (DMS) is a fully managed service offered by Microsoft as part of Azure that simplifies and accelerates the process of migrating on-premises databases to Azure. The service supports a variety of databases, including SQL Server, Oracle, MySQL, and PostgreSQL.

    Azure DMS can perform both online and offline migrations, depending on the requirements and availability of the source database. Online migrations are performed while the source database is still in use, while offline migrations require the source database to be taken offline during the migration process.

    The service supports several migration scenarios:

    • One-time migration: Move a database to Azure once and leave it running in Azure
    • Lift and shift: Migrate an on-premises database to Azure and keep the same database engine running in Azure
    • Change the engine: Migrate an on-premises database to Azure, but change the database engine running in Azure to a different one.

    The Azure DMS provides a simple and efficient way to move your on-premises databases to Azure with minimal downtime, and it’s integrated with Azure Migrate for a holistic migration experience.

    Azure Data Box

    Azure Data Box is a service offered by Microsoft as part of Azure that allows customers to transfer large amounts of data to Azure for storage and processing. The service is designed for customers who have a large amount of data that needs to be transferred to Azure, but have limited network bandwidth or other constraints that make it difficult to transfer the data over the Internet.

    Azure Data Box provides customers with a physical device (called an “Azure Data Box”) that they can use to transfer data to Azure. The device is shipped to the customer, who can then copy data to the device and ship it back to Microsoft. Microsoft will then transfer the data to the customer’s Azure storage account.

    There are several different types of Azure Data Box, each with different storage capacities and data transfer speeds:

    • Azure Data Box: 100TB storage capacity, data transfer speed of 100Mbps
    • Azure Data Box Disk: 40TB storage capacity, data transfer speed of 40Mbps
    • Azure Data Box Heavy: 1PB storage capacity, data transfer speed of 100Mbps
    • Azure Data Box Edge: 100TB storage capacity, data transfer speed of 100Mbps and AI and IoT edge computing capabilities.

    Azure Data Box is particularly useful for customers who have large amounts of data to transfer, and don’t have the network bandwidth or other resources to transfer the data over the Internet.

    Azure Synapse Pathway

    Azure Synapse Pathway is a new feature of Azure Synapse, which is a fully managed cloud-based data integration and analytics platform. Azure Synapse Pathway is designed to make it easy for customers to move their existing data integration and analytics workloads to Azure Synapse without requiring significant re-architecting or rewriting of their existing code.

    Azure Synapse Pathway supports several migration scenarios, such as:

    • Lifting and shifting on-premises SQL Server Integration Services (SSIS) packages to Azure Data Factory.
    • Migrating on-premises SQL Server Analysis Services tabular models to Azure Analysis Services.
    • Migrating on-premises Power BI reports to Power BI in Azure Synapse.

    Azure Synapse Pathway also provides a migration tool that helps customers to move their existing data integration and analytics workloads to Azure Synapse, with minimal changes to their existing code. This allows customers to take advantage of the benefits of Azure Synapse without significant re-architecting or rewriting of their existing code.

    Azure Migration Strategy and Best Practices

    Prepare an Inventory of Digital Assets

    Understanding your digital estate is an important part of creating an effective Azure migration strategy. A digital estate refers to the collection of digital assets and resources that an organization uses to run its business, including applications, data, and infrastructure. In order to migrate your digital estate to Azure, you need to understand what you have and where it’s running. This information can be used to create a migration plan that is tailored to your specific needs.

    To understand your digital estate, you should start by creating an inventory of your applications, data, and infrastructure. This inventory should include information such as:

    • The name and purpose of each application
    • The technology stack used by each application
    • The location of the application and its data
    • The dependencies between applications and data
    • The number of users and usage patterns for each application
    • The storage and compute requirements for each application

    Once you have an inventory of your digital estate, you can use it to create a migration plan that takes into account the specific needs of each application, as well as any dependencies or constraints that need to be considered.

    Best practices for inventorying your digital assets:

    • Start by identifying the most critical applications and data that need to be migrated first.
    • Assess the readiness of your existing infrastructure and applications for migration to Azure.
    • Understand the dependencies between your applications and data, and plan for them accordingly.
    • Identify any constraints or limitations that need to be considered when planning your migration.

    By understanding your digital estate and creating a migration plan that takes into account the specific needs of your applications and data, you can ensure that your migration to Azure is successful and that your business is able to continue to operate during and after the migration.

    Estimate Monthly Workload Costs

    Estimating the monthly costs of your workloads is an important part of creating an Azure migration strategy. By understanding the costs associated with running your workloads in Azure, you can make informed decisions about how to migrate your workloads and how to optimize your cloud costs after migration.

    To estimate the monthly costs of your workloads in Azure, you can use the Azure pricing calculator. The pricing calculator allows you to estimate the costs of running various Azure services, such as virtual machines, storage, and networking, based on the specifications of your workloads. You can also use the Azure TCO calculator to estimate the total cost of ownership of your workloads in Azure, taking into account costs such as data transfer, management, and support. There are multiple other Azure calculators you can use to estimate storage costs, networking costs, and more.

    Image Source: Azure

    When estimating the costs of your workloads in Azure, it’s important to consider the following factors:

    • The size and number of virtual machines required to run your workloads
    • The storage and network requirements of your workloads
    • The number of users and usage patterns of your workloads
    • The services and features required to run your workloads, such as Azure Backup or Azure Security Center

    It’s also important to consider how the costs of your workloads may change over time, such as how the number of users and usage patterns may change or how the costs of Azure services may change.

    Best practices for estimating the monthly costs of your workloads in Azure:

    • Use the Azure pricing calculator and TCO calculator to estimate the costs of running your workloads in Azure.
    • Understand how the costs of your workloads may change over time and plan accordingly.
    • Optimize your costs after migration by using Azure cost management and Azure reservations.
    • Continuously monitor your costs and usage to ensure you’re getting the most value from your investment in Azure.

    By understanding the costs associated with running your workloads in Azure, you can create a migration strategy that is optimized for cost and that enables you to get the most value from your investment in Azure.

    Migrate Workloads and Optimize

    Migrating your workloads to Azure and optimizing them for the cloud environment is an important step in creating an Azure migration strategy. Once you have a clear understanding of your digital estate and have estimated the costs of running your workloads in Azure, you can begin migrating your workloads and optimizing them for the cloud.

    There are several ways to migrate workloads to Azure, depending on the type of workload and the migration requirements. Some common migration methods include:

    • Lift and shift: This method involves moving an existing workload to Azure without making any changes to the workload. This is typically used for simple workloads that do not require significant changes to run in the cloud.
    • Re-architecting: This method involves making changes to the workload to take advantage of the features and capabilities of Azure. This may include re-architecting the application to suit the target Azure service, such as Azure SQL Database or Azure Storage.
    • Refactoring: This method involves making changes to the workload to optimize it for the cloud environment. This may include refactoring the application using cloud-native patterns and practices, such as microservices or containerization.

    Once your workloads are migrated to Azure, you can optimize them for the cloud environment by:

    • Scaling your workloads to match demand.
    • Using Azure services and features to improve the performance and security of your workloads.
    • Monitoring and managing your workloads to ensure that they are running efficiently.
    • Optimizing costs by using Azure cost management and Azure reservations.

    Best practices for migrating your workloads and optimizing for the cloud environment:

    • Start by migrating your most critical workloads first.
    • Understand the migration method that is best suited for your workloads and plan accordingly.
    • Use Azure services and features to improve the performance and security of your workloads.
    • Continuously monitor and manage your workloads to ensure that they are running efficiently.
    • Optimize your costs by using Azure cost management and Azure reservations.

    By following these best practices, you can ensure that your workloads are migrated to Azure and optimized for the cloud environment, which will help you to get the most value from your investment in Azure.

    Monitoring

    Monitoring is an important part of creating an Azure migration strategy and ensuring that your workloads are running efficiently in the cloud. By monitoring your workloads, you can identify and address issues that may arise during or after migration and ensure that your workloads are meeting the needs of your business.

    Azure provides a variety of tools and services for monitoring your workloads, including:

    • Azure Monitor: A service that provides a unified monitoring solution for Azure resources, including virtual machines, storage, and networking.
    • Azure Log Analytics: A service that allows you to collect and analyze log data from your workloads.
    • Azure Application Insights: A service that allows you to monitor the performance and availability of your applications.
    • Azure Security Center: A service that provides security monitoring and management for your workloads.

    When monitoring your workloads in Azure, it’s important to consider the following factors:

    • The performance and availability of your workloads
    • The usage and costs of your workloads
    • The security and compliance of your workloads

    It’s also important to consider how your monitoring needs may change over time, as your workloads and usage patterns may change.

    Best practices for monitoring your workloads in Azure:

    • Use Azure Monitor, Log Analytics, Application Insights, and Security Center to monitor your workloads.
    • Continuously monitor your workloads to ensure that they are running efficiently.
    • Use Azure alerts to notify you of issues that may arise.
    • Use Azure dashboards to create custom views of your monitoring data.
    • Continuously assess and monitor your digital estate to ensure that it’s up-to-date and that your migration plan is still valid.

    By monitoring your workloads in Azure, you can identify and address issues that may arise during or after migration, and ensure that your workloads are meeting the needs of your business. This will help you to get the most value from your investment in Azure and ensure a successful migration.

    Conclusion

    In conclusion, migrating to Azure is a great way to take advantage of the scalability, security, and cost benefits of the cloud. However, migrating your workloads to Azure requires a well-planned strategy that takes into account the specific needs of your business and your digital estate. 

    To create an effective Azure migration strategy, it is important to understand your digital estate, estimate the costs of running your workloads in Azure, choose the right strategy, and optimize them for the cloud environment. 

    Additionally, monitoring your workloads in Azure is crucial to ensure that they are running efficiently, and to address any issues that may arise during or after migration. By following best practices and using the tools and services provided by Azure, you can ensure a successful migration and get the most value from your investment in Azure.

  • OpenAI Hires 1,000 Low Wage Coders to Retrain Copilot ¦ Netflix Blocks Password Sharing

    OpenAI Hires 1,000 Low Wage Coders to Retrain Copilot ¦ Netflix Blocks Password Sharing

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: ChatGPT darling OpenAI wants people to write code in English, and the unintended consequences of blocking shared accounts. (more…)

  • Stream Big, Think Bigger: Analyze Streaming Data at Scale

    Stream Big, Think Bigger: Analyze Streaming Data at Scale

    Data streams are all the rage. Once a niche element of data engineering, streaming data is the new normal—more than 80% of Fortune 100 companies have adopted Apache Kafka, the most common streaming platform, and every major cloud provider (AWS, Google Cloud Platform and Microsoft Azure) has launched its own streaming service.

    This shift is fueled by the growing demand to deliver events quickly, reliably and at scale to support a wide range of internal and external applications. From always-on, customer-facing applications to mission-critical operations, a growing number of use cases depend on subsecond updates—further upping the ante for real-time data processing and dissemination.

    There’s no doubt 2023 will be a big year for streaming technology.

    Meanwhile, as streaming moves toward ubiquity, we’re seeing another shift in the way businesses use the data: Analyzing events as they’re created for real-time insights. With the right tools in place, people can instantly compare what’s happening now to what happened in the past, triage problems as they occur and make time-sensitive decisions in the moment.

    With the proliferation of data streams, a new set of use cases and requirements for real-time analytics has emerged. To unlock the power of streaming in 2023, data teams will need to look beyond the traditional batch-oriented stack and adopt a streaming-native approach to the analytics architecture.

    The Streaming Paradigm Shift

    Data has always been at the center of how businesses operate. Until recently, we lived in a very batch-dominant world of business and data. The goal of data infrastructure was to identify data at a fixed point in time and store it for eventual use. The evolution from daily batch operations on mainframes to the internet-driven always-on world replaces fixed, static “data at rest” with fast-moving data in motion, as information flows freely between applications and data systems within and between organizations.

    Data at rest still exists, and batch systems continue to serve a variety of reporting purposes. But reality isn’t fixed or static, and to meet the demands for seamless, authentic data experiences, the systems we build must be designed for data in motion.

    Hence, the rise of streaming technology—and with it, a new mindset for how we think about data. As streaming increases, streaming data platforms increasingly act as an organization’s central nervous system, connecting the different functions and driving mission-critical operations. Evolving technologies are purpose-built to support data in motion systems: Stream processors and event databases.

    Apache Druid fits into the purpose-built category; as a real-time analytics database, it is designed to enable users to query every event as it joins the data stream and do it at a tremendous scale while enabling subsecond queries on a mix of stream and batch data.

    Thousands of businesses are already using streaming processors like Kafka and Amazon Kinesis with Druid to build cutting-edge systems that make terabytes to petabytes of streaming data accessible to applications and humans in milliseconds. Some of those businesses—including Reddit, Citrix and Expedia—were highlighted at Current 2022, the annual streaming event organized by Confluent.

    The next evolution of data intelligence is here, and it’s based on the ability to react to events as they are happening. But on the growth chart of streaming adoption, we are only at the beginning of an upward curve—a critical inflection point where streaming (and the technology that is built for it) starts to become the bedrock of everyone’s data architecture.

    So, when it comes to enabling scalable, subsecond analytics on streaming data, many developers and data innovators are wondering, ‘How?’

    We’re Streaming! Now What?

    At Current, we spoke with hundreds of Kafka users who were turning their attention to building out the next phase of their data streaming pipelines and embedding them more deeply into their organizations.

    While streaming adoption is becoming more widespread, most companies still have just one or two use cases they’re solving with a streaming platform. At Current, many people spoke about how Kafka effectively set their data in motion. But when it came time to analyze or put those streams to work in a user-facing application, their “data in motion” turned into “data in waiting” because their analytics system was designed for batch data, not streaming.

    To solve this problem, a new type of database is needed—and that’s why Druid was created. Turning billions or trillions of events into streams that can then be immediately queried by thousands of users simultaneously is where stream processors like Kafka, in combination with Druid, have unlocked a new set of capabilities that are enabling new developer-built analytics applications.

    For example, Reddit generates tens of gigabytes of event data per hour from advertisements on its platform. To let advertisers understand their impact and decide how to target their spending, Reddit needs to enable interactive queries across the last six months of data—hundreds of billions of raw events to slice and dice. But they also needed to empower advertisers to see user groups and sizes in real-time, adjusting based on interests (sports, art, movies, technology…) and locations to find how many Reddit users fit their target. To achieve this, they built a Druid-powered application that ingests data from Kafka and enables Reddit’s advertising partners to make real-time decisions and get the best possible ROI on their campaigns.

    Stream-to-Batch Vs. Stream-to-Stream

    Reddit chose Druid as the database layer of their application in large part because of its close integration with Kafka, as Druid was designed to ingest and analyze streaming data. This sets Druid apart from virtually every other analytics database, the rest of which were built for batch ingestion.

    Batch ingestion works exactly the way it sounds—it takes chunks of stream data, dumps them into a file, processes the file, then loads it into the database. Using a batch-based system to analyze streams introduces a bottleneck to your streaming pipeline. By contrast, Druid provides a connector-free integration with the top streaming platforms and handles the latency, consistency and scale requirements for high-performance stream analytics cost-effectively.

    In addition to out-of-the-box integrations, Druid has built-in indexing services that provide event-by-event ingestion. This means streaming data is ingested into memory and made immediately available for use. No waiting for events to be processed through batch processes before they can be queried. This capability, coupled with exactly-once semantics, guarantees data is always fresh and consistent. If there’s a failure during stream ingestion, Druid will automatically continue to ingest every event exactly once (including events that enter the stream during the outage) to prevent any duplicates or data loss.

    However, perhaps the most powerful reason to use Druid for streaming data analytics is its near-infinite scalability. For even the largest, most complex ingestion jobs, Druid can easily scale into the tens of millions of events per second. With variable ingestion patterns, Druid avoids resource lag (as well as overprovisioning) by enabling dynamic scaling. It combines the query and ingestion performance of shared-nothing architecture with a cloud data warehouse’s flexibility and non-stop reliability. This means you can add computing power and scale out without downtime or rebalancing, which is handled automatically.

    In essence, while using a cloud data warehouse to serve both batch-oriented and real-time use cases might sound efficient, doing so defeats the purpose of a streaming pipeline.

    “The weakest link in the chain sort of dominates everything,” Databricks CEO Ali Ghodsi told the audience at Current. “If you have one step of your pipeline that is batch-processed and really slow, then it doesn’t matter how fast you are in the rest of the pipeline.”

    That’s why when Confluent’s data engineering team needed to build a real-time observability platform, they chose to use Kafka and Druid together. Today, the platform ingests over five million events per second from Kafka and handles hundreds of concurrent queries from a large team of analysts.

    “We do a lot of things today that we didn’t imagine doing three or four years ago,” Matt Armstrong, who leads the observability and data platform team at Confluent, said in a Druid Summit 2022 interview. “From a [product] perspective, you could think of Druid as being able to power a great product [Confluent Cloud] with a great support experience—but perhaps more importantly—an unconstrained future.”

    It’s Time to Think Big

    We’re already moving to this entirely new way of thinking about moving, analyzing, and sharing data. Streaming technology has opened the door for a plethora of new use cases and products to come to fruition. We believe organizations that view streaming data as a force multiplier and implement a streaming-native approach to analytics will gain a competitive advantage in 2023 and beyond.

    Druid is purpose-built for these new streaming use cases, but it’s also built for a new type of application—one that draws characteristics from both the analytics and transactional database worlds. Industry-leading teams have built mission-critical applications that enable operational visibility, customer-facing analytics, drill-down exploration and real-time decision-making using streaming data and Druid. This trend is now starting to take hold and the next wave of analytics applications is already being built.