devopsy.kinsta.cloud

Tag: azure

  • 5 Best Practices for Using VMs on Azure Cloud

    5 Best Practices for Using VMs on Azure Cloud

    When beginning a cloud improvement/optimization project, an easy place to start is to focus on the initial low-hanging fruit of your virtual machine (VM) environments. This article takes a closer look at these steps, using a Microsoft Azure VM environment as an example.

    Here are five best practices that can be easily achieved when using VMs on Azure cloud.

    1. Properly Size Your Virtual Machines: To maximize performance and minimize costs, it’s important to size your VMs appropriately. You can use the Azure portal to determine the right size for your workloads and then select the right VM size based on the CPU, memory and storage requirements of your application. For example, you can use the Azure VM sizing calculator tool to get an estimate of the virtual machine size based on your workload requirements.

    Let’s say you want to host a web application that requires 2 vCPUs, 8 GB of memory and 50 GB of storage. You can use the Azure VM sizing calculator tool to determine the appropriate VM size based on these requirements. The tool may suggest using a “Standard B2s” VM which has 2 vCPUs, 4 GB of memory and a maximum disk size of 32 GB.

    In this case, you would need to increase the disk size to 50 GB. You can do this by adding a data disk to the VM or using a larger VM size with enough disk space. For example, you could use a “Standard B4ms” VM with 4 vCPUs, 16 GB of memory, and a maximum disk size of 64 GB.

    It’s important to note that the actual VM size may vary based on your workload requirements and the specifics of your application. The Azure VM sizing calculator tool provides a helpful starting point, but you should also perform your own testing and validation to ensure that the VM size meets your needs.

    2. Use Managed Disks: Managed disks provide increased reliability, scalability and security compared to traditional storage accounts. For example, you can use managed disks to automatically scale the disk size and performance as the data grows and to replicate data for disaster recovery.

    For example, to create a 100 GB managed disk, follow these steps:

    • In the Azure portal, go to the “Disks” section and click on “Add.”
    • Select “Blank disk” as the source and specify the disk size as 100 GB.
    • Choose the appropriate storage account type (Premium or Standard), replication options and disk type (HDD or SSD).
    • Give the disk a name and create it in the same resource group as your VM.
    • Once the disk is created, you can attach it to your VM by going to the virtual machine’s “Disks” section, clicking on “Attach,” and selecting the newly created disk. By using managed disks, you benefit from automatic disk management and replication as well as improved reliability and security compared to traditional storage accounts. Additionally, managed disks simplify the process of attaching and detaching disks to and from VMs, making it easier to manage your storage resources.

    3. Enable Monitoring and Diagnostics: To monitor the health and performance of your VMs, you can use Azure Monitor which provides real-time insights into resource utilization, application performance and other key metrics. You can also configure alerts to notify you of potential issues and diagnose problems using Azure Log Analytics. For example, you can set up an alert to notify you if the CPU usage of a VM exceeds a certain threshold.

    You can also use Azure Monitor to gain real-time insights into the virtual machine’s resource utilization, application performance and other key metrics by following these steps:

    • In the Azure portal, go to the virtual machine’s “Monitoring” section.
    • Click on “Diagnostic settings” and then click on “Add diagnostic setting.”
    • Give the diagnostic setting a name, select the VM you want to monitor and choose the metrics and logs you want to collect.
    • Choose the storage account you want to use to store the monitoring data and specify the retention policy for the data.
    • Click “Save” to apply the changes.

    By enabling monitoring and diagnostics for your virtual machines, you can proactively monitor the health and performance of your VMs and quickly identify and resolve potential issues. Additionally, you can use the collected data to generate reports, visualize trends and perform analysis to gain deeper insights into the performance of your VMs.

    4. Implement Security Best Practices: To secure your virtual machines, you can use Azure Security Center to implement security best practices and monitor for potential threats. You can also use Azure Active Directory to control access to VMs and secure your identity management. For example, you can use Azure Security Center to assess the security of your VMs and implement security recommendations such as enabling network security group (NSG) rules to restrict inbound and outbound traffic.

    If you have a VM that you want to connect to other resources in your Azure environment, you can use a managed virtual network to provide a secure and scalable networking solution for your VM.

    Here is how to create a managed virtual network in Azure:

    • In the Azure portal, go to the “Virtual network” section and click on “Add.”
    • Give the virtual network a name, select the appropriate subscription and resource group and specify the address space.
    • Create one or more subnets within the virtual network, and specify the subnet address range and subnet name.
    • Associate the virtual network with a network security group (NSG) to control inbound and outbound traffic.
    • Create a network interface for your VM and associate it with the virtual network and subnet you created.

    By using a managed virtual network, you can create a secure and isolated network environment for your VM and connect it to other resources in your Azure environment. Additionally, you can use the virtual network to control and manage network traffic and implement advanced networking scenarios such as load balancing, VPN connectivity and more.

    5. Keep Virtual Machines Up-to-Date: Regularly updating your VMs with the latest security patches and software updates helps ensure that they are protected against potential threats and vulnerabilities. You can use Azure Update Management to automate the update process and keep your VMs updated. For example, you can schedule regular updates for your VMs to ensure they always have the latest security patches and software updates installed.

    When planning public cloud environments, two commonly used words are “progress” and “results.” Following the best practices above will help position your cloud journey for success.

  • npm is Scam-Spam Cesspool ¦ Google in Microsoft Antitrust Thrust

    npm is Scam-Spam Cesspool ¦ Google in Microsoft Antitrust Thrust

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: The npm registry suffers spam infestation, and Microsoft makes Google sad. (more…)

  • Microsoft Outage Outrage: Was it BGP or DNS?

    Microsoft Outage Outrage: Was it BGP or DNS?

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: All of Microsoft’s cloud services go down, everywhere except China. Redmond’s IaaS, PaaS and SaaS—including GitHub—were dead for several hours, and are still running unreliably—despite Microsoft saying it’s fixed the problem. (more…)

  • Best of 2022: Microsoft Adds Raft of Tools and Azure Cloud Extensions

    Best of 2022: Microsoft Adds Raft of Tools and Azure Cloud Extensions

    As we close out 2022, we at devopsy.kinsta.cloud wanted to highlight the most popular articles of the year. Following is the latest in our series of the Best of 2022.

    At its Build 2022 conference this week, Microsoft added a bevy of tools and platform updates as part of its effort to improve the productivity of developers and the DevOps teams that support them.

    A Microsoft Dev Box offering, for example, provides access to a set of pre-configured environments on the Azure cloud that enables developers to be onboarded in seconds, while a previously announced GitHub Copilot tool that makes use of artificial intelligence to make writing code faster will be made generally available this summer.

    Donovan Brown, a partner program manager in the Azure CTO Incubations team, told conference attendees the overarching goal was to enable developers to stay agile while they continue to work remotely in the wake of the COVID-19 pandemic.

    For DevOps teams, a Smart Maps capability was added to the Application Insights module within Azure Monitor. Smart Maps leverages machine learning algorithms to surface performance bottlenecks or failure hotspots and is also available as a technical preview.

    There is also a tool that discovers and groups dependent Hyper-V virtual machines (VMs) and physical servers to ensure all required components are identified and included during an Azure migration.

    Other updates include an App Service Landing Zone Accelerator that provides a combination of artifacts and documentation to automate application deployment using the Azure App Service 3.0 and preview tools for migrating ASP.NET web applications to the platform. In addition, Microsoft is previewing support for Google Remote Procedure Call (gRPC) on the Microsoft Azure App Service. A .NET Multi-Platform App UI framework allows for building natively compiled applications for iOS, Android, macOS and Windows using C# and XAML in a single codebase.

    Microsoft has also extended the reach of its cloud platform for managing application programming interfaces (APIs). The Azure API Management service now supports the Azure Private Link virtual private network (VPN) along with a capability that automatically renews certificates. In addition, Microsoft is previewing support for the GraphQL query language for APIs.

    In addition, a GitHub OpenID Connect (OIDC) with Azure Active Directory (Azure AD) workload identity federation capability has also been added to eliminate the need to store and access secrets outside of the Azure cloud platform.

    Microsoft, in partnership with VMware, is adding a managed Azure instance of the Spring framework for building Java applications and making previews of the Dynatrace observability platform and NGINX proxy software available as software-as-a-service (SaaS) applications on the Azure cloud.

    Finally, Microsoft has added a raft of tools to make it simpler to build cloud-native applications on Kubernetes clusters.

    It’s not clear how quickly developers are adopting these tools, but DevOps teams can expect to see the rate at which applications are developed on the Azure cloud increase in the months ahead. The challenge, of course, will be managing all those applications—both as they are being built and after they are deployed either in Azure or some other compatible IT environment.

  • Dead Downtown: It’s YOUR Fault | Pentagon’s FOUR Cloud Vendors | Apple Adds MORE Price Flexibility

    Dead Downtown: It’s YOUR Fault | Pentagon’s FOUR Cloud Vendors | Apple Adds MORE Price Flexibility

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: Home working is ripping the heart out of cities, the DoD’s bizarre cloud strategy, and Apple adds a $10,000 app price option. (more…)

  • Three Key Steps To Going Multi-Cloud

    Three Key Steps To Going Multi-Cloud

    Are you considering a multi-cloud strategy for your organization? Getting this sequence of foundational steps right can make a huge difference, both economically and performance-wise. FinOps or a cloud resource optimization solution like Archera, for example, is often brought into the picture after a company has already embarked on a multi-cloud approach and is knee-deep in long-term commitments. These solutions can help organizations avoid—or get further entangled in—problematic costs and contracts. We recommend a three-step approach for organizations that are going multi-cloud. The sequence of steps is:

    Cloud zoning policy: Determine which applications, teams and projects go where—and which cloud they will reside in.

    Architecture: Draw up the blueprint for your multi-cloud infrastructure. This specifies how applications run within a cloud and/or across clouds, particularly focusing on which applications need to be portable between clouds and cannot use vendor-specific services like AWS Lambda.

    Pay for it: Typically this is called the “contracts and commitments” phase, but we strongly encourage putting off decisions on contracts until you have created an overall plan, started to roll out and forecast usage so you understand the overall financial requirements for each platform.

    Step One: Cloud Zoning

    First, is cloud zoning. Your cloud zoning choices will have a major impact on commitments and costs. You need to map out what applications and processes will run on each provider. That entails deciding what can be locked into a single cloud, what needs to be portable between clouds and what applications need to be fully multi-cloud.

    Cloud zoning policy would cover, for example, whether you keep analytics and web serving on separate cloud providers.

    This also involves identifying your special focus areas. For example, if you want to ensure constant uptime even in the rare instances that AWS goes down, you might concentrate to extremes on load balancing among multiple clouds.

    If global availability is absolutely paramount to your business, you would set cloud zoning policies to support that imperative.

    Do not overlook the possibility that different teams need to use different best-in-class tools that may be specific to a provider; for example, an AI team that needs GCP TPUs.

    Step Two: Architect Your Multi-Cloud

    The high-level design and knowing what goes where are the crucial foundations for multi-cloud implementations. Forecasting is an art that comes into play here because you will need to forecast how your utilization of different services will grow.

    Hopefully, you will know how it’ll grow. Where do data science and AI go? Where is production application serving? Where is data warehousing? How will they grow relative to each other?

    The architecture phase is where you determine projects that need to be cloud-agnostic, and which don’t need to be portable and therefore can lean on proprietary CSP managed services.

    In general, in the long term, we see it is cost-advantageous for companies to take a flexible and containerized approach that can work with any infrastructure-as-a-service provider, or build with common standards if using managed services, such as using EKS for orchestration. This ensures not only portability between vendors’ common compute hosting solutions, but also portability within individual vendors’ managed service ecosystems, allowing customers to find an optimal fit for the application needs. For example, a containerized application can run on EC2, Lambda, ECS and a number of other options within AWS alone.

    Forecasting comes into play for specific parameters, including invocations of lambdas, the data storage your computing will need, the number of compute nodes and database utilization.

    Step Three: Financial—Contracts and Billing

    Forecasting continues to be important as you move into the selection of services and preparing for contracts. You need a clear understanding of your flexibility needs as the infrastructure you are consuming changes over time, both today and up to three years out. Then comes the process of matching costs to each forecast to build your cloud consumption budget,

    By studying your forecasts and the amount of uncertainty associated with each of them, you can begin to optimize commitment plans to zero-in on the best blend of contracts to cover the expected future usage.

    Throughout this challenging phase, look for ways to mitigate the risk of the commitments you’ll make. One way to do this is via commitment buy-back guarantees or by taking advantage of more natively flexible commitment options from AWS—such as compute savings plans, which can be consumed by any AWS compute usage in any region—in exchange for much lower savings rates.

    You can avoid confusion and lost time with careful budgeting and accounting. Make sure that commitment costs and savings are attributed specifically to the right applications, not spread like peanut butter across everything, which is often the default (unfortunately).

    Finally, when planning migrations of applications between vendors, interpret carefully between the big three and other providers. The performance semantics of a virtual machine at vendor X usually means something different than a VM at cloud provider Y, even if they appear to have the same specs.

    Going Multi-Cloud: Real Perspective

    In real-life multi-cloud computing where services are billed by the second, it is almost impossible to achieve 100% optimization of costs, commitments and performance. With manual approaches, very few organizations can expect to come close. Some companies do a very good job with attribution and tagging but do not accurately compare the hundreds or thousands of options between the big three providers to understand the optimal strategy.

    It can be extremely challenging, and it’s useful to understand the perspective of Amazon, Microsoft and Google. They respond first and primarily to the technical needs of their customers and often put off servicing their other wants (contracts, cost, transparency). For example, they make VMs spin up faster, but cannot make them less costly.

    Your challenge with going multi-cloud is much greater because you confront complexity by design, which makes optimization grounded in simplifying processes and automating as much as possible indispensable. Optimizing your multi-cloud smorgasbord requires involvement on your part, as well as external, neutral partners that exist to help the hundreds of thousands of organizations that consume cloud services.

  • Microsoft Taps vFunction to Refactor Java Apps for Azure

    Microsoft Taps vFunction to Refactor Java Apps for Azure

    Microsoft has teamed up with vFunction to make it easier to convert monolithic Java applications into a set of microservices that can be deployed on the Microsoft Azure cloud.

    The Microsoft Services Industry Solutions group is making available the Java Refactoring Service, through which IT teams will be able to both lift and shift Java applications to the cloud and then convert them into a set of microservices using agent software and the vFunction Application Transformation Engine, which is infused with artificial intelligence (AI), to analyze Java code.

    vFunction CEO Moti Rafalin said the goal is to eliminate the heavy lifting currently required to shift Java applications into the cloud. The vFunction Application Transformation Engine is required because most of the Java applications that were easy to move to the cloud have been, Rafalin noted. The remaining Java applications, he said, have levels of dependencies that make them too complex to move to the cloud without the aid of a static byte code analysis platform capable of parsing more than 10 million lines of code and thousands of Java class libraries.

    Once those applications are converted into a set of microservices, it then becomes possible to achieve a level of scale that would otherwise be beyond the reach of any on-premises IT environment, he added, noting the same vFunction platform can be used also to help organizations continuously optimize microservices applications as they continue to evolve in a cloud computing environment.

    Longer-term, vFunction is also looking to add support for Microsoft.Net applications that need to be migrated to the cloud, said Rafalin. Many complex Java and Microsoft.Net applications are still running in an on-premises IT environment because organizations are encountering inertia that arises from paralysis of analysis, he noted.

    In general, Microsoft continues to gain momentum in its battle for cloud supremacy against Amazon Web Services (AWS). It’s not clear at what rate Java applications are being shifted to the cloud. However, most monolithic applications have components that would be easier to maintain and support as a distinct microservice. In general, organizations are embracing microservices to build applications that are more resilient. In the event a microservice becomes unavailable, requests for services are rerouted to another microservice to make sure the application degrades gracefully, rather than simply crashing altogether. The challenge, of course, is that over time the dependencies that can exist between hundreds of microservices can be more challenging to manage than those of the monolithic application being replaced.

    It’s not likely that microservices-based applications with entire replace monolithic applications anytime soon, but the bulk of new applications are being built on top of a microservices-based framework. As such, many DevOps teams are now managing a broad mix of monolithic and microservices-based applications across an increasingly complex IT environment. Less clear is to what degree teams will be able to leverage existing DevOps tools and platforms to build and deploy both types of applications at the scale enterprise IT organizations require.

  • DevSecOps in Azure

    DevSecOps in Azure

    DevSecOps is everywhere, and chances are your organization is shifting toward this convergence of development, security and operations. In an on-premises environment, you build your own DevSecOps process by mixing and matching existing and new tools. However, in the cloud, much of the environment is determined by your cloud provider. Microsoft’s Azure cloud is known for its robust security capabilities. Specifically, Microsoft has developed a set of capabilities and services that can help you ease into a DevSecOps workflow without much effort. The tools are pre-integrated and support everything from CI/CD to identity management. Let’s see how they work.

    What is DevSecOps?

    Security is an important part of DevOps. But it can be challenging to ensure that software developed in a CI/CD process is really secure.

    DevSecOps is an organizational pattern that can make this possible, even at high development velocity. It is an approach the DevOps team can use to prevent security issues from the outset of the development life cycle, ensuring their software is not only high quality but also secure and resilient to failure.

    DevSecOps is a discipline that combines application development, security, operations and infrastructure-as-code (IaC) with an automated continuous delivery cycle. Here are three practices that can help a team move towards a DevSecOps process:

    • Focus on average detection and recovery times—These metrics show how long it took to detect a breach and how long it took to recover. This can be tracked through continuous field testing. Improving these indicators should be an important consideration when evaluating potential policies.
    • Defense in depth—Every component in the architecture should be secure so that if and when attackers manage to compromise the system, there are multiple security measures and controls holding them back. This puts an end to the notion of a ‘secure perimeter’ that the organization must defend at all costs. Each component must have its own secure micro-perimeter—this is a zero-trust approach.
    • Continuous learning—Teams should conduct regular assessments of security practices and their environment, especially after security incidents. Every time a security incident is discovered and resolved, the team must evaluate what went wrong in the development life cycle, explore how it enabled the incident and identify ways to improve the process.

    Features and Services That Enable DevSecOps in Azure

    The Azure cloud provides several built-in capabilities that make it possible to adopt a DevSecOps process.

    1. Azure DevOps

    Azure DevOps provides developer services that enable teams to plan tasks, collaborate on code development and build and deploy applications. Azure DevOps supports a collaborative culture and process in which developers, project managers and contributors work together to develop software. You can use Azure DevOps Services in the cloud or Azure DevOps Server on-premises.

    2. Build and Deploy Containers With Azure Pipelines

    You can integrate Azure Pipelines, the Azure cloud’s CI/CD solution, with your Kubernetes clusters. Use the same YAML document to build multi-stage CI/CD pipelines.

    Azure Pipelines lets you track metadata, such as commit hashes and issue numbers, from Azure Boards into container images. This provides direct traceability from any security issue to a specific change made in development. It also provides clear, easy-to-read documentation that can improve the feedback loop between development, operations and security teams.

    3. Run and Debug Containers with Dev Spaces

    When developing Kubernetes applications, there is a need to test applications locally and understand how they interact with dependent services. You may need to develop and test multiple services in collaboration with other developers or teams.

    Azure provides Bridge to Kubernetes, which lets you run and debug code on your development machine while connecting to your Kubernetes cluster. You can test your code end-to-end, set breakpoints on code running on the cluster and share your development cluster among team members. This makes it possible to test and resolve Kubernetes security issues in a realistic environment before deploying to production.

    4. Manage Identities and Access with Azure AD

    The Microsoft Identity Platform takes the Azure Active Directory (Azure AD) developer platform one step further. It allows applications to accept logins from any Microsoft identity, and obtain tokens that can be used to call Microsoft APIs or APIs created by other developers. This creates a large, interoperable ecosystem.

    Azure Active Directory B2C provides B2C identity services—customers get single sign-on (SSO) access to applications and APIs using their preferred social, enterprise or local account ID. Another option is to integrate Azure AD with on-premises Active Directory for hybrid and Azure migration scenarios.

    You can also use Azure Role-Based Access Control (RBAC) to manage access to cloud resources. RBAC lets you manage who can access your Azure resources, what they can do with them, and which regions can access them.

    You can also use the Microsoft Identity Platform to protect DevOps tools themselves, including native support for Azure DevOps and integration with GitHub Enterprise.

    5. Manage Keys and Secrets with Azure Key Vault

    Exposed secrets are a severe security issue that is very common in modern applications. Azure Key Vault lets you manage the distribution of secrets by storing them centrally in Azure Key Vault. Key Vault greatly reduces the chance of accidentally disclosing secrets. With Key Vault, application developers no longer need to store credentials or other sensitive information in their application code.

    6. Azure Policy and Azure Security Center

    Azure Policy lets you specify a default allowed configuration that is automatically applied to all cloud resources. This can avoid misconfigurations that violate security policies. Azure Policy works on the basis of desired state configuration (also known as declarative configuration), letting you specify to what degree resources and services should be secured and whether to alert or block/modify deployments in Azure if they don’t meet the policy.

    When using Azure in multi-tenant mode, policies can be applied at the management group, subscription or resource group level. You can define specific policies and enforce compliance requirements for test, staging, and production environments.

    The Azure Security Center (ASC) and Advisor can also be easily integrated into the DevOps process. ASC makes it possible to augment event streams in Azure with threat intelligence or analysis by third-party security tools.

    7. Penetration Testing

    Penetration testing is the recommended method for examining your environment for infrastructure or application configuration vulnerabilities that could create vulnerabilities that an attacker could exploit.

    An Azure penetration test should check for vulnerabilities on endpoints, apply fuzzing (malformed input) to discover business logic vulnerabilities, and perform port scanning to identify network vulnerabilities.

    Microsoft provides specific guidance for penetration testing in Azure with recommended products and penetration testing service providers.

    8. Configuration and Infrastructure Scanning

    To ensure security for cloud subscriptions and resource configuration across multiple subscriptions, you can use the tenant security solution part of the Secure DevOps Kit for Azure (AzSK).

    In addition, you can leverage Microsoft security technologies like Microsoft Defender for Cloud and Microsoft Sentinel. These solutions provide monitoring and security features designed to detect and alert on unusual events or configurations that require investigation and possible remediation.

    Conclusion

    In this article, I introduced DevSecOps and showed how the Azure cloud provides a range of services that let you easily implement DevSecOps practices. These include:

    • Azure DevOps: The venerable toolset with strong support for DevSecOps environments.
    • Azure Pipelines: The home-grown CI/CD product.
    • Azure AD: A comprehensive zero-trust identity management solution.
    • Azure Policy and Security Center: Enables unified control over security policies for cloud automation and security visibility over cloud workloads.

    I hope this will be useful as you take your first steps to practice DevSecOps in the cloud.

  • Facebook Bans Innocent Users | Azure and ARM/Ampere | ‘Great Resignation’—No End in Sight

    Facebook Bans Innocent Users | Azure and ARM/Ampere | ‘Great Resignation’—No End in Sight

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: No remedy for banned Facebook users, new ARM-based VMs on Microsoft Azure, and The Great Resignation will still be a Thing for the foreseeable. (more…)

  • Database Migration from Microsoft Azure to Snowflake on AWS: Part 1

    Database Migration from Microsoft Azure to Snowflake on AWS: Part 1

    Technological advancements have increased the demand for enhanced infrastructure with quick deployments. Public cloud providers are constantly upgrading their policies and infrastructure to match ever-growing business requirements. This competition gives businesses the ability to choose the cloud provider(s) that best fit specific governance and cost-effectiveness needs.

    In this blog, we discuss migrating databases from SQL Server on Azure VM to Snowflake on AWS. 

    Understanding the Problem

    Microsoft Azure is a cloud computing offering for Microsoft-managed data centers, whereas Snowflake is a cloud-based data warehousing solution that provides software-as-a-service (SaaS) based on various public cloud providers.

    In this scenario, we need to migrate the SQL Server databases to Snowflake. There is also a need for the migrated data to be put in Snowflake with multiple schemas for database names with the data in the precise and correct form. 

    While Azure supports ingestion from various sources and clouds, it did not support direct egress to other cloud providers. Finding a workable solution for moving egress to AWS-based Snowflake was the first challenge. 

    Our Approach to Migration

    According to the Snowflake documentation, Snowflake has a stage feature that could address the above issue. It is, essentially, a path in which the data files that need to be ingested are stored, similar to the concept of SMB Samba mount. Using Snowflake Stage allowed it to load Azure Blob storage, after which Snowflake could read and ingest the data in flat files. The next step was to move data from SQL Server to Azure Blob storage.

    The Migration Process

    The migration process consisted of the following steps: 

    • Replicate the database schema in Snowflake as per Azure SQL database
    • Set up Azure Data Factory pipeline to create parquet snappy format flat files on Blob storage.
      • Use parquet files for data compression and quick data load in Snowflake
    • Create file format in Snowflake
      • Create or replace file format <file_format_name> type = ‘parquet’;
    • Create Stage in Snowflake
      • create or replace stage <Stage_Name>
      • url='<Azure Blob file/folder path>’
      • credentials=(azure_sas_token= <token>)
      • file_format = <file_format_name>;
    • To verify if files are staged
      • list @ <Stage_Name> ;
    • Finally, load data to Snowflake table from Stage 

    (Note that all parquet data is stored in a single column ($1))

    copy into TEST1

    from (select

    $1:CustomerID::varchar,

    $1:NameStyle:name::varchar,

    $1:Title:city.bag::variant,

    $1:FirstName::varchar,

    $1:MiddleName::varchar,

    $1:LastName::varchar,

    $1:Suffix::varchar,

    $1:CompanyName::varchar,

    $1:SalesPerson::varchar,

    $1:EmailAddress::varchar,

    $1:Phone:name::varchar,

    $1:PasswordHash::varchar,

    $1:PasswordSalt::varchar,

    $1:rowguid:name::varchar,

    $1:ModifiedDate::datetime

    from @ <Stage_Name>);

    Let’s break down the details of each step in the process.

    1. Leveraging Azure Data Factory

    Azure’s Data Factory is a GUI-based tool that facilitates an end-to-end ETL solution and provides a step-by-step guide for building the pipeline. It has Source (SQL server), Target (Blob storage) and the necessary settings for tuning the performance of the pipeline. These offerings made it a perfect solution for the customized needs of this migration project—which was to wrangle the data before exporting to Blob storage. This was addressed seamlessly by Data Factory which is covered in detail in the later section of this blog.

    2. Tuning the Performance of Data Factory Pipeline

    The tricky part of this migration was that while the pipeline was easy to build, utilizing its full potential and deriving optimum performance was a challenge.

    There were terabytes of data that needed to be exported to Blob from SQL, which would have taken weeks to transfer without tuning. After adequate POC, it was found that Data Factory could support dynamic range in reading data from the source.

    Let’s say there is a table XYZ that is 800 GB. As per the approach mentioned above, Data Factory is required to move the huge amount of data into Blobs. With the traditional method, the GUI, by default, writes the data to Blob serially which would be slower.

    Now, if we look at the table XYZ with a column “date”, the 800 GB of data can be partitioned into small sets depending on month or year. This would mean that each partition is not directly dependent on other date partitions and can be written in parallel. This will be quicker and more resource-efficient.

    This can be achieved by using the dynamic range filter which can be only applied by writing the select statement rather than selecting the checkbox of the existing tables.

    3. Using Parquet File

    The exported data needed to be stored in a flat file while maintaining integrity and compression. CSV was the first choice but during POC many challenges were faced while writing the file, maintaining the spaces and new line characters which corrupted data. The Data Factory offered the Parquet format of a file that had a great compression rate (75%) and also maintained the integrity of the data. Parquet was optimized to work with complex data in bulk and thus was suitable for this project. With respect to the above figure, it can be seen that  40GB of data was compressed to 11GB.

    4. Integration Runtime

    For the Data Factory to work, it required more compute power which was facilitated in the following ways: 

    • Auto-Resolve Integration Runtime

    As the name suggests, the compute resources were managed and assembled by the Microsoft data centers and the cost was incurred on a per-unit basis. The region of these resources was automatically decided based on availability. This is selected by default when running a Data Factory pipeline.

    • Self-Hosted Integration Runtime

    This runtime uses the resources that already exist. For example, the self-hosted IR allowed downloading a client program on the machine for the resources required and creating a service and coupling it with the Data Factory.    

    4. Setting up the Self-Hosted Integration Runtime

    This was the best available option as the SQL server was already hosted on a standalone Azure VM, which provided the freedom to use the full capacity of resources attached to it. It included the following steps: 

    1. Setting-up Self Hosted IR

    • For Azure Data Factory to work with Azure VM it needed to set up the Azure Data Factory Integration Runtime
      • In Azure Data Factory, select “Manage” and then “Integration Runtimes”
      • Select “+ New”, then “Azure, Self Hosted”, 
      • Next, select “Network Environment -> Self hosted”
      • Next, give a suitable name to the self-hosted IR
      • Once the IR is created, Auth keys will be presented. Copy these keys
    • Now as per the last screen, a link was provided to download Microsoft Integration Runtime
        • Download and install the integration runtime from the Microsoft link
        • Once installed, enter the Auth Key1 value and register “Launch Configuration Manager”
        • When registered, the self-hosted IR will bind with the data factory
        • Now, install the Java Runtime 64 bit as it is required for the self-hosted IR to work. Refer to this manual.

      2. Create Linked Service in Data Factory

    • Proceed to Data Factory 
      • Select “Manage” and then “Linked Services”
      • Select SQL server as type of service and give it a suitable name
      • Now under “Connect via Integration Runtime” select the created self-hosted IR
      • Put in the server name. It is important to note that the server name should be the same name used to connect the SQL server successfully
      • Put in the credentials and test connection

    In the next blog, we shall look at some of the challenges encountered during the migration, cost-saving actions and our approach to data validation.