devopsy.kinsta.cloud

Category: DevOps Practice

  • VMware Backup Guide for Hyper-V Admins

    VMware Backup Guide for Hyper-V Admins

    At first glance, the Hyper-V and VMware virtual environments may seem similar, but upon closer inspection, a number of important differences between these two virtualization platforms become apparent. Initially, performing administration for VMware vSphere could present some challenges for administrators who are familiar with Hyper-V. There are several reasons a business might be switching from Hyper-V to VMware. The company could be deploying a multi-hypervisor environment – e.g., using Hyper-V for Windows-based virtual machines (VMs) while introducing VMware vSphere for VMs running Linux, Free BSD, Solaris, etc. This blog post is intended to help Hyper-V administrators understand the differences and similarities between these two major virtualization platforms, making the transition to VMware easier.

    Comparing Hypervisor Design

    The Hyper-V Role for Servers and Workstations

    Hyper-V users can set up the Hyper-V role not only on the Windows Server operating system (OS) but also on desktop versions of Windows, such as Windows 7, 8 or 10. The installation is similar for both server and desktop solutions. Hyper-V is classified as a type 1 hypervisor, meaning that it is run directly on hardware. There is also a lighter-weight alternative to installing the Hyper-V role on Windows Server: Microsoft Hyper-V Server. Hyper-V Server is a standalone product that is installed on bare metal.

    VMware Workstation for Workstations and VMware ESXi for Servers

    VMware provides VMware Workstation for installation on desktops with Windows and Linux-based operating systems. VMware Workstation is a type 2 hypervisor. A type 2 hypervisor is installed on the underlying OS. As you can see, both Hyper-V and VMware support installation on users’ PCs. If you want to install a type 1 hypervisor on bare metal, you should use VMware ESXi Server. VMs running on VMware Workstation can be migrated to ESXi Server with the VMware vCenter Converter.

    Management Tools

    vCenter as the Analog to SCVMM

    System Center Virtual Machine Manager (SCVMM) is a special universal product created by Microsoft for centralized virtualization management of Hyper-V hosts, clusters, virtual networks and storage. SCVMM is useful for the management of complex multi-site and multi-domain environments.

    Hyper-V Manager is a common tool that is available by default upon installation of the Hyper-V role. With Hyper-V Manager, you can connect to an available Hyper-V host. You can manage the host, the VMs residing on that host, the virtual devices of those VMs and VM replication. You can also create VM checkpoints, start or stop VMs and connect to VMs in order to interact with the user interface of a guest VM.

    Failover Cluster Manager is a tool for managing failover clusters and VMs within a cluster. With this tool, you can create a failover cluster. Failover Cluster Manager is not a substitute for Hyper-V Manager; you still need the latter to manage non-clustered VMs, Hyper-V hosts, virtual switches, etc.

    The equivalent of SCVMM in VMware vSphere is vCenter Server, which can perform the tasks that Failover Cluster Manager and Hyper-V Manager handle in Hyper-V environments. With vCenter Server, you can manage almost all components of vSphere – hosts, clusters, virtual networks (including standard vSwitches and distributed vSwitches), datastores, VM virtual devices, etc. vCenter is installed on a virtual machine, and the vCenter web interface can then be accessed through a browser. You can use vSphere Web Client (which comes in HTML5-based or Flash-based versions) to access vCenter via a web browser. To do so, simply enter the IP address of your vCenter server in the address bar of your browser.

    If you have a standalone ESXi host, you can similarly access that host using a browser with the vSphere client. Enter the IP address of your ESXi server in the address bar of your browser. Note that you cannot create data centers, configure distributed virtual switches or perform VM migrations when accessing a standalone ESXi host with the vSphere client.

    Below, you can see vCenter v.6.5 accessed via browser with vSphere Web Client. If you want to access the VM’s user interface, click Launch Web Console.

    A new tab is opened and you can interact with the guest OS.

     Another way to access the user interface of a VM is using VMware Workstation. For connection to a VM running on the ESXi host from VMware Workstation, click File > Connect to Server.

    Input the IP address of vCenter or ESXi Server, as well as the user name and password required to access that server. Click Connect.

    In the left pane of the VMware Workstation interface, you can see a list of VMs residing on the ESXi host to which you are connected. If you connected to vCenter, you would see the objects such as Datacenters, ESXi hosts and VMs that are managed by vCenter Server.

    PowerShell Vs. PowerCLI

    With the Hyper-V platform, the alternative to using a GUI for management of the virtual environment is using PowerShell. PowerShell is command-line shell that is available on Windows-based systems, allowing users to manage system components. PowerShell provides automation using combinations of cmdlets and scripts.

    In VMware vSphere, you should use PowerCLI as the alternative to a GUI. PowerCLI is a scripting tool built on PowerShell. It extends PowerShell and includes a set of more than 600 cmdlets. You can automate installation and configuration of any features, virtual machines, system patching, etc.

    Clusters

    VMware HA and DRS Clusters Vs. Hyper-V Failover Cluster With PRO

    The Hyper-V virtual environment offers failover cluster functionality. A failover cluster provides automatic failover of VMs from failed hosts to healthy hosts within a cluster. Upon host failure, VMs from the failed host are restarted on other Hyper-V hosts in the cluster. The Performance and Resource Optimization (PRO) feature collects data about resource usage in the virtual environment and generates PRO tips, recommending migration or reconfiguration of VMs for more rational usage of hardware resources. Since Windows Server 2012, PRO was replaced with Dynamic Optimization.

    There are two types of clusters commonly used in VMware vSphere – High Availability (HA) clusters and Distributed Resource Scheduler (DRS) clusters. The HA cluster provides high availability of VMs and carries out automatic VM failover in case of host failure within a cluster. The VMs that were running on the failed ESXi host are restarted on healthy ESXi hosts. The DRS cluster is used for load balancing. vCenter server monitors the usage of hardware resources such as CPU and memory for VMs and ESXi hosts within a cluster. If, after calculations and analysis of these metrics, the DRS decides that some of VMs can migrate from one ESXi host to another (e.g., from an overloaded host to a host with free resources that is almost idle) for more rational resource usage, then migration recommendations are generated. These recommendations can be applied manually or automatically depending on your DRS configuration.

    Power Optimization Vs. Distributed Power Management

    Power Optimization is a feature of Dynamic Optimization in Hyper-V. Power Optimization allows the migration of VMs from Hyper-V hosts with low workloads to other hosts that have enough resources within a cluster. The unused hosts are then powered off in order to save energy. The powered-off hosts are powered back on as soon as their hardware resources are needed for the cluster.

    Distributed Power Management (DPM) is a component of the DRS cluster in the VMware vSphere virtual environment. Similarly to Dynamic Optimization, the DPM executes VM migration from less loaded ESXi hosts to ESXi hosts that have sufficient resources to run the migrated VMs. The freed-up ESXi hosts are powered off (put into a standby state) to save power. When load is increased and the cluster needs more hardware resources again, those ESXi hosts are powered back on.

    Fault Tolerance in VMware HA Clusters

    VMware also provides a useful feature for HA clusters: Fault tolerance. If you use an HA cluster, in the event of ESXi host failure, there is a period of VM downtime between the VM’s failure and the VM being loaded on another healthy host. Using the fault tolerance feature in the HA cluster helps avoid any downtime. This is because a second VM (ghost replica), which is an exact copy of the VM protected with fault tolerance, is already running on another ESXi host within the cluster. The ghost replica is in an inactive state (its network connections and output are disabled) until the primary VM fails. When the host on which the primary VM was running fails, the ghost replica becomes active in an instant, and the network connections of the replica become enabled. Switching to the ghost replica is smooth; you would only notice a slightly higher latency for one packet if you were to ping the IP address of the protected virtual machine. There is no data loss or service interruption if you are using fault tolerance.

    Migration

    Live Migration Vs. vMotion

    There are two types of VM migration for Hyper-V: Live migration and quick migration. Quick migration puts the VM into a saved state, which is similar to hibernation. The VM’s memory is saved to the disk where the other VM files are located. There is a small period of VM downtime once the VM is put into a saved state until it is registered on the new host. Live migration is used for VM migration between Hyper-V hosts (including failover clustering) without losing service availability. The migration procedure consists of different stages. The most important stage of live migration is the iterative transfer of VM memory pages.

    The analog of Hyper-V Live Migration in VMware vSphere is vMotion. With vMotion, running VMs can be migrated from one ESXi host to another without interrupting the VMs and services that are running on the hosts. When the VM migration process from a source ESXi host to a destination host is initiated, the following operations are performed.

    First, compatibility is checked to ensure that the VM can be run on the target host. Hardware resources are reserved on the target ESXi host and the VM process is started. A VM memory checkpoint is created; all further VM memory changes are written to a special area. The data stored in memory before creating a checkpoint is copied to the target VM. The source VM is running and some memory pages are changed (these pages are called dirty pages). They are written to change buffers, which are copied to the target VM and integrated into the memory of the target VM. The process of copying dirty pages is repeated over and over (an iterative process) until there are no memory changes in the source VM’s memory compared to the destination VM’s memory. vMotion then stops the CPU of the source VM, copies the last change buffer, and integrates the memory changes stored in that buffer to the virtual RAM of the target VM. Disk access is discontinued on the source ESXi host and is started on the target ESXi host. vMotion sends a reverse ARP (Address Resolution Protocol) to the physical switch for the purposes of updating the MAC and IP address mappings. The source VM is then shut down and the VM process is deleted on the source ESXi host.

    Storage Migration Vs. Storage vMotion

    Hyper-V Storage Migration can move VM configuration files, checkpoint files, virtual hard disks and mounted ISO files to a new location. Storage migration can be used when the VM must continue running on the same Hyper-V host but must migrate to another storage location. If the VM must be run on another host and placed on another storage location, then live or quick migration must be used along with storage migration.

    Storage vMotion is the vSphere analog of Hyper-V’s Storage Migration. Storage vMotion can perform live migration of VM components such as VM virtual disks (VMDK files) and VM configuration files to another storage (e.g., ESXi local datastore, shared SAN datastore, etc.). First, the VM metadata (including configuration, swap and log files) stored in the VM’s home directory is copied, then the virtual disks are copied.

    Changed block tracking is used to track the changes after the first replication of the source VM’s virtual disk. While the source VM is running, the disk changes are copied in several iterations to the target virtual disk until there are no differences between the source and target virtual disks.

    The virtual disks can be transformed from thick-provisioned disks to thin-provisioned disks or vice versa. During migration with Storage vMotion, the ESXi host on which the VM is running does not change. Storage vMotion is used together with vMotion when the VM must change hosts and datastores at once.

    Storage

    Cluster Shared Volumes (CSV) Vs. VMFS Cluster File System

    Cluster shared volumes are developed to provide simultaneous access to the same disk by multiple Hyper-V hosts in a cluster for reading and writing data. Disk volumes with NTFS or ReFS file systems must be used for CSV.

    VMFS is a virtual machine file system that is optimized for virtual machines and is suitable for clusters in vSphere. A VMFS can be shared by multiple ESXi hosts in a cluster for reading and writing data. This high-performance file system holds files with metadata and has specific locking mechanisms to prevent data corruption caused by writing metadata by multiple hosts concurrently.

    SMB 3.0 Vs. NFS

    SMB 3.0 is a network-sharing protocol developed by Microsoft that can be used for accessing file-level shared storage by Hyper-V hosts in Hyper-V environments as well as Windows environments in general.

    Network File System (NFS) is a sharing protocol developed by Sun Microsystems that provides access to files in file-level shared storage. NFS can be used by different vendors, by different operating systems, and in networks that have different architectures. NFS can be used by ESXi hosts in vSphere for accessing VM files located on shared storage.

    VHDX Vs. VMDK (VM virtual disks)

    VHDX (VHD) is a file format of Hyper-V VM virtual disks. Gen1 VMs can use either VHD or VHDX virtual disk files. Gen2 VMs can use only the VHDX virtual disk file format.

    VMDK is an open file format of VMware VM virtual disks developed by VMware.

    Dynamic Disks vs. Thin-Provisioned Disks

    Dynamic disks are dynamically expanding virtual disks for Hyper-V VMs. The main purpose of using dynamic disks is to save storage space. When Dynamic Disk is created, not all provisioned space is occupied by the virtual disk file. Initially, the virtual disk is a small file. It grows as data is written to the disk.

    Thin-provisioned disks in the VMware virtual environment are similar to Hyper-V’s Dynamic disks. A thin-provisioned disk has a minimal size upon creation and grows as needed as data is written to the virtual disk.

    Checkpoints Vs. Snapshots

    A Hyper-V checkpoint is used to save the state of a virtual machine at a given point in time. You can create a checkpoint, work with a VM, and then revert the VM back to the state it was in when the checkpoint was created.

    The VMware snapshot is the analog to the Hyper-V checkpoint; it, too, is used to save the current state of a VM. You can take a snapshot at any time, e.g., before updating software configuration, updating installations, testing new software, etc. A snapshot can be taken even while a virtual machine is running. The snapshot captures the following VM data: Virtual disk states, virtual memory content, and VM settings. The following files are created and stored in the VM directory when a snapshot is taken: .vmdk, -delta.vmdk, .vmsn, and .vmsd.

    Pass-Through Disks Vs. Raw Device Mapping

    A pass-through disk is a physical disk that is mounted to a VM and used instead of (VHD/VHDX) virtual disks in Hyper-V environments.

    Raw device mapping (RDM) is an alternative to the use of virtual disks for accessing the VM disks in vSphere. Depending on your settings, you can let virtual machines access the physical storage device directly when RDM is used. RDM can be used in virtual compatibility mode or physical compatibility mode.

    Virtual compatibility mode provides full virtualization of the mapped device; snapshots can be used.

    In physical compatibility mode, the guest OS can access the hardware directly; minimal SCSIvirtualization is provided. If a VM snapshot were taken, the RDM disk would not be included in that snapshot.

    For example:

    Suppose DiskName.vmdk is a disk heading file.

    DiskName-rdmp.vmdk is the mapping file (as evidenced by the -rdmp suffix).

    A mapping file resembles a symbolic link to a mapped device.

    RCT Vs. CBT

    Resilient Change Tracking (RCT) makes it possible to copy only those data blocks that have changed since the last backup when an incremental backup of Hyper-V VMs is performed. RCT technology was released with Windows Server 2016. Files with .RCT and .MRT extensions are created when RCT is enabled.

    Changed Block Tracking (CBT) is the vSphere analog of RCT for Hyper-V, helping perform incremental backups. CBT allows backup software to track the blocks that were changed since the last backup and copy only the changed blocks. Files with .CTK extensions are created for each VMDK virtual disk and snapshot file. Block changes are tracked on the virtualization layer. The VMware vSphere API for Data Protection allows third-party software to access the CBT feature for the purposes of performing backups and replications more efficiently. Changed Block Tracking was introduced by VMware in 2009 with ESX 4.0.

    Storage Spaces Vs. vSAN

    Windows Storage Spaces is a technology that abstracts storage from physical disks, used for storage virtualization and provisioning. With Storage Spaces, you can create cost-effective, scalable, highly-available software-defined storage for Hyper-V virtual environments by using servers with HDD devices.

    VMware vSAN (Virtual Storage Area Network) is the equivalent of Windows Storage Spaces. vSAN provides a distributed concept of data storage that is integrated with vSphere at the cluster level and abstracted from hardware disks. ESXi hosts managed with vCenter must be used to deploy vSAN. This approach allows disks of ESXi hosts to be used for vSAN storage while ESXi hosts are used to run VMs. SSDs can be used for caching data, while HDDs are used to store data. Disk groups are aggregated into pools that are accessible for the entire cluster. The data can be mirrored on one or more hosts, which ensures protection against failure. This is defined by the number of failures setting. Using vSAN, you can achieve reliable and scalable enterprise-grade storage.

    Dynamic Memory Vs. Virtual Memory Overcommit

    Dynamic memory allows a VM to use the amount of memory that it currently needs, rather than all virtual memory provisioned for that VM. The hardware RAM on Hyper-V servers can be used more rationally with this feature.

    VMware Virtual Memory Overcommit is the equivalent feature in VMware. Both features use dynamic memory allocation methods, but the methods differ somewhat. With VMware, you can allocate more memory than the ESXi host has for VMs, thereby increasing the density of VMs on the host. The idea is to optimize physical memory consumption – thus, more VMs can run on the ESXi host, but there is no guarantee that all the memory provisioned for it can be provided to a particular VM at a given point in time. Using Memory Overcommit presumes that a memory can be rearranged between the VMs depending on demands. 25% of unused memory is reserved on an ESXi host in case the memory requirements of its VM suddenly increases.

    Hyper-V Integration Services Vs. VMware Tools

    Hyper-V “Integration Services” is a set of tools and drivers for providing proper functionality of child VMs and time synchronization. When Integration Services are installed, the standard drivers for mouse, keyboard, disk controllers, and network controllers are replaced by drivers included as part of the Integration Services suite for better performance and user experience.

    VMware Tools is a collection of drivers, modules, and utilities that is the VMware equivalent of Hyper-V Integration Services. VMware Tools offers advanced functionality to improve performance, user interaction with guest VMs, and integration. When VMware Tools is installed, you can shut down the guest OS directly from menu in the vSphere client or VMware Workstation. Video resolution and graphics performance are increased. Other useful features like Drag & Drop, shared folders, time synchronization, copying/pasting between VMs and hosts, and grabbing/releasing of the mouse cursor also become available. VMware Tools is usually provided as an ISO image or Operating System Specific Package (OSP).

    Migration from Microsoft Hyper-V to the VMware vSphere Platform

    Transitioning from Hyper-V to vSphere can be useful if the vSphere virtualization platform better meets your company’s needs. First, you should prepare the hardware and software. Check VMware’s hardware compatibility list to ensure that your hardware is officially supported. Install ESXi servers, vCenter, and other software that you need. Download and install VMware vCenter Converter. Run VMware vCenter Converter and click Convert Machine.

    Select Hyper-V Server as a source type. Select the IP address or server name of your Hyper-V Server. In this walkthrough, vCenter Converter is installed on the Hyper-V Server and localhost is input as the server name.

    Next, input the username and password of an account that has sufficient permissions to perform the operation. The Windows Administrator account is used in this example. Click Next.

    Select the virtual machine to be converted. Click Next.

    Select VMware Infrastructure virtual machine as the destination type. Type the IP address of your vCenter Server or ESXi host, then enter the username and password. Click Next.

     

    Select the folder and the name that should be applied to the destination VM after conversion. Click Next.

    Select a destination location, including the ESXi host and datastore. Click Next.

    Configure the options for the destination VM. You can see how to change the type of virtual disk provisioning in the screenshot provided below. Click Edit in the Data to copy section.

    In the dropdown list, select the disk provisioning type (thick or thin). Click Next.

    Check the summary and, if you are satisfied, click Finish.

    Wait until the conversion task is complete.

    Once the VM conversion is finished, you can open vCenter with vSphere Web client in your browser and find the converted VM. In this walkthrough, the VM name is Server2016-01. As you can see on the preview screenshot, the VM is successfully running. Click Launch Web console to access the user interface of the VM. You can now tune the virtual machine and install VMware Tools.

    You now know how to migrate from Hyper-V to VMware vSphere.

    Conclusion

    Knowing the differences and similarities between features of VMware and Hyper-V environments makes migration from one platform to another easier and is particularly useful for administrators of a multi-hypervisor environment. Understanding the architecture as well as the working principles of both virtualization platforms, along with their feature sets, helps administrators work comfortably and effectively when they are tasked with maintaining the less-familiar platform. There is one important rule that applies to both VMware and Hyper-V environments: The VMs must be backed up regularly. Robust VM backup can protect your business-critical data against natural disasters, technological failures, human errors and virus attacks.

  • DevOps, Drought and Climate | Meta❤️PTP

    DevOps, Drought and Climate | Meta❤️PTP

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: Data centers cause climate change, and Meta is rolling out Precision Time Protocol. (more…)

  • Culture a Stumbling Block to DevOps, DevSecOps

    Culture a Stumbling Block to DevOps, DevSecOps

    Organizations are struggling to improve DevSecOps practices as they confront a lack of cultural alignment and investment, according to a global Progress survey of more than 600 IT, security, application development and DevOps decision-makers.

    Many companies are behind in achieving their DevOps and DevSecOps goals, with 73% of organizations saying they could be doing more.

    More than three-quarters (76%) of respondents acknowledged they need to be more strategic about how they manage DevSecOps, and 17% said they still consider themselves at an exploratory and proof-of-concept stage.

    Prashanth Nanjundappa, vice president of product management at Progress, said the most surprising finding was the evident contradiction between what organizations think is very important and their actual alignment, commitment or investment. 

    “While security is the number-one driver behind most DevOps and DevSecOps implementations, just 30% said they felt confident in the level of collaboration between security and development,” he added. 

    Meanwhile, 86% of respondents are experiencing challenges in their current approaches to security and 51% admitted that they don’t fully understand how security fits into DevSecOps.

    Nanjundappa pointed out that company culture is, in fact, biggest barrier to DevOps and DevSecOps success. 

    “We found 71% of respondents agreed that culture is the biggest barrier to DevSecOps progress, yet only 16% were prioritizing culture as an area to optimize in the next 12 to 18 months,” he explained. 

    The study revealed the top business factors driving the adoption and evolution of DevOps inside their organizations included a focus on agility as well as reducing the business risk of quality, security and downtime or performance issues.

    Organizations also recognized the need to implement DevOps to support a cloud mandate or their move to the cloud.

    In addition, concerns about public cloud workloads, increasing regulatory requirements and concerns about financial impacts are driving how security around DevOps is approached. 

    “Organizations are juggling lots of priorities in how they manage security with key focus areas for security being digital marketing efforts, employee apps and customer-facing apps,” Nanjundappa said. 

    Challenges with security include prioritizing the development of external-facing capabilities over internal apps, the ability to secure different workloads using different development and delivery methods, meeting delivery deadlines and difficulties meeting audit requirements.

    “Through the survey, we have seen some of the differences that stand out between organizations that have achieved higher maturity versus others,” he noted.

    Some are related to cultural aspects and bring to the fore a focus on business agility through fast and frequent delivery of application capabilities, as well as using DevOps to better manage distributed and/or remote work environments.

    Nanjundappa said among the best practices to ensure fruitful collaboration are stakeholder sponsorship on leadership for tools and outcomes aligned with the strategic direction of organizational goals.

    “Let teams innovate at the grassroots level with regard to tools and processes to accomplish expected outcomes,” he explained. “Identify teams which have seen success, celebrate those successes and implement those processes across other teams in an iterative model.” 

    As organizations head into 2023 with a tight IT labor market persisting, Nanjundappa pointed out a key concern will be to find the talent they need to make these DevSecOps initiatives a reality.

    “While it is ideal to hire pre-trained talent which fits their organizational needs, it will either be expensive or sometimes close to impossible for multiple reasons,” he said. 

    He recommended using a multi-pronged approach to bridging talent gap, which includes an on-ramp for hiring experienced key talent and reskilling through cross-pollination of experiences from different teams, including security, dev and ops. 

    Nanjundappa also advised upskilling existing teams with modern technologies, replacing legacy platforms and tech with cloud and cloud-native technologies and augmenting that with the right level of tools and processes, including policy-as-code and CI/CD. 

  • Five Great DevOps Job Opportunities

    Five Great DevOps Job Opportunities

    DevOps.com is now providing a weekly DevOps jobs report through which opportunities for DevOps professionals will be highlighted as part of an effort to better serve our audience.

    Our goal in these challenging economic times is to make it easier for DevOps professionals to advance their careers.

    Of course, the pool of available DevOps talent is still relatively constrained, so when one DevOps professional takes on a new role it tends to create an opportunity for others.

    The five job postings shared this week are selected based on the company looking to hire, the vertical industry segment and, naturally, the pay scale being offered.

    We’re also committed to providing additional insights into the state of the DevOps job market. In the meantime, for your consideration:

    Indeed.com

    Gridiron IT
    McLean, Va.
    DevOps Engineer
    $195,292 to $205,031

    Dice.com

    Odyssey Information Services
    Remote
    Senior DevOps Engineer
    Up to $150,000

    ZipRecruiter

    K&L Wine Merchants
    San Francisco
    DevOps Engineer
    $125,000 to $150,000

    LinkedIn

    Compassion International
    Remote
    DevOps Specialist – Splunk
    $111,930 to $139,910

    SimplyHired

    Zoom Video Communications
    Virginia
    DevOps Engineer
    $97,920 to $181,170

  • Palo Alto Networks Buys Cider Security to Lock Down Pipelines

    Palo Alto Networks Buys Cider Security to Lock Down Pipelines

    Palo Alto Networks this week extended its efforts to secure application environments by agreeing to acquire Cider Security, a provider of a platform for securing continuous integration/continuous delivery (CI/CD) platforms, for approximately $195 million in cash.

    The acquisition of Cider Security, scheduled to close this quarter, will extend the reach of the company’s Prisma Cloud platform that was updated last year to include a set of tools for securing infrastructure-as-code (IaC) used to provision IT infrastructure. Palo Alto Networks previously acquired Bridgecrew, which had developed an open source Checkov policy-as-code tool. Last month, Palo Alto Networks added software composition analysis tools to Prisma Cloud, as well.

    Cider Security developed what it described as an operating system for application security. The solution creates a graph that enables DevOps teams to visualize the relationships between all elements that make up a software development environment, including code. It then makes it possible to apply a set of controls to remediate any vulnerabilities and attack paths that might be identified using any number of third-party scanning tools.

    Palo Alto Networks CEO Nikesh Arora told industry analysts the acquisition is an example of the company doubling down on securing software supply chains alongside its existing portfolio of platforms and services for securing production environments.

    It’s not yet clear who is in charge of DevOps platform security, but increasingly there will be some type of central security function that works with application development teams to lock down software supply chains, said Mike Rothman, general manager for Techstrong Research, an arm of the parent company of DevOps.com. “There’s going to be a central security group focused on securing the pipeline,” he said.

    As Palo Alto Networks continues to extend the reach of Prisma Cloud left, it continues to make a case for centralizing the management of cybersecurity through a portfolio of platforms that can be centrally managed via the cloud. It’s not clear how much organizations are centralizing management of security across their software supply chains and production environments, but Arora noted that interest in consolidating security vendors is high as organizations look to reduce the total cost of cybersecurity.

    In the meantime, organizations of all sizes are looking to employ DevSecOps best practices to better secure software supply chains in the wake of a series of recent high-profile breaches. In addition to scanning code for vulnerabilities, many of those organizations are starting to realize the tools and platforms used to build applications are also vulnerable to cyberattacks. The goal of those attacks is to insert malware that will manifest itself in any number of downstream applications that are eventually deployed in a production environment.

    It’s not yet apparent just how compromised those tools and platforms may be, but it’s clear software consumers are moving toward holding developers more accountable for vulnerabilities. The best-known example of those requirements is an executive order issued by the Biden administration that will require federal agencies to include software bills of materials (SBOMs) that list known vulnerabilities. Many enterprise IT organizations are expected to adopt similar requirements.

    One way or another, however, the tools and platforms that make up a software supply chain, along with the code they produce, are going to be subject to much greater scrutiny than ever before.

  • How a Mature DevOps Practice Can Strengthen Competitive Advantage

    How a Mature DevOps Practice Can Strengthen Competitive Advantage

    Saman Pourkermani, practice lead, operations effectiveness, Inspirant Group, co-authored this article.

    Software development and delivery has become an integral part of every aspect of a business and is no longer simply a support function. Nowadays, most of us, including those in business operations, know the anxieties of having to wait for specific environment release dates to deploy code for urgent fixes or updates to Dev, QA, UAT or production, which slows down productivity. Or worse, an environment has gone down with no backup environment. We all know systems have to be in place to ensure quality and security, but what makes some companies more successful than others when it comes to their DevOps operations?

    Any organization that builds and delivers software applications and digital services can benefit fromDevOps. The main difference is the maturity of their DevOps culture and practices. Companies with mature DevOps practices and cultural philosophies are not only able to build and deliver applications faster than those with traditional DevOps practices, but also learn and innovate faster.  Mature DevOps capabilities, tooling, and cultural philosophies enable faster innovation by automating and streamlining development, testing, deployment, and infrastructure management processes. The ability to outpace others from both a delivery and learning standpoint is a huge competitive advantage.

    Most organizations understand this and as a result, have been heavily investing in maturing their DevOps capabilities and practices. However, most struggle to make meaningful progress to catch up to leading companies.

    Why is That?

    The most obvious reasons are resource constraints–budgetary, human and/or systems. Many companies have fixed budgets and must balance the tradeoffs between setting up an expensive infrastructure (equipped with yearly upgrades) with other business needs for short-term/long-term benefit gains. 

    We believe a company’s DevOps maturity is directly associated with and significantly impacted by how well and disciplined they are at making business prioritization and tradeoff decisions, picking what they will focus/work on and what they will not – ideally leaning out how much work they commit to completing.

    It goes without saying that most companies that want to but don’t yet have mature DevOps practices have more work than they can handle, from setting up and documenting CI/CD processes to creating business continuity and disaster recovery plans to ensuring they have the right people for execution. These critical elements take time, effort, expertise, and budget to put together. While everyone wishes their teams could get it all done in a timely manner, the reality is that everyone has a finite amount of budget and capacity.

    Quite often, progress on DevOps capabilities and automation gets sidelined in favor of ongoing customer delivery work. The trick is to establish a mechanism for balancing teams’ capacity allocation so that it includes a healthy mix of work focused on becoming more effective and efficient at delivery (e.g., maturing DevOps capabilities, improving architecture and design, etc.) and work to fulfill customer needs. In this way, a percentage of a team’s capacity can always be devoted to advancing DevOps delivery capabilities while the rest can be used to address customer requests.

    How Can This be Accomplished? 

    Start With Culture. Culture is a critical factor when it comes to having a mature DevOps practice. Team members must fulfill their individual roles but also operate as a cohesive unit. The mindset of each team member must be focused on the greater good of the whole – not individuals. Has the team established a way of working together that fosters accountability along with encompassing growth and innovation?

    Focus on Process. Progress cannot be made without identifying a clear, sensible process. For example, having a well-documented and executed CI/CD process is paramount to DevOps maturity to address all developer pain points and ensure a smooth deployment process. The key is having a documented process that establishes the team’s way of working and addresses the following: Are the right team members engaged? Are there special circumstances that need to be accounted for during the development/configuration/deployment processes? Does everyone have the same understanding of the process at each stage?

    Automation and Tools. Have your processes been automated in a way that allows the DevOps practices to happen in the same way each time, without manual intervention? Do you have and know how to use the fundamental tools that all successful DevOps practices need (e.g., build and test code continuously, source control (e.g., GitLab), environment configuration management, performance measurement (e.g., load balancing))? Are deployment pain points adequately identified and addressed to prevent coding/release bottlenecks?

    Business Prioritization and Trade-off Decisions. We believe this is the most important aspect of becoming a mature DevOps practice because it allows for progress over time. Balancing the teams’ capacity to include advancing DevOps delivery capabilities is critical for establishing and sustaining a competitive advantage. Optimizing your DevOps capabilities allows you to free up and capitalize on the time, talent and performance of your team members for learning and innovation.

    As Jack Welch once stated, “The only sustainable competitive advantage is to learn faster than your competition and to be able to act on what you have learned.” When a software organization becomes too rigid without an adaptive DevOps edge, it will be left in the dust. 

  • Avoiding Developer Burnout With Clean Code Best Practices

    Avoiding Developer Burnout With Clean Code Best Practices

    Employee burnout is common in the tech industry, especially for developers. In fact, 83% of software developers feel burnt out from their work and of those, nearly half of them feel overworked. Often, this is because the processes currently in place at their companies are not effective and too much time is spent doing mundane work.

    Resolving coding errors is a necessary part of software development, but this isn’t always an easy chore. Given the complexity of code today, developers do tend to spend a large percentage of their bandwidth on maintenance issues, refactoring, debugging or just modifying code that they never wrote in the first place. That said, 44% of developers stated that fixing software defects and problems is their biggest frustration.

    Every developer wants to create good code, but repetitively dealing with issues that aren’t part of the code they write steals away time from doing interesting work. More than a third of developers claim that they would prefer spending more time with their families, participating in more physical activity and even getting a full night’s sleep than spending the majority of their time fixing code. Developers also stated that they could use the time at work to focus on new features and capabilities.

    By implementing clean code best practices across the organization, development teams can create an environment that prioritizes new and interesting projects while organically addressing errors. Clean code practices help keep developers engaged and interested.

    What is Clean Code?

    Every organization relies heavily on software and the source code that powers it to run successfully. Companies are aware that the source, or DNA, of their software is what counts as its most important asset. The behavior and functionality are both governed by the source code and if it’s not kept in good condition, the source code can turn from an asset into a liability.

    Clean code refers to a codebase that has reached a flawless and problem-free state. It is code that is maintainable, reliable, secure, readable and embodies other such attributes of code. While developers write clean code for their new projects from the beginning, they simultaneously address existing issues through set quality standards. The overall workload of security and development teams also is greatly reduced when clean coding practices are considered and integrated into the development workflow.

    With this method, fewer issues reach the final stage of inspections. It also avoids pricey post-production rework or drawn-out feedback processes. The outcome is a simplified and effective method for managing the state of your code.

    How Clean Code Helps Developers

    By giving your developers access to the power of clean code practices, organizations can make sure that their development teams are working on projects that are impactful and exciting rather than dealing with mundane repetitive work. This means that they have more time to research, put new ideas into action, and also be part of new initiatives that have a direct impact on their organization’s success.

    When using the clean-as-you-code approach, you create a precedent that, moving forward, new code that is added or updated won’t result in new problems. As new updates are made to old code, the overall quality of that code progressively improves. This is a much more manageable process and a huge step toward improving the overall quality of code.

    Additionally, the clean-as-you-code approach empowers developers to take charge of the quality of the code they deliver. The feedback loop to detect and fix issues is naturally shortened with the proper tooling they can use in their current DevOps workflow. And contextualized education helps them quickly understand the root cause and get guidance to fix them. When their everyday routines automatically incorporate this technique, the possibility of recurring or exposed weaknesses is greatly reduced. In the long run, this helps save development time and effort, making development so much more enjoyable.

    There are many things that are a necesssary part of a developer’s everyday responsibilities. Making it simple and efficient is a win-win for everyone. No matter the state of your codebase, the maturity of your development team, or the size of your organization, clean code best practices are easy to embrace and will make your developers happier and more satisfied at work.

  • Data Centers IN SPAAACE | Discord GDPR Fine | AWS Fires Dead Wood

    Data Centers IN SPAAACE | Discord GDPR Fine | AWS Fires Dead Wood

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: The EU wants to put servers in orbit, a GDPR penalty for Discord, and Amazon has the hatchet out. (more…)

  • Visually Staffing Your Developer Teams

    Visually Staffing Your Developer Teams

    Effective collaboration within developer teams is key to creating and improving any product, so who you choose to put on the team—and how it’s structured—can make a huge difference. There are many considerations that go into these decisions, and the challenge is even greater thanks to the widespread movement to a hybrid work environment. To make the process a bit easier and more transparent and to set teams up for success from the outset of each project, it can be helpful to use a visual approach to staffing. 

    Considerations When Assembling Your Team

    When a general manager is assembling a team in professional sports, they must take into account personalities, abilities and leadership styles to make sure the team functions smoothly and can win games. There are similar considerations necessary when putting together a developer team.

    For example, a team must have the right balance of expertise. Having a team that is loaded with frontend experience but little to no backend experience will likely prove problematic. Also, a team full of interns or entry-level engineers with few or no senior engineers won’t have the leadership and expertise needed to succeed.

    Coding styles are also a major consideration when assembling teams. Some engineers may be “cowboy coders”—coders who aren’t afraid to take risks when developing a product. Other engineers, however, may suffer from “analysis paralysis,” spending too much time making sure the code is as perfect as possible. It’s important for teams to have a mix of these styles so the development process doesn’t lean too far in either direction and so the overall team has the right balance for the project at hand.

    Using Visual Collaboration to Assemble Teams 

    As a development team leader, I’ve found staffing our teams using a visual approach—and visually collaborating within those teams—is invaluable. It makes it much easier to ensure we have the right expertise and coding style on the right teams while managing the fast-paced changes that are a reality in today’s business environment. And with the staffing for each project visualized, leaders and team members can easily see who leads the projects and how many people are on each team within those.

    Managers or project leads should be able to see all of the relevant factors of staffing before making an informed decision on who should be on the team or if they need additional support. To begin visually staffing your teams, start with a diagram that—in addition to displaying which team each person is on—clearly shows each individual’s responsibility and flags if they’re new to the team. When decision-makers are able to see each of these factors at a glance, it helps them quickly determine if a team has a gap and needs additional support. 

    Caption: Here, team leaders can see team breakdowns, responsibilities and if someone is new to the team (Lucidchart)

    To make visual staffing even more effective, you may want to get more granular in detail when viewing team structures. This could include labeling team members with relevant attributes such as title, seniority, location, etc. Therefore, if a team needs a senior engineer or someone who’s worked on similar projects, it’s possible to quickly drag their name on your diagram and place them on another team. With this method, teams can see how a new assembly of talent can work together and how this adjustment affects other teams. 

    Caption: Team leaders can identify personnel, job title and the reporting structure (Lucidchart)

    We’re also able to diagram reporting structures. It’s important to know to whom each person reports and how many direct reports each leader has. This helps prevent burnout and ensures each manager has the bandwidth to lead their direct reports and be in touch with the needs of the team.

    Once teams are created, work sessions, such as program increment (PI) planning meetings, will also benefit from a visual collaboration solution such as a virtual whiteboard. Before the recent predominance of hybrid and remote work, many planning meetings included in-person discussions in front of a physical whiteboard. Also, voting on decisions included physical, color-coated sticky notes which each person would place on the whiteboard to indicate their vote. 

    Caption: Developer teams can participate in virtual PI planning meetings for more seamless collaboration. (Lucidspark)

    Using visual collaboration will help not only recreate this experience in a hybrid environment but also make it more effective and efficient. For example, working with a virtual whiteboard solution could improve a team’s experience with common project management tools, such as Jira. The right visual collaboration solution can allow teams to pull stories and display the latest project information for everyone to provide input directly within a collaborative space. The result is an ability for team leaders to create high-level plans with both real-time collaboration and asynchronous input from teams around the world—all while massively reducing the slow, manual steps that are so common with collaboration in front of a physical whiteboard. Those slow manual steps could include transferring stories from project management software onto physical cards, moving those cards around as discussions continue and then—once decisions are final—manually uploading new information back into Jira or another project management tool. With the amount of dependencies on developer teams today, removing these steps in favor of more efficient processes is invaluable. 

    Visually Staffing Your Teams 

    Staffing teams and enabling collaboration can be a smooth experience, even in a hybrid environment. I’ve found that visualizing how teams work together provides the clarity needed to make the best decisions. It also allows easy changes to team composition should it be necessary. Visually staffing teams allows team leaders to start developers off on the right foot and consistently work toward project goals from start to finish.

  • Infrastructure Abstraction Will Be Key to Managing Multi-Cloud

    Infrastructure Abstraction Will Be Key to Managing Multi-Cloud

    There’s no doubt we’re now living in a hybrid multi-cloud age. Companies are using major cloud providers like AWS, Azure and GCP with niche cloud computing providers emerging within the market. Simultaneously, many organizations still retain workloads in on-premises data centers or embrace colocation. Regardless of deployment pattern, DevOps teams desire the same elasticity as the cloud, but it’s becoming increasingly difficult to consistently manage the nuances of each service provider.

    To support today’s distributed polyglot architectures, a common layer could ease issues when deploying applications and provisioning various servers. By decoupling the infrastructure control plane from specific cloud providers, a unified management console could bring visibility into cost and performance, empowering CloudOps with knowledge before they commit. But engineering such an outcome would rely on plugins for each environment and a robust set of APIs to enable programmatic control.

    I recently met with Jake Warner, Cycle.io, to explore how increased infrastructure abstraction can solve the above problems. According to Warner, a standard layer for computing management is needed to enable consistent control over multi-cloud and bespoke on-premise environments. Below, we’ll define this concept and consider the benefits infrastructure abstraction could bring to modern enterprise software development.

    The Problems Facing Infrastructure Management

    We like to think that the cloud is pervasive and that everyone’s on board with a similar computing style. However, this is far from the truth. Multi-cloud is rising, and 95% of businesses are making multi-cloud a strategic priority in 2022, a Valtix study found. Yet only about half of respondents felt they had the appropriate tools and skills required to execute a multi-cloud strategy.

    “There’s a lot of value in multi-cloud, including reducing risk, ensuring better uptime and the flexibility to select the best hardware for the use case,” said Warner. “But multi-cloud is extremely complex.” Using multiple clouds means supporting increasingly variable operational processes for each environment. Whether it’s AWS, GCP, Azure, Equinix or Vultr, operators also require a deep level of visibility into telemetry data. This becomes complicated to track as software is increasingly modular and distributed across multiple computing regions.

    Furthermore, on-premises data centers are far from dead—surprisingly, a Spiceworks study found that 98% of enterprises still used on-premises servers. Traditional servers are often preferred due to corporate restrictions or to meet security compliance mandates. It could also be a cultural case of “if it works, don’t fix it.” The dilemma is that you can’t easily obtain common cloud features like auto-scaling and load balancing while on your own infrastructure.

    Another issue Warner described is that traditional platform-as-a-service (PaaS) companies present inherent limitations if you aim to build substantive applications and services. “Many of these PaaS offerings make it really easy to get started but, more often than not, developers are quickly forced to make compromises with their approach. These compromises can create long-term technical issues, especially when it’s time to scale,” he said. “Need a server with a specific hardware configuration in a targeted region? Good luck. For most PaaS companies, their exit ramp is just as defined as their on-ramp.”

    Movements Toward Infrastructure Abstraction

    We’re already seeing movement within the market to manage the operational burdens of multi-cloud. One example is the Infrastructure Abstraction Layer (IAL), open sourced by Cycle.io, which Warner explained can act as a standardized system for deployments. With the specification, an infrastructure provider can generate an API endpoint to enable the provisioning of infrastructure without requiring a deep integration between both services. With this endpoint live, an application development platform or orchestrator gains the ability to send standardized JSON payloads to provision servers, allocate IPs, configure regions and a number of other requests. The response, another JSON payload, confirms that the desired request has been completed and includes a unique ID.

    With more programmatic control over infrastructure, an engineer could interface with a standardized list of all the servers, across a number of providers, to deploy to and act upon this information in one unified console. This layer is an opportune location to help compare costs and server configurations for different regions too. It’s also an advantageous spot to filter by proximity to showcase the closest available servers to optimize delivery.

    Benefits of Unifying Multi-Cloud Deployment

    Decoupling the infrastructure management layer from service providers can bring potential benefits to DevOps, said Warner. For one, it enables operators to extend infrastructure automation into their private data centers, helping them achieve the same programmability they would have in the cloud. A degree of standardization can remove many of the cons of multi-cloud, too—”Developers can move from one provider to another provider without any major changes along the way,” he said.

    Of course, IT doesn’t want an added burden. This is why an API-based solution is required to enable programmability, said Warner. By taking a RESTful API approach to managing a hybrid infrastructure, operators could create custom flows based on fine-grained environmental characteristics that change over time, like latency, expense or location. “One of the neat things is it allows infrastructure providers to constantly make tweaks and improvements without having to resubmit their integration with an underlying platform,” said Warner.

    The Flexibility of LowOps

    Today’s software deployment is increasingly wide and varied—a recent HashiCorp report found that 76% of organizations are already multi-cloud. Multi-cloud brings many advantages, yet it also multiplies the number of operational tasks required. At the same time, organizations want to leverage investments into on-premises data centers for as long as they are economically viable. Automating DevOps for such a varied infrastructure can be difficult without a unified layer decoupled from each niche computing area.

    According to Warner, companies require a LowOps platform and the flexibility to abstract this automation on their own infrastructure when appropriate, which can vary widely. For example, operators may be working with a fleet of tens of thousands of IoT devices. Furthermore, as we’ve covered previously, a LowOps method can be beneficial to small-to-medium-sized companies that may lack the resources to support complex infrastructure like Kubernetes.

    Could something like IAL ever become an open standard? Well, at the time of writing, the IAL specification has only been implemented by Cycle.io and a few of its infrastructure partners, but there’s no reason it couldn’t be extrapolated into a common specification down the line, said Warner. He even foresees a community marketplace or shared registry for infrastructure abstractions, enabling platforms to support hundreds of cloud providers.

    In conclusion, the need for LowOps, combined with the rise of multi-cloud and the staying power of on-premises data centers, is fueling a requirement for a decoupled management center. Infrastructure abstraction could enable the convenience of a PaaS while empowering operators with custom infrastructure choices.