devopsy.kinsta.cloud

Tag: Pulumi

  • Pulumi Adds Registry to Share Secure IaC Code

    Pulumi Adds Registry to Share Secure IaC Code

    Pulumi this week added a public registry to its portfolio that makes it easier to discover packages that have been created using its tools to manage infrastructure-as-code (IaC).

    Joe Duffy, Pulumi CEO, said the Pulumi Registry will make it simpler for DevOps teams to share and reuse IaC tools that have been vetted versus always requiring each development team to write new code each time they want to programmatically provision infrastructure. Pulumi Packages provide modern cloud reference architectures in the form of software development kits (SDKs), code samples and how-to guides.

    One of the primary reasons that security has become a major cloud issue is that developers often misconfigure infrastructure. The registry provides a way to reduce those configuration errors by making it easier for developers to reuse code written in TypeScript/JavaScript, Python, Go and .NET that they have stored in repositories such as npm.

    In addition to the major cloud service providers, there are also Pulumi Provider Packages for Auth0, CloudFlare, Confluent Cloud, Datadog, DigitalOcean, Docker, GitHub, Kong, MinIO, MongoDB Atlas, PagerDuty, Snowflake, Spot by NetApp and others.

    There are also Pulumi Component Packages for deploying container applications on instances of Kubernetes and other related platforms in addition to deploying applications on serverless computing frameworks.

    Duffy said in addition to providing a “golden image” of their configurations the registry reduces the overall level of friction that often exists between developers, IT operations teams and cybersecurity professionals.

    The latest Pulumi offering is part of an ongoing effort to democratize cloud computing in a way that is repeatable and reliable, noted Duffy. Pulumi, for example, just added support for a set of AWS Cloud Control application programming interfaces (APIs) that reduce the number of APIs that developers would otherwise have to master. The more APIs there are, the more likely it is a developer will make a mistake.

    IT teams should expect that other cloud service providers will similarly streamline the APIs they present developers as part of an effort to make their services both more accessible and secure. In the meantime, frameworks provided by Pulumi provide a means to invoke those lower-level APIs at a higher level of abstraction for both developers and internal IT operations teams alike.

    It’s not clear today how many developers routinely employ IaC tools, but as the number of workloads deployed in the cloud steadily increases, so too does the percentage of developers that employ these tools. While that shift represents a major boon in productivity for developers, it brings with it additional security risks that many organizations fail to appreciate until it’s too late.

    The only way to really solve that issue is for organizations to adopt DevSecOps best practices that require a level of collaboration between DevOps and security teams that is still relatively rare. However, as the tooling provided continues to improve, the technical challenges associated with achieving that goal continue to decline. The next big issue, as always, is bringing together all the diverse cultures within an IT organization to actually implement those tools and define a set of best practices that work for all concerned.

  • Pulumi Moves to Automate Cloud Infrastructure Provisioning

    Pulumi Moves to Automate Cloud Infrastructure Provisioning

    Pulumi announced today it is expanding the scope of its automation ambitions to make it easier to securely provision cloud infrastructure-as-code.

    Joe Duffy, Pulumi CEO, said the Cloud Engineering Platform enables DevOps teams to use Pulumi Packages to create reusable components for automating IT infrastructure provisioning in a way that can be embedded within workflows and applications using an application programming interface (API) that Pulumi has exposed. IT teams can define a Pulumi Package in the programing language of their choice, Duffy noted.

    The Cloud Engineering Platform is now generally available on the Microsoft Azure cloud, while a preview edition of the offering is available on Google Cloud Platform. Support for Amazon Web Services (AWS) is planned for later this year.

    The platform itself is based on version 3.0 of Pulumi, an open source tool that developers have previously employed to manage infrastructure-as-code. Pulumi is also making generally available a tool for integrating its tools with more than a dozen continuous integration/continuously delivery (CI/ CD) platforms.

    That capability, coupled with existing support for integrated testing capabilities and the tools for managing compliance-as-code based on identities, provides IT teams with a more comprehensive approach to managing infrastructure-as-code, said Duffy.

    Most cloud resources today are directly provisioned by developers using open source tools such as Terraform. The issue that organizations have encountered is that developers are prone to making configuration mistakes that cybercriminals then exploit. Pulumi is making a case for an alternative approach to provisioning infrastructure-as-code that makes it easier to validate configurations using testing tools, while at the same time limiting who can access infrastructure resources by including support for the secure access markup language (SAML) and single sign-on (SSO) capabilities.

    It’s not clear at what rate enterprise IT organizations will be shifting toward platforms that enable infrastructure to be managed as code in a more robust fashion. Developers that don’t always have the greatest appreciation for cloud security issues routinely employ tools such as Terraform to provision infrastructure-as-code with little to no supervision. Cloud resource misconfigurations often result in ports left wide open, and cybercriminals now make use of tools to scan for those types of misconfigurations.

    Pulumi Packages provides a means for IT teams to exercise more control over the provisioning process, using vetted reusable components in a way that doesn’t compromise the rate at which developers can spin up cloud resources, noted Duffy.

    As organizations focus more on software supply chains in the wake of some recent high-profile breaches, it’s only a matter of time before more questions arise about how cloud infrastructure is being provisioned. Much of the concerns organizations already have about cloud security doesn’t stem from the platforms themselves. Rather, it’s the processes employed to provision infrastructure under a shared cloud security responsibility model that results in so many vulnerabilities. Many developers assume the cloud service provider is securing configurations, only to discover later that it was their responsibility to validate those configurations. Security teams, meanwhile, can’t keep pace with the rate at which cloud infrastructure resources are being provisioned.

    Hopefully, there will come a day when DevSecOps best practices leverage automation to resolve this issue once and for all. The challenge is finding the best way to achieve that goal in a way the average developer will accept and embrace.

  • IaC Frameworks: Vendor-Specific or Multi-Cloud?

    IaC Frameworks: Vendor-Specific or Multi-Cloud?

    Over the past few years, I have been watching the shift to infrastructure as code (IaC) frameworks closely. With the shift from monolithic to microservices, and with the shift from mostly virtual machines (VMs) to cloud-native architecture, applications are much more complicated, making it much more important to automate infrastructure to respond with speed.

    In that sense, I refer to IaC as “The third cloud revolution.” Looking at the history of cloud transformation, the shift from physical servers to virtual machines (led by VMware about 20 years ago) would be the first cloud revolution. The move from virtualized data centers to the public cloud (led by AWS about 10 years ago) would be the second cloud revolution. With the shift to public clouds, people were suddenly able to click buttons to provision new resources, including “hardware.”

    In the last few years however, we see fewer clicks and more code. Engineers write and maintain that IaC, and something executes that code. Engineers influence the infrastructure much more than they used to – that is the third cloud revolution. We mainly see four IaC frameworks that drove this revolution – Terraform and Pulumi (both multi-cloud), CloudFormation and ARM (vendor specific).

    I recently talked with Joe Duffy, CEO, Pulumi, Mark Russinovich, CTO, Azure and Anton Babenko, AWS community hero and Hashicorp ambassador, the head of Terraform at Microsoft, the head of Terraform at IBM Cloud and one of the founders at Gruntwork. After our brainstorming session, it’s clear we all agree that IaC will have a huge effect on cloud operations. In this blog, I’ll share my point of view regarding multi-vendor IaC versus single-vendor IaC frameworks; discuss how they compare to the differences between iOS and Android and which direction I believe will win the day.

    The big cloud vendors seem to go “all-in” and create full ecosystem, iOS-style, and they have everything working nicely with the relevant services and tools. AWS just released AWS Proton, which works with CloudFormation. They say that Terraform is on their roadmap, but time will tell. Microsoft seems to offer more solutions that are ARM-based, like Bicep. When I talk with executives at those companies, they strongly believe in the great value of this vendor lock-in/closed approach. They truly believe that this will help their customers in the best way possible. Perhaps they are right.

    On the other hand, when I talk with the community, I hear different opinions. DevOps engineers at Azure customers do not debate whether to choose Terraform or ARM. DevOps engineers at AWS customers do not debate whether to choose Cloudformation or Terraform. While this was the case three years ago, it is not, anymore. Now, the real debate is whether to choose Terraform or Pulumi. Engineers prefer a language that is multi-vendor, open-source, no vendor lock-in and that is compatible with other open source solutions built on top, like Open Policy Agent and Terragrunt, to help Terraform users. This is the Android-style approach – open platform.

    Moreover, I hear new phrases from the community that make me think the future is going in this direction. Erik Osterman at Cloud Posse started using the acronym  TACOS – Terraform Automation and Collaboration Software (see the video here). It takes time, but, going forward, developers and the community will hold the key to the future of IaC, not executives and large vendors.

    You can, of course, challenge this by replying that, “iOS is a great OS.” This is obviously true, but I think there is one big difference here as to why the Android approach is winning the IaC market. The answer lies with the end user; the Apple ecosystem is a solution for the end user and consumer, and IaC is about larger-scale infrastructure decisions. Like the shift from monolith to microservices, developers prefer small and isolated parts, not one big system. With Terraform and Pulumi, developers get more freedom. Maybe the company will move from AWS to GCP or Azure or vice versa. Maybe IT will need an Auth0 provider, as well, and will want to manage it the same way. Maybe the use of Open Policy Agent to govern deployments becomes the status quo. Developers need freedom, whereas consumers just need a great product.

    But the question has still not been answered: Terraform or Pulumi? It is still too early to tell, and, frankly, there is more than enough room in the IaC space for both. Maybe more open-source and multi-vendor frameworks will emerge and one that’s yet unknown will rule the day. Terraform provides a great ecosystem of providers, support and maintainers. Pulumi has started to provide all of those, together with its imperative approach, which seems to be a bit more appealing to developers.

  • Pulumi Tightens DevOps Platform Integration

    Pulumi Tightens DevOps Platform Integration

    Pulumi today released the 2.0 release of its open source platform for automating the provisioning of IT infrastructure in the cloud, providing tighter integration with a dozen different continuous integration/continuous delivery (CI/CD) platforms and source code management (SCM) systems. Via the latest update Pulumi also now includes CI/CD Apps, through which pull requests will trigger Pulumi previews and updates.

    In addition, Pulumi has added tools to preview changes, while making it easier to share architectural patterns, libraries and best practices.

    Finally, Pulumi 2.0 adds additional tools to enforce policies, simplify testing and manage secrets.

    Pulumi is designed to enable developers to provision cloud infrastructure using familiar programming tools such as Python, JavaScript, TypeScript, Go and .NET/C#. That approach eliminates the need to master YAML files or other proprietary languages and templates.

    Pulumi CEO Joe Duffy said the goal is to accelerate the rate at which management of IT is shifting left toward developers by providing them with the “superpowers” need to manage cloud infrastructure. Thus far, Pulumi claims to have more than 10,000 users of its tools.

    In the wake of the economic downturn brought on by the COVID-19 pandemic, it is expected the rate at which control over IT has been shifting left will accelerate. As part of an effort to either contain or eliminate headcount, many organizations will reduce their reliance on IT operations staff by giving developers more direct control over IT infrastructure.

    At the same time, however, Duffy noted that many IT operations personnel are finding it relatively simple to master Python, which they can then use to leverage Pulumi to automate the provisioning of large amounts of cloud infrastructure to become more efficient. Some organizations may prefer to retain IT operations staff that have learned how to program.

    It remains to be seen to what degree tighter integration with CI/CD platforms might encourage developers to take more control over IT infrastructure. Support for the CI/CD platforms supported by Pulumi—Jenkins, Spinnaker and GitHub—will make it easier for developers to achieve that goal. Less clear is the degree to which developers want to take control over infrastructure. Regardless, many of them may soon have little choice in the matter.

    Of course, Pulumi is not the only framework via which the provisioning of cloud infrastructure can be automated. It does, however, make it easier for developers to provision infrastructure without having to spend time learning the nuances of another set of programming tools. Even in organizations that do shrink the size of their IT staffs, there will still be pressure on developers to deliver more code faster. The amount of time developers may want to allocate to managing IT infrastructure is likely to be limited at best.

    Whatever the path forward, many organizations will soon discover they can’t afford to wait any longer for their internal IT cultures to embrace best automation and DevOps practices. Economic pressure to optimize constrained IT resources are likely to become intense. No one may know for certain how long the downturn may last, but it is already apparent many organizations will be trying to do a lot more with less in the days ahead.

  • Pulumi Advances DevOps on AWS

    Pulumi Advances DevOps on AWS

    Pulumi announced the availability of Pulumi Crosswalk for Amazon Web Services (AWS), an open source framework for deploying applications on Amazon Web Services (AWS).

    Company CEO Joe Duffy said Pulumi Crosswalk for AWS provides a software-as-a-service (SaaS) application through which DevOps teams can automate the deployment of applications on AWS infrastructure using software development kits (SDKs) from Pulumi. The SDKs are based on programming tools such as JavaScript, Python, TypeScript and Go, which most developers already know how to use. That approach eliminates the need for DevOps teams to learn how to employ YAML files or to acquire tools such as CloudFormation or Terraform to deploy applications on AWS, he said.

    Pulumi Crosswalk for AWS also embeds the best DevOps processes recommended by AWS directly within its SaaS tools, including patterns for containers and serverless computing frameworks, Duffy noted.

    Pulumi Crosswalk for AWS is available on GitHub. It is designed to be integrated with existing Team and Enterprise Editions of Crosswalk that come with support and training provided by Pulumi.

    In addition to AWS, Pulumi Crosswalk supports other cloud platforms, including Microsoft Azure and Google Cloud Platform (GCP). The company also plans to provide support for private clouds and leverage Kubernetes clusters to automate deployments across hybrid cloud computing environments.

    Duffy noted that despite all the enthusiasm for DevOps, many IT operations teams have been left behind. While much time and effort has been put into unifying backend cloud services, the operational experience in the cloud still revolves around low-level application programming interfaces (APIs) or specialized tools that are challenging for IT operations teams to employ. That situation often creates a backlog of applications for IT operations to deploy. Crosswalk enables IT operations teams to leverage the skills developers have to deploy applications they approve on AWS, he said.

    As cloud computing evolves in the age of DevOps, the relationship between developers and the IT operations team is becoming more nuanced. IT operations teams still need to put guardrails in place, but they can’t afford to be perceived as a bottleneck slowing down the rate at which applications can be deployed in the cloud. The challenge IT operations teams must overcome is finding a way to enable developers to self-service their application deployment requirements as much as possible within a set of policy guidelines defined by the internal IT team. That approach will enable developers and IT operations teams to get out of each other’s way.

    It may be a while before IT operations and developers find the right DevOps balance that best suits their organizations. Individual teams within enterprises tend to define their own best practices based on the expertise available within the team. Ideally, those best practices are then shared and implemented as appropriate across the rest of the enterprise.

    In the meantime, it will be interesting to see how aggressively cloud service providers embrace tools such as those from Pulumi. Cloud service providers have a vested interest in getting as many workloads deployed in their clouds as quickly as possible. They tend to be a little less enthusiastic about anything that makes it easier to move those workloads anytime a DevOps team sees fit.

    — Mike Vizard