devopsy.kinsta.cloud

Tag: Terraform

HashiCorp Releases Slew of Updates to Improve DevSecOps

At a HashiDays event this week, HashiCorp launched updates to its Terraform implementation and a software-as-a-service offering, dubbed HCP Vault Secrets, that promise to simplify infrastructure security and ensure that secrets are not leaked.

In addition, HashiCorp has added a self-hosted edition of its remote access security tool, Boundary, that can now also record SSH sessions. The company is also announcing general availability of Vault Secrets Operator, which provides a high level of abstraction for managing secrets in a Kubernetes environment and which synchronizes with the secrets management interfaces defined by the Kubernetes Technical Oversight Committee.

Finally, HashiCorp is making available in beta a 1.16 update to the Consul service mesh for containers and virtual machines that adds a Sameness Groups capability to improve the resiliency of microservices by redirecting requests to an identical service. HashiCorp has also added a transparent proxy mode to enable failover between services along with a permissive mutual TLS (mTLS) that enables a service to be configured to accept both mTLS traffic from mesh-enabled services and non-Consul mTLS traffic from external downstream applications.

Chris Van Wesep, senior director of product marketing for HashiCorp, said Version 1.5 of HashiCorp Terraform adds config-driven import workflow and check blocks capabilities that make it possible to declaratively use code to add existing resources into a Terraform state. Previously, the only way to update a Terraform state was to use an import command that could only be applied to one resource at a time. Terraform 1.5 also adds automatic code-generation capabilities for imported resources.

In addition, version 1.5 of Terraform also includes a tool to validate that provisioned infrastructure is behaving as intended. That latter capability makes it possible to reduce the number of misconfigurations created by developers that might one day be exploited by cybercriminals to breach an IT environment.

HCP Vault Secrets, meanwhile, provides a beta release of a SaaS edition of the existing Vault offering specifically designed to enable organizations to better manage application secrets that might inadvertently be exposed to cybercriminals in a production environment, said Van Wesep.

HashiCorp is moving toward providing a portfolio of tools that enable DevOps teams to securely provision and manage hybrid cloud computing environments using a common workflow, he noted. A Forrester Consulting survey found nearly three-quarters (73%) of respondents are reporting this strategy is helping them reach their business goals, with another 19% adding they expected this strategy to help them achieve their business goals in the next year.

It’s not clear whether organizations are fully embracing multi-cloud computing, but each additional cloud added to an enterprise IT environment increases the total cost of IT. The only way to mitigate those costs is to define a set of best practices for building and deploying applications that can be applied to any cloud computing environment.

Unfortunately, many organizations will discover that, as they attempt to define those best practices, they have inadvertently become dependent on a set of proprietary application programming interfaces (APIs) that thwart those efforts.

June 13, 2023
Cloudify Integrates DevOps Automation Platform With Terraform

Cloudify this week announced a free user interface (UI) module that extends its infrastructure provisioning platform and enables DevOps professionals to integrate unlimited Terraform modules.

Nati Shalom, Cloudify CTO, said Terraform UI provides a familiar construct for DevOps teams using the open source DevOps automation platform to provision infrastructure-as-code (IaC). The module enables DevOps professionals to seamlessly import Terraform modules to Cloudify without writing blueprints or any additional glue code, he said.

Cloudify now automatically generates the wrapper required for each Terraform template to enable DevOps teams to collaboratively manage secrets, execute concurrent deployments, store states remotely and monitor deployments, users and versions via any Git repository. Those Terraform modules can then be integrated with application programming interfaces (APIs) and continuous integration/continuous delivery (CI/CD) platforms as DevOps teams require.

DevOps teams can also import existing automation modules and scripts into Cloudify. Those modules and scripts can be automatically converted to environments that can be centrally managed via the Cloudify console or via export to ServiceNow’s IT service management (ITSM) platform as a software-as-a-service (SaaS) application.

That approach provides the added benefit of enabling DevOps teams to centrally manage all Terraform modules and versions via a single console as part of an effort to reduce misconfigurations that lead to security breaches, he added.

Cloudify enables DevOps teams to create a self-service catalog and portal through which infrastructure can be managed. Many DevOps teams have built their own portals to manage this process. However, Shalom noted those DevOps teams then need to devote resources to maintaining that portal when those resources could be allocated to other critical tasks.

DevOps teams can not only use Cloudify to apply patches, change topologies, add security features, make configuration changes or make additional resources available, Shalom added, they can also now integrate Terraform tooling with platforms such as Kubernetes or other automation frameworks such as Ansible. Cloudify also makes available an enterprise-grade version of the platform with these integrations.

It’s not clear how much time and effort IT organizations are investing in building and maintaining DevOps environments. In many cases, however, organizations should probably revisit decisions about building capabilities that, at the time, didn’t exist. That doesn’t necessarily mean that replacing custom DevOps tools makes sense across the board, but it is likely that, in the intervening years, the range of tooling made available via DevOps platforms has greatly expanded. Relative newcomers to DevOps, meanwhile, are enjoying the benefit of platforms they did not have to construct themselves.

In fact, Shalom noted that, in terms of DevOps adoption, an inflection point has been reached as more organizations launch digital business transformation initiatives that require applications to be both built and deployed faster than ever. The challenge, of course, is enabling that to occur both safely and at a reasonable cost.

February 17, 2022
How to Migrate Existing Infrastructure to Terraform
Infrastructure-as-code (IaC) enables managing and provisioning infrastructure through code instead of manual processes. IaC is crucial for DevOps teams as it lets them manage infrastructure components, such as networks and load balancers, and enables test applications in production-like environments early in the development cycle. It allows DevOps teams to build, change and manage infrastructure in a safe, consistent and repeatable way.

While infrastructure as code might traditionally be frustrating, as it requires cloud infrastructure to be created manually, Terraform brings the existing infrastructure under code management safely while removing manual processes. Terraform is one of the most popular open source IaC tools because it lets a user define resources and infrastructure in human-readable, declarative configuration files and easily manage infrastructure’s life cycle. It provides several other advantages such as the ability to manage infrastructure on multiple cloud platforms, write infrastructure code quickly, easily track resource changes throughout deployments, version control functionality and more. Terraform can operate on all cloud providers such as AWS, GCP and Azure, and is easy to install on machines or remote build servers.

How is Terraform Different?

Increasing fault tolerance by spreading infrastructure across multiple clouds is a common practice, as using a single cloud provider may provide a single point of failure. Multi-cloud deployment allows for easy recovery of the loss of a region or entire provider. But it can be challenging, as many existing tools for infrastructure management are cloud-specific.

Terraform addresses these challenges as it is cloud-agnostic and allows a single configuration to be used to manage multiple providers. It can also handle cross-cloud dependencies by simplifying management and orchestration.

Terraform provides a flexible abstraction of resources which allows DevOps teams to represent everything from physical hardware, virtual machines and containers to DNS providers easily. This flexibility can also manage a single application or even an entire data center—making it highly sought-after compared to other IaC tools.

Migrating From Existing Infrastructure to Terraform

Terraform brings existing infrastructure under as-code management in a safe and incremental way. One can easily migrate from an existing infrastructure by following these steps:

Install Terraformer from the below link:

Terraformer Install

To import resources from all services, use -- resources="*"

To exclude certain services, one can combine the parameter -- excludes to exclude resources from services. For example, --resources="*" --excludes="iam"

After downloading Terraformer on the machine, one can migrate existing infrastructure into Terraform. For a smooth migration, everything must be included in a single command; otherwise, the state file may be lost or replaced. To migrate manually created resources, you need to specify everything in a single command.

For setting it up on Google Cloud, for example, one must access the information from Google Project and provide authentication information by setting up the environment variable “Google_Application_Credentials” along with the file path to the JSON file that contains the service account key. In this example, we migrated GKE clusters in GCP.

terraformer import google --resources=gke --filter=container_cluster=<Clusters-name separated by :> --filter=container_node_pool=<node-pools name separated by :> --regions=<region-name> --projects=<project-name>

Running this command will fetch all the resources of GKE into the local machine. However, this requires changing the backend to make these changes to Terraform cloud. Here’s how to do that:

backend "remote" {

organization = "<organisation-name>"

workspaces {

name = "<Your workspaces>"

The backend should be kept in a remote setting. The above steps ensure that the GKE cluster is migrated to Terraform without any interruption.

Challenges in Migrating Infrastructure
While Terraformer is an important tool to automate infrastructure and reduce the complexity of managing resources manually, migrating the infrastructure to Terraform can have its own set of challenges. It is important to include all the services in the import command. Failing to do so might require migrating the infrastructure by writing individual files which can be an intimidating task. Another challenge is that while importing different services, the backend file must be replaced or it may lead to losing the first infrastructure backup in the remote backend.

Importance of Migrating Existing Infrastructure

Migrating existing infrastructure to Terraform has several benefits such as:

Easy migration of all the existing infrastructure into the cloud without any interruption

Easy management of resources on the cloud while keeping track of changes

Secure deployment of resources in Terraform cloud

One further caveat: Terraform is a relatively new technology and adapting it to manage an organization’s cloud resources might take some time and effort. It requires specialized skills and training to use Terraform to effectively manage cloud infrastructure. It may also be challenging to use it while dealing with concepts such as a remote backend. Despite these hiccups, Terraform offers huge advantages for DevOps teams to easily manage infrastructure.
February 16, 2022
Building on Terraform: Evolution, not Revolution
Terraform is widely adopted among both enterprises and service providers who embrace infrastructure-as-code (IaC) as their model for infrastructure provisioning automation in DevOps environments. And while the technology has limitations, innovations seeking to unseat Terraform are highly fragmented and have seen tepid adoption. This points to a market that’s settling on a de-facto standard, so rather than trying to replace Terraform with the next great innovation, infrastructure operators should instead find ways to extend and augment Terraform to suit their unique needs. Let’s explore several of the better-known shortcomings of Terraform, some drivers for the rise in new alternatives to the technology, and what users need to consider when contemplating a different path.

Without a doubt, Terraform is big. Across the big three public clouds, the number of Terraform provider installs tells the story: At the end of August, there were 457 million provider installs for AWS, and around 72 million each for Azure and GCP. Yes, compared to longer-in-the-tooth infrastructure configuration management tools like Ansible, Chef and Puppet, IaC as a category is still early. But, the trend is clear: IaC is becoming widely used at scale, and Terraform is the IaC platform most enterprises go with.

It’s also no secret that there are shortcomings to Terraform. A quick search on Reddit and Hackernews unearths the litany of complaints, some more valid than others:
- It’s tricky creating a Kubernetes cluster and then adding a resource to it
- Challenges passing values through a config or dumping a pure representation of an elevated config
- Cluster deletion is a pain
- An overall lack of visibility into what’s happening during the provisioning process
- The need for glue code to drive everything is a drain on developer productivity
- Lingering challenges with backward compatibility
- Continuous update challenges and the need to refactor
- Scaling across multiple workspaces and multiple cloud accounts
- Rollbacks can be tricky, especially when dealing with immutable infrastructure
Terraform’s parent company, HashiCorp, of course, steadily works to address these and other pain points. But the whole thing highlights an interesting question: Why is Terraform so popular?

Terraform is the only IaC solution that meets the needs of developers who want an IaC tool that preserves state, fits well into their CI/CD pipelines, is intent-based and gives them the right level of granularity. Terraform, even with its challenges, is better than the alternatives in these regards.

Here’s why:

Terraform was, from the start, designed for and targeted to developers. It maintains state and supports iteration, making IaC a powerful model for automating infrastructure. PaaS solutions, by contrast, are not robust enough for most developers who do not need or want all of the infrastructure decisions abstracted away. Most cloud management platforms suffer from the least common denominator problem, and even though the technology has improved in this regard, it’s a solution that developers generally abhor for its granularity; it’s geared more to the needs of IT management. Configuration management tools, as an alternative, are not intent-based, so flow becomes too complicated.

It’s interesting to note that state (maintaining the mapping between the resources that you’ve represented in your definition files and the actual physical resources that are in the cloud provider) was added into Terraform, helping resolve the issue of infrastructure drift. Terraform also realized that developers don’t necessarily need a UI. These tangible examples of the developer-centric design ethos at HashiCorp point to why it is so widely adopted.

So, what’s driving the interest in new approaches to infrastructure automation? The proliferation of domain-specific use cases and an ever-present need to reduce time-to-value are the leading reasons. Domain-specific automation options are becoming more dominant over OOB solutions because they take too long to get to automation; one tool does not fit all. Domain-specific automation addresses issues such as security automation, AIOps, DevSecOps, security compliance, integration with container scanners—not just generic orchestration. People are looking for value very fast, which can be found only with domain-specific automation—and therefore, fragmentation happens as different solutions emerge to address those cases.

There are several interesting alternatives to Terraform. Terragrant is used along with Terraform to expand its capabilities to multiple workspaces. It allows you to use the same template across many environments. Pulumi is another alternative that has achieved a measure of adoption and visibility. It provides native code support which addresses the problem with YAML file development complexity. Another more ambitious undertaking is the Crossplane open source project. It takes a Kubernetes-centric approach to creating a universal, cross-cloud control plane. It looks good conceptually, but it’s a fundamentally different approach, and adoption is nascent.

Turning to one of the Terraform alternatives isn’t necessarily a bad idea, but is fraught with landmines. In the end, this choice is less about technology and more about adoption and ecosystem momentum. Vendor and professional investments in tools, training, and skill sets surrounding Terraform is a formidable moat, and it cannot be easily replicated. Tools that would win over Terraform would have to win adoption, and achieving adoption means delivering all the enterprise-grade features and support that Terraform users currently enjoy. Chef and Puppet are examples of how this could happen. Both were successful technologies, but they were slow to react to the challenge presented by Terraform. Ultimately, these projects suffered from a legacy mindset and were overtaken by Terraform’s more focused, simple solution.

Most of the issues with Terraform lie in managing it at scale. When you have many workspaces and templates, it becomes hard to manage; it’s also a challenge that templates cannot be generalized among environments, resulting in a lot of template redundancy. Also, Terraform still needs help handling Day 2 operations and visibility into drift detection.

When considering ways to extend Terraform to address a broader set of use cases, you should first consider how well things will interoperate across your environment. If you have multiple clouds, multiple service layers and multiple cloud vendors, this is probably the most important factor. Also, investigate any new technology’s ability to support the things about Terraform that are critical to your Day 2 operations. Things like secrets and state come to mind. You’ll want enough management and monitoring in the new solution without it being a new burden to carry. You should be able to use your existing Terraform files, and you should be able to support all the versions of Terraform that you run. Compliance with governance and audit are key, and the new solution should support Terraform for both your on-premises/air-gapped environments and your cloud environments. Ideally, anything you add to Terraform would be open source, eliminating new complications from cross-vendor lock-in.

Architecture for how 8451 evolved its Terraform deployment

For all its shortcomings, Terraform remains the best option on the market for DevOps organizations, and there are ways to manage it with fewer headaches and without developers wasting so much time writing glue code or reinventing the wheel over and over again. One way this happens is by integrating Terraform with meta-orchestration tools that allow Terraform to become more flexible in managing environments and workspaces at scale. The open source Cloudify project is one example.

Yevgeniy Brikman gave a talk two years ago at HashiConf that has aged well. At the time, DevOps was still in the Stone Age. Today, it’s probably in the Bronze Age. Those of us building IaC today are being asked to deploy infrastructure that looks like modern cities, but the tools we have to work with remain relatively primitive. What we’re building takes too long to provision and it looks bespoke and temperamental at best or kludgy and fragile at worst.

It’s a story repeated over and over: Teams try to build their own, homegrown solution first as a wrapper on top of Terraform. Ultimately, they realize that it’s unsustainable as a lack of engineering resources becomes a bottleneck. We need solutions that augment—not replace—what they’ve already built and what works. That’s almost always Terraform. With Terraform, DevOps organizations get a proven IaC option that’s targeted to the needs of developers, with a substantial ecosystem and market uptake. When considering its shortcomings, it’s best to embrace an evolution strategy, and not start from ground zero.
November 16, 2021
CloudBolt Survey Shows DevOps Frustration in the Enterprise

A survey of more than 200 senior IT leaders at organizations with more than 1,000 employees published today finds only 4% consider their organizations to be experts when it comes to continuous integration/continuous delivery (CI/CD).

Conducted by CloudBolt Software, a provider of a platform for many IT environments, the survey also found only 11% consider their CI/CD infrastructure to be reliable. Another 69% said their CI/CD infrastructure was somewhat reliable, while 21% said their platforms were somewhat unreliable or not reliable at all. A full 85% also report they regularly test the infrastructure they rely on to drive DevOps processes.

Overall, the CloudBolt survey finds only 5% of respondents report their organization is deploying code multiple times a day. Most respondents (85%) said their organization only deploys code on a weekly or monthly basis.

A full 70% said that, on average, it also takes multiple days or even weeks to set up a pipeline. Nearly two-thirds (63%) said there are too many manual processes involved when setting up pipeline infrastructure. More than half (56%) also noted they have no automated processes for post-deployment tasks.

More than half (53%) said it is also challenging to proactively detect infrastructure issues, while 55% said it is difficult to create a consistent CI/CD pipeline environment across multiple development teams. Improving those pipeline environments requires speeding up the process of provisioning infrastructure through automated processes (70%); continuously detecting infrastructure issues to reduce testing challenges and failures (62%) and simplifying remediation of infrastructure issues proactively (56%), according to survey respondents.

The CloudBolt Software survey specifically identifies issues organizations encounter when employing Terraform to manage infrastructure-as-code (IaC). Only 5% of respondents said they were “very satisfied” with using Terraform and another 75% reported they are somewhat satisfied. Top challenges include difficulty using it in on-premises infrastructure deployments (62%), lack of Day 2 management for deployed resources (53%) and governance (42%). Only 2% of respondents said they were employing Terraform to deploy 75% or more of their infrastructure. A full 90% noted that Terraform requires more custom-coded integrations with other critical tools and systems to work properly.

Finally, 29% said their organization lacks specific Terraform expertise. The most sought-after Terraform capabilities are enabling non-Terraform experts to easily deploy infrastructure using Terraform; ensuring that only Terraform plans with IT “guardrails” are executed and the ability to visualize and cost-optimize infrastructure deployed by Terraform. More than a third (35%) want to simplify deployment of on-premises infrastructure when using Terraform.

Jeff Kulkowski, CloudBolt CEO, said the survey makes it clear there simply isn’t enough expertise available to implement DevOps best practices at scale without enabling IT administrators—that often lack programming expertise—to manage infrastructure and DevOps platforms. That requirement creates the need for an IT management platform based on a set of visual tools that enables DevOps professionals and traditional IT managers to work more collaboratively, he added. DevOps teams can manage IaC whenever they desire, but it’s also crucial to enable IT operations staff to manage IT infrastructure alongside those teams.

Obviously, DevOps as an IT discipline still has a long way to go before it is pervasively adopted across enterprise IT organizations. The challenge now is to remove many of the low-level barriers that conspire to block IT modernization initiatives.

August 25, 2021
Cycloid Tool Can Reverse Engineer Provisioned IaC

Cycloid this week unveiled a tool that makes it possible to reverse engineer code used to manually provision cloud infrastructure. The Infra Import tool is part of an effort to create a more consistent and reliable version of that code using open source Terraform software.

Benjamin Brial, Cycloid CEO, said Infra Import is based on Cycloid’s open source TerraCognita software, which enables IT teams to reverse engineer infrastructure-as-code (IaC) created using open source Terraform tools.

Infra Import extends that capability to cloud platforms such as Amazon Web Services (AWS), Microsoft Azure, Google Cloud Platform or an IT environment employing the open source OpenStack cloud platform that may have been manually provisioned. IT teams can then create a configuration based on that Terraform code that could be employed across multiple clouds.

Infra Import is designed to connect to a cloud provider and then automatically reverse engineer Terraform files and IaC stacks based on the existing deployment. It also ensures that the latest edition of Terraform is being used to create that configuration code, noted Brial. That capability eliminates the need for DevOps teams to keep track of what version of Terraform might need to be updated as new versions of the IaC tool become available.

Just as significantly, Infra Import also makes it a lot easier to onboard new members to a DevOps teams because the tools uses valid secure configurations they know can be relied on. Misconfiguration of cloud services is one of the biggest security issues that IT teams today are wrestling with, simply because most developers lack the security expertise required to configure cloud services properly.

Cycloid is in the process of rolling out a comprehensive suite of management tools that complement a lightweight framework it created to make DevOps best practices more accessible. The challenge is many IT organizations have already manually provisioned cloud infrastructure in a way that is, at the very least, often suboptimal if not downright insecure.

Misconfigurations are rife across cloud computing environments, but most organizations are not prepared to manually address the issue. Automating the reverse engineering of those configurations is the first step toward enabling IT teams to address the problem at scale.

It’s not clear to what degree configuration concerns have impacted the number of applications that would otherwise have been deployed in the cloud. Security is always listed as the top concern IT leaders have when employing cloud platforms. The issue, however, is not the security of the platforms themselves but rather the processes employed to deploy applications on the platforms. Cloud service providers have promoted a shared responsibility model for security under which developers are accountable for both securely configuring services and ensuring the overall security of the software environment. The trouble is, not every developer understands the implications of that shared responsibility model much less how to actually secure an application workload.

It’s not likely security concerns will slow down the rate at which applications are deployed in the cloud. However, as more cloud security issues are encountered, more scrutiny will be applied.

July 30, 2021
HashiCorp Increases Terraform’s Enterprise Appeal

During the online HashiConf Europe conference today, HashiCorp debuted the general availability of a 1.0 release of HashiCorp Terraform along with updates to HashiCorp Terraform Cloud service.

Meghan Liese, senior director of product marketing for HashiCorp, said in both cases HashiCorp is moving to enable organizations to better secure and govern IT infrastructure-as-code provisioned using open source Terraform tools.

Version 1.0 of HashiCorp Terraform provides access to a stable application programming interface (API) that will foster interoperability and simplify future upgrades.

The Terraform Cloud service, which eliminates the need for IT teams to set up and manage a Terraform platform, adds workflows enhancements to the Terraform Cloud that include the ability to publish curated modules to a private registry, enhancements that make it easier to visualize elements of a workspace and a preview of an ability to check with third-party tools related to a Terraform run.

Overall, the goal is to make it simpler for organizations to compose, collaborate and reuse infrastructure-as-code using modules and the public and private registry options, said Liese. The public registry already has more than 5,000 modules. Terraform Cloud now offers a native workflow to publish modules from the public registry directly into an organization’s private registry.

Terraform, of course, has been widely employed since it first debuted in 2014. There have been 15 major updates prior to the release of Terraform 1.0, which is intended to provide a foundation that ensures backward compatibility between releases going forward.

In some instances, Terraform can be too much of good thing. It has considerably accelerated the rate at which IT infrastructure can be provisioned within the context of a DevOps workflow. The challenge, though, is developers often lack the security expertise required to prevent infrastructure from being misconfigured at a time when cybercriminals have become more adept at scanning for those vulnerabilities. To enable IT teams to prevent those misconfigurations, HashiCorp makes available both an enterprise edition of Terraform as well as its Terraform Cloud service. Terraform Enterprise is used by more than 1,200 enterprises, while Terraform Cloud has more than 120,000 users that execute more than 30,000 runs per day.

It’s not clear if security concerns might one day lead to a backlash against the concept of shifting more responsibility left toward developers to improve productivity. In some cases, developers are simply deploying more insecure applications faster. HashiCorp is making a case for Terraform platforms that allow centralized IT teams to exercise more control over how infrastructure is provisioned as code. The need for that approach is becoming more pronounced, as organizations not only employ Terraform to provision cloud infrastructure but also on-premises IT environments.

Regardless of how infrastructure is provisioned by developers, security of infrastructure, both inside and outside of the cloud, remains a priority as more workloads than ever are deployed. In effect, the attack surface that needs to be defended has dramatically expanded. Of course, the best security issue of all is the one that never happened because infrastructure is being provisioned right the first time and every time thereafter.

June 8, 2021
Cisco Leans on HashiCorp to Manage Infrastructure as Code

HashiCorp and Cisco today announced a Cisco Intersight Service for Terraform offering based on HashiCorp Terraform Cloud Business software.

Cisco Intersight is an IT operations management platform that enables IT teams to manage IT infrastructure across a hybrid cloud computing environment. Open source Terraform tools have emerged as the dominant tool for provisioning infrastructure as code in the cloud and, increasingly, in on-premises IT environments.

HashiCorp Terraform Cloud Business is an instance of Terraform that has been extended to include some additional security and governance capabilities enterprise IT organizations typically require.

In recent years, Cisco has significantly expanded its presence in on-premises IT environments as many IT teams opted to deploy its servers alongside its networking equipment. Like every other provider of on-premises IT environment, Cisco has been moving to extend the reach of its management frameworks to public clouds to help foster the rise of hybrid cloud computing.

Prior to the pandemic, most public clouds and on-premises IT environments were managed in isolation from one another. However, in the wake of the economic downturn brought on by the pandemic, more IT teams are trying to centralize the management of IT platforms as part of an ongoing effort to reduce the total cost of IT. The multiyear alliance with HashiCorp fills a major hole in the Cisco management portfolio, as Terraform increasingly becomes a de facto standard for provisioning IT infrastructure-as-code.

Vijay Venugopal, head of product management for Cisco Cloud and Compute Software, said the alliance with HashiCorp will make it possible to provide Terraform support across a management portfolio that, over time, continues to be enhanced with machine learning algorithms that add artificial intelligence (AI) capabilities, also known as AIOps. That platform is already being employed by 6,000 customers to manage more than 600,000 servers, running primarily in on-premises IT environments, added Venugopal.

It’s unclear how the battle for control over hybrid cloud computing will ultimately play out. While providers of on-premises platforms such as Cisco are making a case for using existing management frameworks to also manage workloads deployed on public clouds, the providers of those platforms are now extending their reach into on-premises IT environments. Cisco is, of course, betting that most organizations would prefer to extend an existing framework to achieve that goal, versus having to rip and replace existing IT infrastructure to employ an alternative framework from a cloud service provider.

Regardless of how organizations ultimately decide to manage IT in the age of the hybrid cloud, it’s clear developers, thanks to tools such as Terraform, are exercising more direct control over IT infrastructure. Incorporating Terraform within a larger IT management framework presents an opportunity to bridge the divide between developers and IT operations teams that have historically relied on visual tools to provision infrastructure. The challenge now is finding a way for developers and IT administrators – with varying levels of programming expertise – to work more closely as management of IT environments become increasingly more complex.

March 25, 2021
IaC Frameworks: Vendor-Specific or Multi-Cloud?

Over the past few years, I have been watching the shift to infrastructure as code (IaC) frameworks closely. With the shift from monolithic to microservices, and with the shift from mostly virtual machines (VMs) to cloud-native architecture, applications are much more complicated, making it much more important to automate infrastructure to respond with speed.

In that sense, I refer to IaC as “The third cloud revolution.” Looking at the history of cloud transformation, the shift from physical servers to virtual machines (led by VMware about 20 years ago) would be the first cloud revolution. The move from virtualized data centers to the public cloud (led by AWS about 10 years ago) would be the second cloud revolution. With the shift to public clouds, people were suddenly able to click buttons to provision new resources, including “hardware.”

In the last few years however, we see fewer clicks and more code. Engineers write and maintain that IaC, and something executes that code. Engineers influence the infrastructure much more than they used to – that is the third cloud revolution. We mainly see four IaC frameworks that drove this revolution – Terraform and Pulumi (both multi-cloud), CloudFormation and ARM (vendor specific).

I recently talked with Joe Duffy, CEO, Pulumi, Mark Russinovich, CTO, Azure and Anton Babenko, AWS community hero and Hashicorp ambassador, the head of Terraform at Microsoft, the head of Terraform at IBM Cloud and one of the founders at Gruntwork. After our brainstorming session, it’s clear we all agree that IaC will have a huge effect on cloud operations. In this blog, I’ll share my point of view regarding multi-vendor IaC versus single-vendor IaC frameworks; discuss how they compare to the differences between iOS and Android and which direction I believe will win the day.

The big cloud vendors seem to go “all-in” and create full ecosystem, iOS-style, and they have everything working nicely with the relevant services and tools. AWS just released AWS Proton, which works with CloudFormation. They say that Terraform is on their roadmap, but time will tell. Microsoft seems to offer more solutions that are ARM-based, like Bicep. When I talk with executives at those companies, they strongly believe in the great value of this vendor lock-in/closed approach. They truly believe that this will help their customers in the best way possible. Perhaps they are right.

On the other hand, when I talk with the community, I hear different opinions. DevOps engineers at Azure customers do not debate whether to choose Terraform or ARM. DevOps engineers at AWS customers do not debate whether to choose Cloudformation or Terraform. While this was the case three years ago, it is not, anymore. Now, the real debate is whether to choose Terraform or Pulumi. Engineers prefer a language that is multi-vendor, open-source, no vendor lock-in and that is compatible with other open source solutions built on top, like Open Policy Agent and Terragrunt, to help Terraform users. This is the Android-style approach – open platform.

Moreover, I hear new phrases from the community that make me think the future is going in this direction. Erik Osterman at Cloud Posse started using the acronym TACOS – Terraform Automation and Collaboration Software (see the video here). It takes time, but, going forward, developers and the community will hold the key to the future of IaC, not executives and large vendors.

You can, of course, challenge this by replying that, “iOS is a great OS.” This is obviously true, but I think there is one big difference here as to why the Android approach is winning the IaC market. The answer lies with the end user; the Apple ecosystem is a solution for the end user and consumer, and IaC is about larger-scale infrastructure decisions. Like the shift from monolith to microservices, developers prefer small and isolated parts, not one big system. With Terraform and Pulumi, developers get more freedom. Maybe the company will move from AWS to GCP or Azure or vice versa. Maybe IT will need an Auth0 provider, as well, and will want to manage it the same way. Maybe the use of Open Policy Agent to govern deployments becomes the status quo. Developers need freedom, whereas consumers just need a great product.

But the question has still not been answered: Terraform or Pulumi? It is still too early to tell, and, frankly, there is more than enough room in the IaC space for both. Maybe more open-source and multi-vendor frameworks will emerge and one that’s yet unknown will rule the day. Terraform provides a great ecosystem of providers, support and maintainers. Pulumi has started to provide all of those, together with its imperative approach, which seems to be a bit more appealing to developers.

March 4, 2021
HashiCorp Unfurls Cloud Platform for Managed Services

HashiCorp today at its online HashiConf Digital event launched HashiCorp Cloud Platform (HCP), through which it intends to make all of its software available as a fully managed service.

At the same time, HashiCorp announced a public beta of an update to HashiCorp Terraform, a curated instance of an open source tool for configuring IT infrastructure that adds additional workflow capabilities.

Finally, an update to Nomad, a cluster manager and scheduler for containerized application environments, adds support for multiple clusters as well as the Container Network Interface (CNI) specification defined by the Cloud Native Computing Foundation (CNCF).

Rob Genova, director of product marketing for cloud services at HashiCorp, said although the company previously made some of its offering available as a managed service, HCP will serve as a foundation on which HashiCorp will manage updates to its software on behalf of end customers.

The first managed service on HCP, now available in private beta, is an instance of the Consul service mesh deployed on Amazon Web Services (AWS), which will be followed by an instance of Vault, secrets management software developed HashiCorp, that will also be deployed on AWS.

At the core of HCP is a HashiCorp Virtual Network (HVN), which offers a common abstraction across cloud providers around an isolated single-tenant network. Each HVN can have multiple services deployed within it. In the future, HashiCorp is promising HVN will enable peering across cloud providers and cross-provider clustering of its software. Support for additional cloud services will be added based on customer demand, added Genova.

HCP will be offered on a pay-as-you-go pricing model to minimize upfront costs, which Genova said should make Consul and other HCP services more accessible to smaller teams and organizations. Costs will increase only when workloads scale. Both development and production pricing options will be made available, he said.

HashiCorp already provides access to hosted instances of Consul and Terraform, which it will continue to offer alongside HCP, Genova added.

Genova said it’s become apparent more organizations would prefer to have HashiCorp manage Consul, Vault, Terraform and Nomad on their behalf. At a time when many organizations have limited IT resources, Genova said many organizations want to devote the bulk of their limited internal resources to building and deploying applications rather than the infrastructure required to maintain them.

It’s not clear how many organizations will opt to rely more on managed services in the wake of the economic downturn brought on by the COVID-19 pandemic. At a time when many organizations are either reducing or freezing headcount, all options are being considered. Many internal IT teams may occasionally employ a managed service to extend their own capabilities. In other cases, developers may bypass an IT operations team altogether.

In the wake of the pandemic, Genova noted more organizations simply want to deploy resources in the cloud to ensure availability while gaining increased flexibility.

Regardless of the path chosen, the days when IT teams felt more compelled to do everything themselves are, for better or worse, changing utterly.

June 22, 2020