devopsy.kinsta.cloud

Tag: IaC

  • Infrastructure Abstraction Will Be Key to Managing Multi-Cloud

    Infrastructure Abstraction Will Be Key to Managing Multi-Cloud

    There’s no doubt we’re now living in a hybrid multi-cloud age. Companies are using major cloud providers like AWS, Azure and GCP with niche cloud computing providers emerging within the market. Simultaneously, many organizations still retain workloads in on-premises data centers or embrace colocation. Regardless of deployment pattern, DevOps teams desire the same elasticity as the cloud, but it’s becoming increasingly difficult to consistently manage the nuances of each service provider.

    To support today’s distributed polyglot architectures, a common layer could ease issues when deploying applications and provisioning various servers. By decoupling the infrastructure control plane from specific cloud providers, a unified management console could bring visibility into cost and performance, empowering CloudOps with knowledge before they commit. But engineering such an outcome would rely on plugins for each environment and a robust set of APIs to enable programmatic control.

    I recently met with Jake Warner, Cycle.io, to explore how increased infrastructure abstraction can solve the above problems. According to Warner, a standard layer for computing management is needed to enable consistent control over multi-cloud and bespoke on-premise environments. Below, we’ll define this concept and consider the benefits infrastructure abstraction could bring to modern enterprise software development.

    The Problems Facing Infrastructure Management

    We like to think that the cloud is pervasive and that everyone’s on board with a similar computing style. However, this is far from the truth. Multi-cloud is rising, and 95% of businesses are making multi-cloud a strategic priority in 2022, a Valtix study found. Yet only about half of respondents felt they had the appropriate tools and skills required to execute a multi-cloud strategy.

    “There’s a lot of value in multi-cloud, including reducing risk, ensuring better uptime and the flexibility to select the best hardware for the use case,” said Warner. “But multi-cloud is extremely complex.” Using multiple clouds means supporting increasingly variable operational processes for each environment. Whether it’s AWS, GCP, Azure, Equinix or Vultr, operators also require a deep level of visibility into telemetry data. This becomes complicated to track as software is increasingly modular and distributed across multiple computing regions.

    Furthermore, on-premises data centers are far from dead—surprisingly, a Spiceworks study found that 98% of enterprises still used on-premises servers. Traditional servers are often preferred due to corporate restrictions or to meet security compliance mandates. It could also be a cultural case of “if it works, don’t fix it.” The dilemma is that you can’t easily obtain common cloud features like auto-scaling and load balancing while on your own infrastructure.

    Another issue Warner described is that traditional platform-as-a-service (PaaS) companies present inherent limitations if you aim to build substantive applications and services. “Many of these PaaS offerings make it really easy to get started but, more often than not, developers are quickly forced to make compromises with their approach. These compromises can create long-term technical issues, especially when it’s time to scale,” he said. “Need a server with a specific hardware configuration in a targeted region? Good luck. For most PaaS companies, their exit ramp is just as defined as their on-ramp.”

    Movements Toward Infrastructure Abstraction

    We’re already seeing movement within the market to manage the operational burdens of multi-cloud. One example is the Infrastructure Abstraction Layer (IAL), open sourced by Cycle.io, which Warner explained can act as a standardized system for deployments. With the specification, an infrastructure provider can generate an API endpoint to enable the provisioning of infrastructure without requiring a deep integration between both services. With this endpoint live, an application development platform or orchestrator gains the ability to send standardized JSON payloads to provision servers, allocate IPs, configure regions and a number of other requests. The response, another JSON payload, confirms that the desired request has been completed and includes a unique ID.

    With more programmatic control over infrastructure, an engineer could interface with a standardized list of all the servers, across a number of providers, to deploy to and act upon this information in one unified console. This layer is an opportune location to help compare costs and server configurations for different regions too. It’s also an advantageous spot to filter by proximity to showcase the closest available servers to optimize delivery.

    Benefits of Unifying Multi-Cloud Deployment

    Decoupling the infrastructure management layer from service providers can bring potential benefits to DevOps, said Warner. For one, it enables operators to extend infrastructure automation into their private data centers, helping them achieve the same programmability they would have in the cloud. A degree of standardization can remove many of the cons of multi-cloud, too—”Developers can move from one provider to another provider without any major changes along the way,” he said.

    Of course, IT doesn’t want an added burden. This is why an API-based solution is required to enable programmability, said Warner. By taking a RESTful API approach to managing a hybrid infrastructure, operators could create custom flows based on fine-grained environmental characteristics that change over time, like latency, expense or location. “One of the neat things is it allows infrastructure providers to constantly make tweaks and improvements without having to resubmit their integration with an underlying platform,” said Warner.

    The Flexibility of LowOps

    Today’s software deployment is increasingly wide and varied—a recent HashiCorp report found that 76% of organizations are already multi-cloud. Multi-cloud brings many advantages, yet it also multiplies the number of operational tasks required. At the same time, organizations want to leverage investments into on-premises data centers for as long as they are economically viable. Automating DevOps for such a varied infrastructure can be difficult without a unified layer decoupled from each niche computing area.

    According to Warner, companies require a LowOps platform and the flexibility to abstract this automation on their own infrastructure when appropriate, which can vary widely. For example, operators may be working with a fleet of tens of thousands of IoT devices. Furthermore, as we’ve covered previously, a LowOps method can be beneficial to small-to-medium-sized companies that may lack the resources to support complex infrastructure like Kubernetes.

    Could something like IAL ever become an open standard? Well, at the time of writing, the IAL specification has only been implemented by Cycle.io and a few of its infrastructure partners, but there’s no reason it couldn’t be extrapolated into a common specification down the line, said Warner. He even foresees a community marketplace or shared registry for infrastructure abstractions, enabling platforms to support hundreds of cloud providers.

    In conclusion, the need for LowOps, combined with the rise of multi-cloud and the staying power of on-premises data centers, is fueling a requirement for a decoupled management center. Infrastructure abstraction could enable the convenience of a PaaS while empowering operators with custom infrastructure choices.

  • Where Should I Store My IaC?

    Where Should I Store My IaC?

    The most successful software companies rely on repeatability, auditability and simplicity when building solutions. The emergence of infrastructure-as-code (IaC) has empowered developers to apply these practices to infrastructure allocation. What are the best practices for storing IaC? Should IaC code be stored with the application and feature code? Should it be stored separately in its own application-specific repo? Or should all IaC code for all applications in an organization reside in the same repo? 

    Let’s explore these questions, determine the tradeoffs between each solution and create guidelines that consider your organization’s needs.

    The Problem With Storing IaC With Feature Code

    The naive option for storing IaC is to keep all IaC code within the application repository that consumes this IaC code as part of its CI/CD pipeline. It’s easy to set up, doesn’t require complex permission allocation and will be fully visible to developers who work on this codebase, thus making it easy to retrieve context. This ease is why 20% of polled developers state they always store feature code and IaC in the same repo. However, there are a number of problems with holding IaC code with feature code.

    Software development best practices include minimizing redundancy. Storing all IaC code within the dependent application’s codebase violates best practices by requiring the infrastructure-as-code to contain not just application-specific allocation information but also all foundational infrastructure. Organizations may classify common infrastructure shared across multiple applications and ecosystems as foundational infrastructure. Examples of such base, foundational infrastructure include Kubernetes clusters and log storage accounts. 

    In addition, storing IaC code with feature code introduces risk. Allowing developers admin-level access to all deployment scripts needed to bring code to production violates the principle of least privilege and makes lateral movement easier for attackers who manage to hijack a developer account.    

    Organizations benefit from a degree of secrecy regarding their deployment processes; the deployment pipeline and its configurations are bespoke and attackers who have a mapping of the services an organization uses can more easily architect supply chain attacks. Storing IaC with feature code makes it so that a code leak reveals all CI/CD configurations for not just the application, but the whole organization. The last thing an organization should do is perform an attacker’s reconnaissance for them, but that’s exactly what happens in the above situation.

    Storing infrastructure-as-code alongside product code may be easier to set up initially, but violates the principle of least privilege and zero-trust principles. This method of storing code introduces unnecessary risk–there must be a better solution.

    Storing All IaC Code Together is Not the Answer

    Since storing IaC with feature code in a common repo isn’t the optimal solution, what about storing all IaC in a dedicated IaC repo? This would allow DevOps and security teams to implement granular access controls and make code redundancy easily identifiable and preventable. However, this organizational structure introduces its own problems.

    When all IaC is stored in a separate, shared repository, it becomes difficult to match feature changes to IaC changes. This scenario introduces friction to the development processes, puts a damper on developer velocity and increases tension between development teams, DevOps teams and security teams. 

    Storing all IaC in a common repository gives developers virtually no control over security decisions affecting their workflows. Between the lack of ownership and the additional steps needed to modify any code-specific deployment scripts, storing all IaC together prevents developers from quickly changing IaC and it unnecessarily hurts the organization.

    IaC Falls Into Two Categories: Base and Feature-Specific

    IaC and other CI/CD scripts ultimately fall into two categories: base and feature. 

    Base deployment scripts may be described as code used broadly by an organization for, among other things, routine deployment actions or to interface with shared resources. Separating this code empowers organizations to allocate base IaC ownership and maintenance responsibilities to their DevOps team, which is well suited to maintain shared infrastructure.

    Feature IaC scripts are the bits of code used to provision and manage resources associated with specific software. Developers frequently interact with the deployment scripts tied to their deliverables; changing release steps, modifying gateways, updating server configurations and many other needs necessitate different handling protocols for feature IaC versus base IaC.

    Now that we understand the difference between the feature and base IaC, we can define proper management strategies for each.

    The Solution: A Hybrid Approach for Storing IaC

    Best practices call for categorizing feature IaC and base IaC differently and allocating the responsibilities to separate teams to maximize security while not adding unnecessary friction to the development and release process. Base IaC should be managed by DevOps teams, while feature-specific IaC should be handled by the code owners of the managed software.

    For your organization, this starts with inventorying the organization’s systems and services. Gaining visibility into how these resources are shared helps inform business decisions regarding IaC code storage, helping minimize redundancies within deployment protocols. Visibility is integral to defining which IaC is foundational to software deployed by the organization versus IaC which is specific to certain applications, libraries and other software deliverables.

    Helping DevSecOps Teams Manage IaC and More

    Best practices don’t mean much if organizations can’t enforce them. Some concluding tips: 

    • Track resources, including registries, repositories, builds and cloud/IaC deployments
    • Implement granular access controls, enabling the appropriate allocations of ownership depending on whether the resource is meant to be shared or application-specific
    • Provide visibility into the use of resources which empowers organizations to adhere to the principle of least privilege without adding unnecessary friction to the development process
    • Provide a comprehensive asset inventory by integrating directly with the components of your organization’s software supply chain

    These integrations allow complex insights to be derived that are simply not possible with other security tools. In addition, these integrations allow for constant monitoring for security updates, thus reducing the time between a zero-day being discovered and patching the vulnerability.

  • How IaC Helps Relieve Development Pain Points

    How IaC Helps Relieve Development Pain Points

    Developers spend much of their day digging into their systems’ guts, allocating storage to an application, connecting the security system to the user interface and tracking which version of the application is where in the development cycle, among other tasks.

    That’s why a new category of software has emerged, infrastructure-as-code (IaC), that promises to automate system infrastructure provisioning and integration. It frees up time so developers can focus more on frontend solutions rather than backend configuration management. First, though, let’s find out what IaC is and how it helps your organization.

    What is Infrastructure-as-Code (IaC)?

    IaC reduces system infrastructure creation and configuration time through the use of sets of tools, protocols, languages and processes. In conventional data center system management, each configuration entry and change has historically required manual action. Now, technology teams can manage everything in an intelligent, automated manner. 

    With IaC, infrastructure configuration management data is stored within standardized files. The tools monitor and maintain the state of the infrastructure and ensure that the right resources are available whenever an application runs. This saves DevOps teams from excessive manual effort. In essence, large-scale configuration and management become dramatically simplified.

    Development Pain Points 

    Currently, development is a cumbersome process because friction arises in many places. A great deal of progress has been made in speeding up the front end of application development—building the feature set. Still, the process slows down on the back end when the software components are tied to the computer infrastructure. 

    Historically, software was deployed as one large, monolithic block of code. Updates were tedious to make because all of the application’s code had to run, even if it wasn’t involved in the update or bug fix. The advent of containers and microservices enabled developers to design applications that are modular in nature and therefore easier to update, add to or debug.

    However, the application still has to be tied into the existing compute infrastructure. Cloud solutions—like AWS, Google Cloud Platform or Microsoft Azure, for instance—provide companies with quick access to large volumes of computing processing power. But that application software must be told how to access the network, present security information, allocate storage and run code on central servers.

    Because of this, developers spend a lot of time building and maintaining infrastructure-as-code. To do so, they use languages like Python and solutions such as Chef and GitHub. They want to use automation to simplify and accelerate the process, but here, again, they encounter some roadblocks.

    Workflow Bottlenecks

    As applications become larger and components become smaller, developers find themselves mixing and matching software from a wide variety of places. Tracking what’s where in the process is a frequent challenge. Current workflows are often non-intuitive and rely heavily on pull requests. Some solutions try to address the problem with multiple workspaces, but the end result is fragile and nondeterministic code.

    Without mapping of projects to branches or tags, someone who comments on an approved pull request can push unapproved code to production. In today’s fast-paced DevOps work environment, an approval meant for a short-lived test run becomes the current release, creating a ripple effect of unintended consequences that create confusion and defective code, slowing development—often substantially.

    Access Control

    Most tools do not offer access control models. Developers can track a single repository and a single Terraform project, for example, but as the number of repositories and projects grows, ensuring that proper access control features are in place becomes more daunting. 

    Workspace Challenges

    Legacy waterfall and static development projects allowed IT teams to set their system infrastructure up once and forget about it. But in a dynamic environment, microservices come and go in the blink of an eye. New environments proliferate and product teams must continually configure their own workspaces, which eats up time and resources.

    Drift Detection

    Generic IaC platforms don’t provide mechanisms to detect if the infrastructure is drifting. Because so much configuration is done at runtime, differences arise between the desired and the actual state of the infrastructure.

    Drift has many causes, such as developer or machine script contingencies or the lack of system resources. Too much drift results in unstable applications. Catching such problems early in the development cycle is more efficient than trying to fix a production system. But needed visibility is often lacking, and shortcomings become clear only when the system is fully in use.

    Resource Visualization

    Given today’s dynamic environment, developers need clear visibility. They must be able to closely examine what happens as their job runs. Though they offer many advantages, CI/CD platforms deliver little insight into resource usage from a real-time or historical perspective.

    So, whenever performance problems arise, these platforms offer little insight into what the source of the problem is or how to fix it. Consequently, developers’ attention shifts from application enhancements to troubleshooting.

    Lack of Reusability

    Legacy infrastructure failed to let teams reuse tried-and-true system images across multiple environments. To address this problem, teams often create a new image for each version of each application, slowing development time. 

    High Prices

    Setting up each IT environment manually requires time and manpower, which raises costs. Companies must have dedicated teams of engineers responsible for creating the infrastructure. But they also need supervisors and data center resources, which ultimately boost technology expenses. IaC tools centrally manage the process and create the right environment. That way, organizations pay only for system resources that they truly need.

    How Does IaC Improve the Business? 

    Legacy IT practices are expensive, slow and inconsistent because they rely heavily on manual input. Infrastructure-as-code automates much of the infrastructure management process, which delivers a range of benefits.

    Speed 

    With IaC, provisioning and management are lightyears faster and more efficient than with a manual system. The environment’s infrastructure is defined in code. That code-based infrastructure significantly accelerates each phase of the software development and delivery cycle. Consequently, DevOps teams deliver applications faster and at scale while improving software quality.

    Consistency

    IaC deploys consistent configurations constantly and with no discrepancies. Because of this, it enables fast delivery of consistent infrastructure and environments. This reduces errors and overhead while boosting productivity.

    Eliminating Management Overhead

    Infrastructure automation helps to reduce the number of people with hands on the routing parts of the software development process. That’s because human resources aren’t needed in the management of networking and storage. Companies free up personnel to work on more strategic initiatives.

    Reducing Risks and Errors

    IaC helps to reduce risks and avoid human errors. It also protects against runtime issues and security weaknesses caused by missing dependencies or configuration drift.

    Visibility

    Developers version IaC configuration files the same way they do any other source code file. This gives them complete traceability of any changes. In this way, IaC ensures efficient, exceptionally transparent workflows and speeds up the development process.

    Finding an IaC Solution 

    Maintaining legacy infrastructure was a time-consuming, complex process. IaC is emerging as a solution that alleviates much of the stress and pain that developer teams feel daily. Now, they can speed up the deployment and connection of system resources to applications. Developers can become more efficient and enable the company to respond to business demands in a faster, more reliable and more cost-effective manner. The development process no longer slows down the business. 

  • Palo Alto Networks Extends Checkov Tool for Securing Infrastructure

    Palo Alto Networks Extends Checkov Tool for Securing Infrastructure

    Palo Alto Networks has added support for GitHub Actions, GitLab Runners, CircleCI and Argo Workflows to Checkov, an open source tool that scans programmatically provisioned infrastructure for misconfigurations.

    Guy Eisenkot, senior director of product at Bridgecrew by Prisma Cloud at Palo Alto Networks, said the goal is to make it easier to secure configurations created using infrastructure-as-code (IaC) tools such as Terraform.

    Those additions are now available as part of a Checkov library of policies, including graph-based checks, that provide a context-aware way of identifying risks within infrastructure and application code within a DevSecOps workflow using a tool that enables IT teams to manage policy-as-code, he noted.

    Misconfigurations of cloud infrastructure have become a major issue. Typically, that infrastructure is programmatically provisioned by developers that have little to no cybersecurity expertise. As a result, cybercriminals are now more aggressively scanning for misconfigurations that they can exploit to, for example, exfiltrate data or illicitly access services via application programming interfaces (APIs). Checkov makes it easier to identify those potential security issues within the context of a DevOps workflow before cloud infrastructure is provisioned, noted Eisenkot.

    There is a lot more focus on securing software supply chains in the wake of a series of high-profile breaches. The Biden administration last year even went so far as to issue an executive order requiring federal agencies to review the security of their software supply chains. The challenge is that most organizations have yet to implement a truly developer-centric approach to ensuring application security, said Eisenkot.

    In general, cloud platforms are more secure than on-premises IT environments; however, the processes used to build and deploy cloud applications today are clearly problematic. A chronic shortage of cybersecurity staff further aggravates the issue, because most organizations are not able to keep pace with the rate at which workloads are being deployed in the cloud.

    As more organizations are also starting to embrace DevSecOps best practices, the overall state of cybersecurity should improve. The challenge is that no matter how much time and effort is made to educate developers, there will always be mistakes a cybercriminal can exploit. Policy-as-code tools like Chekhov make it much less likely those mistakes will carry over into a production environment.

    In the meantime, organizations need to work on work on bridging the long-standing divide between application development and cybersecurity teams. Historically, cybersecurity teams would aggregate vulnerabilities they discovered in spreadsheets that developers would then be asked to remediate. The issue is not only the lack of time to address those vulnerabilities but also the lack of context provided. Many of those vulnerabilities often turn out to not be applicable to the way an application is deployed. Over time, application developers start to ignore many of those requests in favor of focusing their efforts on writing additional code. Of course, the more code that is written the greater the number of vulnerabilities that need to theoretically be remediated until a vicious cycle is created.

    Of course, it’s only a matter of time before one vulnerability winds up being a critical exploit and becomes the exception to that rule.

  • Simplifying DevOps and Infrastructure-as-Code (IaC)

    Simplifying DevOps and Infrastructure-as-Code (IaC)

    Let’s face it, tools for infrastructure-as-code (IaC) configuration management may be vital, but they’re difficult to use. But what if it’s not so much the tools themselves as it is our understanding of them? What if we looked at provisioning and configuration systems differently and did so using vocabulary that was less confusing and more common? 

    Simply put, we’d gain a greater understanding of these tools—and, therefore, they would be simpler to use—if the terminology was simpler. Reframing these complex tools would not only allow us to make greater sense of this realm, but we’d be more effective in addressing the DevOps and software issues we all experience. 

    Coming to Terms With Infrastructure-as-Code (IaC)

    Software engineering encompasses all the actions developers take to create software and technology products. This typically entails writing code, testing it and making sure it satisfies the original requirements for users. Operations is the practice of ensuring those software systems are ready to go and will be able to serve users on a day-to-day basis. Operations engineers aren’t as focused on writing new code as developers are; instead, they are focused on how existing software (mis-)behaves in existing systems.

    Now let’s cover some other key phrases we use and hear all the time in DevOps.

    Forward engineering means creating a system based on requirements that exist for that system. It entails first building a model, implementing it in software, then configuring and deploying it into operation. While the goal is for the new system to closely match the original requirements, the modeling process can be ambiguous, causing a system to perform in unanticipated ways. Furthermore, over time, the system itself may differ from the original model due to interaction with its environment. Consider that some parts can change the configuration of other parts; procedures performed by engineers shift the system state or configuration, too.

    Unfortunately, forward engineering offers no feedback with regard to the eventual success or failure of the product during the model creation process. These models and the subsequent software are created first and then deployed to test for success. Such a system can only provide feedback later based on its behavior, which can then be used to improve and build new models.

    Reverse engineering calls for understanding existing systems, usually by first creating a partial mental model by inspecting such a system and then using it to plan any modifications. Often, changes are made directly to the system without always fully understanding how it works. Feedback for changes is fast because putting the system through its paces and inspecting the results is relatively simple, easy and cheap to do. A good example of reverse engineering is software security professionals who scan network services or memory to identify anomalies and exploitable gaps and then try to manipulate their content for instant feedback.

    Then there’s infrastructure-as-code (IaC). This modeling technique began with declarations about how server operating system content should appear and was later adapted for the purpose of laying out and configuring cloud resources using API calls. While many vendors and products have come and gone, most tools have been declarative—with good reason.

    Anything to Declare?

    With declarative tools, you specify how things should appear without explicitly showing how you’ll get there. Why? It’s the easiest way to create models, templates and maps because engineers only have to declare how things should look, not get them into the correct form or place to actually do so. The emphasis is on evolving from iterative improvements that take place over time.

    Another feature of declarative IaC tools is idempotence, which means repeating an action multiple times with consistent results; the system only changes when it detects deviations from the model. This can be compared with the imperative approach, which requires developers to first find all the differences between a model and system and then write each step that’s necessary for the system to match the model.

    Declarative IaC tools eliminate the need for developers to write an implementation for these changes manually. These tools save thousands of labor hours by publishing templates that can be used repeatedly to build identical systems with less time and effort. The problem is that declarative tools rely on a forward engineering mindset—meaning there’s no way to get feedback from a live system—leading to many complaints from users.

    The Rift With Drift

    Configuration drift is what happens when a declarative forward engineering model no longer matches the state of a system. It can happen when a developer changes a model’s code without updating the systems built using that model. It could also be the result of an engineer that does exploratory ad-hoc operations and changes a system but fails to go back into the template and update its code. 

    Is it realistic to ban operators from ad-hoc exploration? Actually, some companies have policies forbidding any operator or developer to touch a live production environment. Ironically, when a production system breaks, it’s the first rule overridden: Ad-hoc exploration is welcomed by anyone able to get to the bottom of the issue. 

    Engineers who adopt IaC usually don’t like the work that comes with remodeling an existing system. Still, because of high demand fueled by user frustration, there’s a tool for every configuration management language—they just fail to live up to engineer expectations. The best-case scenario is that an engineer can use a reverse-engineered template to copy and paste segments into one but will need to manually write it elsewhere.

    Bridging Gaps

    What is needed is a new type of IaC tool. These must integrate reverse engineering at their core and allow feedback from living systems to update the very models that created them. In doing so, operators can adopt an ad-hoc change or simply revert back to the system as it was declared in the original model.

    Yet, today’s standard tools fall short of this goal, and aren’t much good for helping engineers solve system operations issues. True, there are those capable of auditing templates and singling out misconfigurations and mistakes. Still, while helpful, the greater value would be the ability to review a living system, not just the templates that engineers initially used to create it. 

    We have monitoring and observability tools and IaC tools, but in between them is a wide chasm. For effective DevOps, what’s required is a new breed of IaC tool, one that integrates reverse engineering practices that supports operations engineers and enables companies to bridge any development gaps.  

  • Pulumi Extends Infrastructure-as-Code Reach

    Pulumi Extends Infrastructure-as-Code Reach

    At its online PulumiUP conference, Pulumi announced it will make its Pulumi CrossCode translation technology available to third parties and DevOps teams. The technology was developed to integrate a wide range of programming languages with its infrastructure-as-code (IaC) platform.

    The company also announced it has added support for YAML—widely used to configure Kubernetes environments—as well as any variant of Java. It already supports .NET, Node.js, Go and Python programming languages.

    Finally, Pulumi also added packages for Oracle Cloud, Databricks and EventStore that simplify deployments on those platforms. The company already provides similar integrations for Amazon Web Services (AWS), Microsoft Azure, Google Cloud, Kubernetes, Auth0, CloudFlare, Confluent Cloud, Datadog, DigitalOcean, Docker, GitHub, Kong, MinIO, MongoDB Atlas, PagerDuty, Snowflake, Spot by NetApp and SumoLogic. There are also now components that provide support for container applications, Kubernetes clusters and serverless applications along with an AWS Cloud Development Kit (CDK).

    Pulumi has been making a case for an IaC approach that enables developers to use programming languages they already know to provision infrastructure as an alternative to having to deploy and master tools such as Terraform.

    The Pulumi CrossCode translation technology the company developed is now being made available to any organization that wants to add support for another programming language. It converts any infrastructure-as-code format, including Terraform, CloudFormation, Azure Resource Manager and Kubernetes configuration, to any programming languages supported by Pulumi.

    Pulumi CEO Joe Duffy said the goal is to extend the overall reach of the translation technology the company created to simplify provisioning any type of infrastructure.

    While many developers use tools such as Terraform, the need to limit misconfigurations is driving many larger enterprises to revisit their approach to IaC to improve software supply chain security, noted Duffy. Today, Pulumi claims more than 2,400 organizations have adopted platforms that enable developers to provision infrastructure and apply guardrail policies to reduce errors. That approach prevents developers from making configuration mistakes that can lead to exfiltration of data via, for example, a port that has been accidentally left open, he added.

    Too many developers assume that their cloud service provider is securing both the underlying platform and how it is configured only to discover later that it was their responsibility to validate those configurations. Security teams, meanwhile, can’t keep pace with the rate at which cloud infrastructure resources are being provisioned without some ability to apply policies that enforce rules for provisioning that infrastructure within the context of a larger DevSecOps workflow.

    Ultimately, the goal is to employ automation to resolve these issues once and for all as more organizations adopt DevSecOps best practices. In the meantime, the challenge is doing so in a way the average developer will accept and embrace. Otherwise, they will continue to find ways to work around security teams in the name of productivity, regardless of how insecure the underlying IT environment might become.

  • Russian Dev Exodus | Puppet IPO FAIL | Intel 18A Ahead of Sched.

    Russian Dev Exodus | Puppet IPO FAIL | Intel 18A Ahead of Sched.

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: More than 100,000 tech workers leave Russia, Perforce buys Puppet, and Intel moves faster than planned. (more…)

  • Pulumi Adds Business Critical Edition to Infrastructure Management Platform

    Pulumi Adds Business Critical Edition to Infrastructure Management Platform

    Pulumi today added a Business Critical edition of its Cloud Engineering Platform that provides large enterprises with additional security and compliance features, premium 24×7 support, training, professional services and a self-hosting option.

    In addition, the company announced that Pulumi Enterprise and Business Critical editions are now available via the online marketplace run by Amazon Web Services (AWS), with future availability on other marketplaces planned.

    The Cloud Engineering Platform from Pulumi enables DevOps teams to use Pulumi Packages to create reusable components for automating IT infrastructure provisioning. Those can be embedded within workflows and applications using an application programming interface (API) that Pulumi has exposed. IT teams can define a Pulumi Package in the programing language of their choice.

    Aaron Kao, vice president of marketing for Pulumi, said the Cloud Engineering Platform from Pulumi provides an alternative to managing infrastructure-as-code (IaC) that can be centrally managed more easily than open source tools like Terraform.

    Pulumi reported that adoption has accelerated across enterprises around the world with more than one million downloads per month. The company claimed it has seen 350% year-over-year growth of its enterprise customers, such as Mercedes-Benz, in the last 12 months.

    IT organizations in general are looking to bring some order to chaotic provisioning processes to avoid misconfigurations that often result in, for example, a developer leaving a port open; that open port leaves a vulnerability through which data can be exfiltrated.

    As organizations focus more on software supply chains in the wake of recent high-profile breaches, it’s only a matter of time before more questions arise about how cloud infrastructure is being provisioned. Much of the concerns organizations already have about cloud security don’t stem from the platforms themselves. Rather, it’s the processes employed to provision infrastructure under a shared cloud security responsibility model that results in so many vulnerabilities. Developers often assume the cloud service provider is securing configurations, only to discover later that it was their own responsibility to validate those configurations. Security teams, meanwhile, can’t keep pace with the rate at which cloud infrastructure resources are being provisioned.

    Centralizing infrastructure management is especially important as the number of cloud platforms IT organizations manage continues to expand and the rate at which applications are built and deployed shows no sign of slowing, noted Kao.

    It’s not clear whether development or IT operations teams will ultimately wind up managing the provisioning of infrastructure. Pulumi is making a case for a platform that enables those teams to more closely collaborate within the context of a set of policies and DevSecOps best practices that the IT organization defines.

    It may be a while before the bulk of enterprise IT organizations are able to exercise the political will to regain control over IaC provisioning. However, in the absence of guardrails that allow IT organizations to make sure infrastructure is properly configured, it’s only a matter of time before tougher questions are asked about how software supply chains are managed.

  • Updating and Managing Infrastructure-as-Code (IaC)

    Updating and Managing Infrastructure-as-Code (IaC)

    Infrastructure-as-code (IaC), often embodied by open source Terraform, is an essential ingredient to cloud and cloud-native strategies. But without the ability to scale, secure and manage IaC, you very quickly experience drift. Tim Davis, DevOps advocate with env0, pronounced “N zero”, discusses establishing a single source of truth and reigning in drift by updating and managing infrastructure as code. The video is below, followed by a transcript of the conversation.

    Announcer: This is Digital Anarchist.

    Mitch Ashley: I have the pleasure of being joined by Tim Davis. Tim is DevOps Advocate with env0, and so, we’re going to be getting into some infrastructure as code type discussions, so, looking forward to that. Welcome, Tim—good to be talking with you.

    Tim Davis: Same for you, Mitch. Great to meet you. Thanks for having me.

    Mitch Ashley: You, too. I love your cloud off to your right shoulder, by the way. [Laughter] Anyways, so—introduce yourself, tell us a little bit about env0.

    Tim Davis: Absolutely. So, I am the DevOps Advocate with env0. dnv0 is a TACoS platform, if you will, the Terraform Automation and Collaboration Software space. We really focus on infrastructure as code, automation, teams and governance, and other tools to help you with the life cycle of your infrastructure as code environments.

    Mitch Ashley: Cool. Good. Well, you know, it’s a great conversation to have, because something that I observed always happens in our industry is, you create a new technology, it’s kinda easy to get started and starting to use, sometimes it isn’t always easy, but the complexity comes—alright, how do I scale this? How do I do this across a larger development team or multiple development teams? How do I operate it? How do I secure it through the day two kinda stuff?

    Tim Davis: Right.

    Mitch Ashley: And I wouldn’t be surprised if you told me the same is true for infrastructure as code, and in fact, I know that to be true.

    Tim Davis: [Laughter] Exactly right.

    Mitch Ashley: Let’s talk about that. What are some of the challenges once people get into doing infrastructure as code, particularly with Terraform? What are some of the challenges that you run into?

    Tim Davis: Yeah, and most of those kind of lie around the visibility and kind of the control space of that. Infrastructure as code is great. It is a fantastic way of speeding up, kind of pushing yourself into the future for DevOps, getting into that real GitOps life cycle. But with that, you know, if it usually just starts out with either one developer or one infrastructure person that starts running it locally on their laptop and figuring it out, it’s great, it does what they want.

    As soon as you start to scale that methodology and doing it locally to multiple people, that’s when you start losing that visibility of, “Where are my state files stored? Who just deployed something into the cloud? What variables might they have used for that?”

    So, it really becomes kind of an, “I don’t know who’s doing what in my cloud anymore.”

    Mitch Ashley: Mm-hmm.

    Tim Davis: So, really making sure that you’re staying on top of that, kinda centralizing everything to maintain that visibility is very important when you’re scaling it.

    Mitch Ashley: Let me ask you, then—is it fairly typical that within one software Dev team you have a person or sort of a close enough coordinated group of people that are doing the software as code, infrastructure as code. And then the confusion may exist as you start to go across multiple teams, or does it also occur within one team, just sometimes, it’s every developer for themselves and so—

    Tim Davis: Yep.

    Mitch Ashley: – you could have a wild, wild West kinda situation before you know it.

    Tim Davis: [Laughter] And the answer of that, of course, is yes, I mean, it’s all of the above. It could be even just two people on a small team that start to use these tools and they start to step on each other’s toes by having deployments that kind of counteract each other. Or you have it where somebody is deploying out into the cloud using your infrastructure as code and somebody else is just going into the cloud and clicking around and deploying stuff manually where you get something called drift, where what the infrastructure as code says is out there isn’t necessarily matching with what’s actually there.

    Mitch Ashley: Mm-hmm.

    Tim Davis: So, it can be across a single team, multiple teams—any time you really scale it past a single person on a laptop is really where you start to see those issues.

    Mitch Ashley: Yeah, okay. It’s a familiar problem, right?

    Tim Davis: [Laughter] That’s exactly right.

    Mitch Ashley: We’ve had this in other things, too. Well, what are some of the best practices or kind of good infrastructure as code hygiene to keep your worlds at least sane to start with so you—the first question is always, “What’s there?” If you don’t know what’s there, then—

    Tim Davis: Right.

    Mitch Ashley: – everything is up for grabs, or you question everything because you just don’t feel like you have a good grounding of what you’re looking at and what you’re dealing with. What are some of those best practices?

    Tim Davis: And I really—I try to stay away from the best practices thing, because sometimes what’s good for somebody isn’t great for somebody else, but that—

    Mitch Ashley: Well, and I say that as best practice is not universal, right?

    Tim Davis: Absolutely. And there are some things that you can do to help yourself out, for sure. If you’re using infrastructure as code, having the single source of truth can be very helpful for everybody, essentially stopping that drift that I just mentioned. So, if you are using infrastructure as code, save them in a repository somewhere. Make that the single source of truth where whatever’s in there, whatever’s in those infrastructure as code files, that’s what’s there.

    So, whenever you decide to go for infrastructure as code, stop using the cloud UI, stop going around and click, click, clicking and adding resources and things. Do it by updating the infrastructure as code file and then re-deploying. That can really help you make sure that you’re not causing any problems, nobody’s stepping on anybody’s toes, and you know if I see it in the files here in this one spot, that’s what’s out there.

    Mitch Ashley: Mm-hmm. Yeah, I totally agree with you. That makes a lot of sense to me. What are some of the complexities, then, you have to deal with? Because now we’re working an environment where infrastructure, the stack, the distribution of that, the locations, whether geographic or service provider really can change instantly without necessarily the full development team, which is an advantage, right? You want that sort of abstraction.

    Tim Davis: Exactly.

    Mitch Ashley: But it also can be—introduce new variables and maybe even new problems.

    Tim Davis: Yeah, and it could be new problems, it could be the same old problems, or it could be new problems that you kinda deal with the same, old way. And, you know, adding controls of making sure you’re adhering to policy, like, you’re only allowed to deploy to certain regions or you’re only able to deploy a certain number or size of instance, making sure that you have some form of role based access control there to know, you know, this person is allowed to deploy straight out to development, no problem, but if it’s going to production, it needs to wait for approval and somebody from either the DevOps team or the SRE team needs to validate everything first before we actually make that push.

    Mitch Ashley: Mm-hmm. I think something else that always happens, too, is in—you know, I’m a security guy as well as a software guy, so, I always worry about the security side of it. Now, automation can make that, you know, much better.

    Tim Davis: Exactly right.

    Mitch Ashley: Much more reliable, less human error. There’s a lot of good things that come with infrastructure as code from a security standpoint, but it also can be, it’s sort of security is in the eye of the scripter and the person who’s—

    Tim Davis: Right. [Laughter]

    Mitch Ashley: – doing, you know, configurations or doing the scripting, too, which can be good or bad. How do you address security in this kind of a world?

    Tim Davis: Yeah, and this is just one of those things where we’ve all heard the term shift left, and shifting all of these different things—security, performance, you know, billing—everything, if you shift it left into the deployment process, it really helps if you foster that communication. I mean, we know that old school term of silos where one hasn’t isn’t talking to the other hand. If everybody gets involved, if the security team is involved with writing the policy, they’re involved with double checking that it is implemented into the deployment process so that you don’t have to go and fix a problematic deployment later, it’s just stopped before it starts. That can really make sure that everybody kind of gets what they want, but security folks still have to be involved. They are still in charge of the overall process and procedure and policy, it’s just, it’s kind of a new way of getting that implemented and at what stage of the life cycle you’re implementing it.

    Mitch Ashley: Do you think it shifts—and this is actually, I just did a talk on this about shifting left of, we wanna shift everything left, which is generally a good idea, right?

    Tim Davis: Right.

    Mitch Ashley: To be able to address some of those things early and design it in, design security in, et cetera. Oftentimes, it’s interpreted as, “Let’s put that on the developers, too. Let’s have them—the developers will worry about that. The developers will worry about all those.” Now, it’s pretty much, it could be everything, sort of a ridiculous request, right?

    Tim Davis: Right.

    Mitch Ashley: Because that’s not a developer’s expertise to necessarily know all those things. What is your—when you say shift left or infrastructure as code—

    Tim Davis: Right.

    Mitch Ashley: – in practical terms on a software team, what does that look like?

    Tim Davis: Yeah, and a lot of folks think that that just means, “Hey, I’m taking away your job as the security guy and I’m gonna give it to the developer” and that’s not the case. The developer, they may be a little security conscious, they may understand the infrastructure or the networking piece, but nowhere near as much as the career security guy or the career networking and infrastructure guys. It just kind of brings them in and fosters that conversation.

    I definitely think the developers are going to be part of the conversation, because it is bringing it into their tooling, into their languages, into their life cycle, but it still requires that expertise to be able to make sure that it is done correctly, it’s implemented correctly, and that it’s being checked the way it needs to be done.

    Mitch Ashley: Mm-hmm. So, does it look like security engineers, to use that as a term, they’re sitting down with the developers, saying, “Okay, how are we configuring the environment?” and, “Here’s how you’re building your Terraform configurations”—going through that with them?

    Tim Davis: I think so.

    Mitch Ashley: Is it saying, “Here’s the principles we want you to follow”—

    Tim Davis: Yeah.

    Mitch Ashley: – “and as you’re creating this, please do these things, and then we don’t have to sit on your shoulders”?

    Tim Davis: I definitely think there’s lots of different ways you can do that. I mean, obviously, you know, having your standup meetings, making sure that they are kinda telling them, “This is what the policy should look like, however you want to implement it is fine, but this is the kind of processes and procedures, these are the rules that we need to check.” So, it can definitely be done however works best for everybody, but as long as you’re having that communication and you’re making sure that everything’s going through—yeah, you know, you can always do it different ways and be successful, still.

    Mitch Ashley: Mm-hmm. Interesting. What are some of the things that you’ve seen change? You know, there’s been a lot of acceleration over the last 12 to 18 months.

    Tim Davis: Right.

    Mitch Ashley: And, you know, I’ve heard people talk—of course, you always hear about the, as much digital transformation happened in the last year that was planned for five years.

    Tim Davis: Yeah.

    Mitch Ashley: And there is some truth to that, and I’m sure some exaggeration, too. But there’s a strong belief that, you know, people are now thinking they can deploy applications much, much faster and they’ve proved it in this last window. Have you seen that happening?

    Tim Davis: Yeah, absolutely, and you know, technology changes. They used to say it doubles every six months or what have you, and really, we are iterating and changing and kind of moving forward very, very quickly. A lot of times, we get a new tool and it’s great and then people are like, “Alright, I’m gonna pull this tool in and automate it” and then they figure out, “Oh, there’s some extra complexity here and we’re having to go back through and figure it out.”

    I definitely think there’s a lot changing, you know, even just the shift from on prem legacy, I guess, enterprise architecture through to new, like, cloud native architectures and things like that. We’re moving fast, we’re able to change and pivot and do what we need to do, but as long as we make sure that we’re kind of keeping grasp of what we’re supposed to be doing from either a security or an infrastructure construct, we can make sure that we can iterate and change and adapt as fast as possible without causing any major issues down the line.

    Mitch Ashley: Mm-hmm. Cool. What do you think the next 6 to 12 months look like? If you had to put on your crystal ball, your Magic 8 Ball, that says, “Ask Again”?

    Tim Davis: [Laughter] Yeah, I love to look into that. Obviously, it’s good to see, you know, what am I gonna have for lunch next week, but also, you know, looking into the industry of what’s gonna happen. We’re seeing more and more X as code. We’ve got infrastructure as code, you’ve got security as code, you’ve got policy as code tools. I think we’re gonna see more and more and more, you know, performance as code and things like this that just are able to bring those day two plus operational tools closer left into the deployment cycle and have all of these single source of truth in the repositories of, we know exactly what our infrastructure is supposed to look like, we know how it’s supposed to be secured, we know how it’s supposed to perform. All of this is just declaratively set up there. I think we’re gonna see a lot more of that going forward.

    Mitch Ashley: Mm-hmm. Well, certainly, as you said, it follows the “everything as code,” right?

    Tim Davis: Exactly right.

    Mitch Ashley: And if we’ve learned nothing else, I think that’s been a lesson of the past 12 to 18 months, right? It is software, it’s all about software.

    Tim Davis: It absolutely is.

    Mitch Ashley: What do you think some of the skill challenges are for people as we continue to evolve this and move into this rapid world, shifting things left, doing more infrastructure as code, and maybe asterisk as code—everything? You know, it’s always the self-learners can be at the edge of the curve, other folks have other ways of learning. How do we bring everyone along in terms of their skills and their learning, their development?

    Tim Davis: Yeah, and that’s definitely something. I mean, I ran into myself. I’m an infrastructure guy at heart, that’s where I’ve been, you know, my whole career, and eventually, someone was telling me, “Hey, this Python thing might take off” or, “I’ve heard of this thing called Terraform and it’s basically like code, but it’s our job.” And, you know, just realizing that nobody’s trying to take away your job as a security guy or an infrastructure guy or girl or what have you. They’re not trying to take it away from you and move it over to a developer, it’s just one of those things where you need to adapt to move forward faster.

    Now, it’s not just infrastructure folks that are learning to code, it’s also developers that are kind of learning infrastructure and figuring out how can I optimize my application against this infrastructure a little better? How can we work together to move forward faster? So, it’s kind of a melding of skills from different, you know, old school silos now to work together to solve the problem.

    Mitch Ashley: I made that recommendation to a network and security engineer about 10 years ago. It’s like—learn Python, and he looked at me like I had six eyes instead of two.

    Tim Davis: [Laughter]

    Mitch Ashley: And he came back from the Cisco conference, he goes, “Now I know why you said that.” [Laughter]

    Tim Davis: Exactly right. And that’s a great piece of advice even today in 2021. I mean, if you’re in infrastructure or you’re a network tech or you’re a security person, look into Python. It’s not as hard as you think and you can do so much with it. It’s a fantastic way to kind of break into that DevOps automation type space.

    Mitch Ashley: I like that, too, and not that we’re trying to make everybody a developer, but if it’s not a skill you have, it sort of leaves out this whole domain of things that you can either do to understand what’s happening—

    Tim Davis: Right.

    Mitch Ashley: – or do to automate or et cetera. It’s sort of like you’ve got one arm tied behind your back, why don’t you have two out in front where you can fully work?

    Tim Davis: Yeah. And it also helps from a cohesive understanding between groups. Back when I was a dedicated infrastructure person, myself and the development team didn’t care for each other, because we all were trying to get in each other’s way, it seemed. We were trying to solve the same problems, just going about it in different ways, because we didn’t necessarily understand the methodologies or how those people were trying to get their work done.

    If you as an infrastructure, security, or other operations person start to kind of learn the development methodologies and figure out how things work, it’ll help you to communicate and solve problems across teams better, simply because you kinda see where they’re coming from with their tool sets and the way that they do business.

    Mitch Ashley: Great. Well, it’s been a great conversation. I’m glad we got to explore this path together.

    Tim Davis: Absolutely.

    Mitch Ashley: Where can folks find out more about env0? I know you have a kind of funky spelling of env0.

    Tim Davis: [Laughter] Yes. We are at env0, E-N-V and the number 0.com. You can find us on Twitter @E-N-V-Z-E-R-O and on YouTube at E-N-V-Z-E-R-O.

    Mitch Ashley: Great. So, environment zero, right?

    Tim Davis: Exactly right.

    Mitch Ashley: Perfect. Well, hey, it’s been fun talking with you and hope you’ll come back again.

    Tim Davis: That sounds great. I look forward to it. Great to meet you, Mitch. Thanks for the time.

    Mitch Ashley: Alright, take care.

    [End of Audio]

  • Infrastructure as Code (IaC): Why Collaboration Is Key

    Infrastructure as Code (IaC): Why Collaboration Is Key

    Infrastructure as Code (IaC) is an essential DevOps practice that allows developers and operations teams to automatically manage and provision the infrastructure through code, instead of making configuration changes through a manual process. IaC provides transformational agility and can significantly speed delivery, reduce costs and help mitigate risks.

    However, when people talk about IaC, sometimes they forget to mention its main ingredient: collaboration. The whole purpose of IaC is to enable infrastructure and software development teams to collaborate.

    “It (IaC) is about making operations more developer-like,” said Rob Hirschfeld, co-founder and CEO at RackN, a software automation company that specializes in making IaC collaborative for teams, tools and tech.

    Rob will be joining the Predict 2022 speaker lineup to talk about Infrastructure as Code and DevOps. During his session, “Adding IaC’s Missing Ingredient: Collaboration,” Rob will discuss how to help operations teams become more collaborative and a cohesive part of the business, and how IaC helps create and reuse automation within the system.

    Rob will take a look at the future of IaC, what’s in store for infrastructure and operations teams in 2022 and share his own predictions.

    During a quick interview with our very own Jackie Whetzel, Rob gave us a sneak peek of his session and shared one of his predictions for 2022.

    “People are increasingly nervous about complexity but they can’t define it, that’s what makes it so cloudy,” said Rob. “The number one thing we really need to start doing is figuring out how to talk about complexity and how to manage and tame it. And I’ll give you a sneak peek. Come to my presentation, we’ll talk about using collaboration and teamwork and pipeline as ways to tame that complexity.”

    Predict 2022, taking place Jan. 19-20, is a free virtual event featuring educational presentations that explore the trends, technologies and looming topics that could have the most impact on technology in this new year. Internationally recognized industry leaders, such as Rob, will share their speculations on what the future holds in the realms of DevOps, cybersecurity, digital transformation, cloud-native and IT.

    To check out the full agenda and to register, please visit the Predict 2022 website.