devopsy.kinsta.cloud

Tag: continuous deployment

Choosing the Right Software Deployment Tools
Imagine this scenario: After investing countless hours in developing your software, it is finally ready for deployment. Now, you must select the appropriate software deployment tools to guarantee a seamless release.

This article will guide you through the concept of software deployment and discuss various tools that cater to different needs—from dedicated deployment tools to comprehensive CI/CD pipeline solutions and GitOps approaches. By understanding these categories and their benefits, you’ll be well-prepared to make an informed decision on which tool is the best fit for your organization’s needs.

What is Software Deployment?

Software deployment plays a crucial role in the software development life cycle. It involves packaging, distributing and installing an application on target systems or environments, aiming to make your software accessible and usable for end-users seamlessly.

In today’s fast-paced world, companies must be able to deliver applications rapidly and reliably. A well-planned software deployment strategy ensures that new features reach users promptly while minimizing downtime and errors during updates.

What to Look For in a Software Deployment Tool

When selecting the ideal software deployment tool, consider the following factors:
- Automation: The tool should automate complex software deployments, making the deployment process faster and more efficient.
- Continuous deployment: Continuous deployment tools allow you to deploy software repeatedly, ensuring consistent deployments. Choose a tool that automates release management and integrates automated builds throughout the development process.
- Scalability: The tool should be capable of handling large-scale deployments and distributing software across multiple servers. Opt for a tool that enables applications to deploy to multiple servers simultaneously.
- Integration: The tool should integrate with version control and other third-party tools, making the deployment process seamless. Look for a tool that integrates with modern DevOps processes and connects automated builds to configuration management.
Full CI/CD Tools

Bamboo

Bamboo by Atlassian is a robust tool that integrates seamlessly with other Atlassian products, such as Jira and Bitbucket.

It offers first-class support for multiple languages and platforms while providing built-in deployment projects for smooth releases.

GitLab CI/CD

GitLab CI/CD, an integral part of GitLab, simplifies pipeline management using YAML configuration files stored within your project repository.

This approach enables easy collaboration among team members while ensuring consistent build environments across all stages of development.

Codefresh

Codefresh is a cloud-native solution designed specifically for Kubernetes deployments, It provides a modern interface to manage pipelines visually, along with advanced real-time debugging capabilities.

Dedicated Deployment Tools

Jenkins

Jenkins was originally conceived as a continuous integration (CI) tool, but many organizations use it mainly for deployment. It is an open source automation server that provides support for building and deploying software projects, as well as to schedule and run tasks at regular intervals. Learn more about Jenkins in this in-depth blog post.

Harness

Harness is an intelligent software delivery platform that simplifies deployments through automation and machine learning. Their continuous delivery-as-a-service (CDaaS) approach helps you deploy faster while minimizing risks and errors.

AWS CodeDeploy

If you’re using Amazon Web Services (AWS), AWS CodeDeploy might be a perfect fit for your needs. This completely managed service automates code deployments to any instance or serverless Lambda function in AWS, guaranteeing consistent results every time.

Spinnaker

An open source project backed by Netflix and Google, Spinnaker offers powerful multi-cloud continuous delivery capabilities for deploying applications at scale across various cloud providers like AWS, GCP, Azure, Kubernetes clusters, etc.

GitOps Tools

Argo CD

Argo CD is a declarative, Kubernetes-native continuous delivery tool that uses Git repositories as the source of truth for application definitions. It simplifies rollbacks, provides audit trails and enforces consistency across environments.

Flux CD

Flux CD, another Kubernetes-focused option, automates deployments by monitoring your code repository for changes and applying them to your cluster accordingly. Bonus: It supports multi-tenancy and integrates with Helm charts.

Weave GitOps Core

Lastly, Weave GitOps Core offers an open source foundation for managing Kubernetes applications using the power of GitOps workflows. This versatile tool can be extended to fit various use cases while maintaining simplicity in its core functionality.

Conclusion

There are many software deployment tools available that can help automate the software delivery process, reduce errors and improve collaboration between development and operations teams. From open source tools like Jenkins to cloud-native solutions like AWS CodeDeploy, each tool has its own strengths and features that make it suitable for different use cases.

When selecting a software deployment tool, it is important to consider factors such as compatibility, automation, scalability, security, monitoring, deployment environment compatibility, integration with existing tools and support and documentation. By carefully evaluating these factors and choosing the right tool for your specific needs, you can ensure that your software deployment process is efficient, reliable and successful.

Image Source: Freepik
June 21, 2023
6 Ways To Empower Developers and Increase Productivity

A big goal of DevOps is to increase productivity. Yet many organizations still struggle with the efficiency of their software release processes. In fact, 54% of engineers stated they have too many manual steps within their deployment process, according to an Armory survey. And with the pervasiveness of multi-cloud, developers now face an escalating number of deployment environments, each carrying their own nuances.

Development teams want to move as quickly as possible without barriers. Yet, there are often hurdles to seamless deployments. For example, the code might be 100% ready to deploy, yet someone still must validate it before it reaches production. Manual bottlenecks like this in the CI/CD process can make developers feel like their code isn’t worthy of deploying and may even amplify feelings of burnout.

I recently met with Anna Daugherty, senior product marketing manager at Armory, to see what technology leaders can do to remove barriers within their software release cycles and increase productivity. According to Daugherty, many of these strategies boil down to having empathy for developers and their needs. “Developers aren’t machines; they’re people, and people deserve to have a quality of life when they’re doing their jobs.” Below, we’ll highlight tips and tricks to keep in mind to improve release fluidity.

1. Identify Where Your Manual Checks Are

The first step to speeding up CI/CD is identifying where manual steps exist within the development workflow. The DevOps life cycle, often visually depicted as an infinity loop, includes many states, such as design, build, test, deploy, assess and monitor. But if each step has one or more manual steps, you accrue a lot of manual debt, said Daugherty.

Therefore, engineering leads should seek to remove manual checks whenever possible. Of course, this may not be realistic within certain situations, such as financial institutions that require manual approvals at the last mile. However, everything should be smoothed out and automated up to that point, said Daughtery.

2. Shift From Continuous Delivery to Deployment

Depending on who you ask, the ‘CD’ in ‘CI/CD’ may stand for continuous delivery or continuous deployment. As part of reducing friction in the software release process, Daughtery encourages teams to begin to embrace the latter whenever possible. If deployment remains inhibited by unnecessary manual steps, it can decrease scalability and interoperability.

3. Increase Confidence With Testing

Of course, if we introduce more automation into the CI/CD pipeline, we must ensure the software is rigorously tested for errors and vulnerabilities. Daugherty encourages development to embrace automated, continuous testing approaches at every juncture, whether pre-production, inter-production, or throughout the application lifecycle. There are also advanced deployment strategies to consider, such as canary testing, which rolls out new changes only to a small subset of users or blue-green deployment can deploy to different places at different times.

These deployment strategies can help build confidence while reducing the risk of disrupting a majority of your user base with untested production updates. “The more you can integrate testing into your process, the more confident your process will become,” said Daughtery.

4. Chose The Right Toolsets

So, how do we implement more automation into the DevOps cycle? And what sort of tools should we look for? According to Daugherty, when scouting for DevOps tools, technology leaders should prioritize open solutions that aren’t closed-off environments. Instead, they should offer flexibility and the ability to integrate with the tools you already use every day. “You can have all the tools in the world, but if they don’t fit together, that can drag down the process.”

5. Consider Open Source Software (OSS)

On that note, consider open source software (OSS) where possible. OSS like Argo or Spinnaker has the potential to empower developers and bridge gaps in the CI/CD process. Other projects from the Cloud-Native Cloud Native Computing Foundation (CNCF) and Continuous Delivery Foundation (CDF) are mature with proven stable adoptions.

And although there is some weariness around OSS, particularly around unknown vulnerabilities and project maintenance concerns, “you can still be successful, secure, and scalable with open source,” said Daugherty. Open source also delivers community benefits and helps engineers feel like they are part of a bigger picture. “[OSS] is the opposite of burnout. It brings people together. It’s a community builder.”

6. Become Developer-First

Transitioning into a developer-first organization is another way to increase developer satisfaction and efficiency. But to become a developer-first organization, developers will need more say in the processes—this will involve balancing top-down requirements with total developer control. For example, leaders should suggest tools to enhance productivity but accept feedback on an ongoing basis to see what works best.

To support these efforts, the concept of developer experience (DX) has entered the lexicon in recent years. Building with developer empathy can result in more usable, frictionless software that increases productivity. DX is crucial within the burgeoning area of platform engineering, as the goal is to create reusable, internal self-service platforms for developers to leverage.

Interestingly, Netflix extolls what it calls full-cycle development, in which a developer oversees the building and operational aspects of a single component throughout its entire life cycle. “It’s an interesting way of addressing burnout because you feel more connected to what you create,” said Daugherty.

Final Thoughts: DevOps Isn’t Dead

Above, we’ve highlighted some potential strategies to increase developer productivity. By replacing manual bottlenecks with more automation, you can increase deployment fluidity. Considering how tools are acquired and iterating upon feedback from engineers can also help improve consistency while retaining quality talent.

Looking to the future, Daughtery hopes that similar roles such as DevOps, site reliability engineering and platform engineering can come together and join forces. Because, at the end of the day, they share many goals. And contrary to some grabby headlines, DevOps is far from becoming obsolete. DevOps is not dead. It’s very much alive and increasingly important.

January 20, 2023
How DevOps Helps With Secure Deployments

DevOps is an approach to software development that emphasizes communication and collaboration between teams. It’s best known for bringing together people who previously worked in separate areas like engineering or testing; this allows them all to learn from each other as they work on projects together.

DevSecOps helps organizations monitor and discover security risks throughout the development process rather than spending a lot on post-launch protective measures. This way, they are able to secure their product without overburdening developers with heavy-duty responsibility for what might turn out as an easy fix given time or knowledge differentials between team members—after all, they are responsible for building these things!

The Goal of DevSecOps

The goal of DevSecOps is to provide security best practices in a way that doesn’t disrupt team productivity.

Secure development is the key to a smooth deployment process. It’s frustrating when you have security in your product but don’t see it being implemented or taken seriously. DevSecOps brings back that focus by reminding everyone just how important and necessary good hygiene practices are for both developers and operations.

For security to be more effective and reliable, it should be incorporated from the very beginning. This means that instead of waiting until there is an issue or crisis before implementing measures for protection such as firewalls, encryption keys, etc.; you want your developers working on it up front so they can ensure everything will work well together later.

It helps ensure security issues are found as early in the process as possible so it is close to decision-makers. It’s much easier (and less painful) when security issues can be fixed while you still remember what happened with your project—it’s like having an extra set of eyes on the project.

Advantages of DevSecOps

Security should be a high priority for any company looking to stay afloat in this digital age. From reducing risk to mitigating legal liability, identifying bugs earlier on during the project life cycle makes security management easier when things go wrong. Security becomes less painful as well since stakeholders will understand better why delays happen instead of just because “the team found issues.”

If you shift your focus to include this earlier in the process, then costly issues can be discovered much faster and closer to where they happen, which will allow for more timely responses from developers or infrastructure engineers who must address those problems before moving on to another task.

The goal of this system is to give Dev and Ops faster feedback on the work they are producing so they are notified right away.

Achieving DevSevOps

There’s no one right answer when it comes to implementing DevSecOps, as the needs of each organization will be unique. However, there are some common practices that can help to get you started.

Get Management Buy-In

To make a fundamental shift in how your organization thinks about, builds and deploys software, you will need more than just peer-to-peer agreement. Management buy-in is necessary for this revolutionary change of mindset.

The DevSecOps process is one that can help combat software vulnerabilities being exploited in production. It’s important for those who make decisions on risks versus revenue to see how this will benefit the company, or else it’ll be fruitless.

Security is something that we should all be responsible for. It’s not just a concept or idea, but rather something tangible and real—something which can affect your company greatly if handled incorrectly by someone who doesn’t know what they’re doing. That’s why it always pays off in the end when employees take their own security concerns seriously; this will result in better decisions being made about how best to protect your assets from outside threats.

You can’t just implement a new policy without management backing it up.

Developer Responsibility

DevSecOps takes security seriously by making it a part of the development process, not an afterthought. The entire team can work together to write more secure code and configurations that are robust against threats while also being easily reversible if something goes wrong—because they’re writing this stuff right from their own desks!

It’s not just a job for security professionals anymore, either. It needs everyone from the CEO to contract workers to take responsibility for securing their own work instead of outsourcing the responsibility and expecting others to do it retroactively every time there is an attack or threat.

Security and DevSecOps training

There’s no quick fix for organizational culture, but it can be done. People are always resistant to change. There will initially be frustration as traditional behaviors are questioned and you create new ways of working that might not fit everyone right away.

Managers who have a good understanding of these concepts will be able to alleviate some problems, and so will experienced engineers and other staff. It’s important for everyone involved in the process (from top executives down) to buy into this idea because it can’t succeed without their support.

When everyone understands the DevSecOps approach, they’ll not only understand why it’s important but also see how to properly implement this new way of doing things.

Implementing the Proper Tools and Processes

While it is often easier said than done, getting buy-in and detecting security issues early in the pipeline sounds great. But achieving DevSecOps integration does not happen overnight, so you should start by looking at what you can do today to move this process forward more quickly.

But there isn’t just one right way of achieving DevSecOps—every organization has its own unique methods for building software that needs to be considered when implementing these concepts across different organizations with varying cultures and workplaces.

Key Points

By introducing a new security component to the pipeline, DevSecOps is expanding our fundamental development and operations components with the DevOps best practices. DevSecOps is the future of software development. It ensures that security risks are monitored throughout all stages, not just after launch time when it’s too late to fix vulnerabilities and bugs before someone else exploits them for their own gain!

November 4, 2022
Pulumi Adds Deployment Capability to IaC Platform

At the Pulumi Cloud Engineering Days event, Pulumi announced today it has added a code deployment capability to its Pulumi Cloud platform for managing infrastructure-as-code (IaC).

Pulumi CEO Joe Duffy said Pulumi Deployments will make it possible for DevOps teams to provision infrastructure and deploy applications via a single Git commit that can be invoked via an application programming interface (API) or graphical tool.

A Git Push to Deploy capability allows DevOps teams to connect to any Git repository to drive deployments for a given project path, push to any given branch and set up GitOps workflows such as pull request reviews. Deployment actions include both updates and destroys in addition to providing drift detection and remediation capabilities.

The Deployments and Automation API Pulumi developed also enables ephemeral code review environments, automatic cleanup and deployment orchestrations across multiple cloud platforms using a library of templates Pulumi provides. In effect, Pulumi is now providing a GitOps workflow out of the box, said Duffy.

Pulumi has already integrated these capabilities with GitHub and plans to add support for GitLab, Atlassian Bitbucket and other DevOps platforms. The goal is to make it possible for DevOps teams to effectively manage ten times the amount of cloud infrastructure resources per engineer using GitOps workflows than is currently possible relying on IaC tools such as Terraform, said Duffy.

As an alternative to Terraform, the Pulumi platform has been gaining traction because it makes it possible to more easily centralize the management and provisioning of infrastructure using a set of embedded guardrails. One of the major issues enterprise IT organizations are encountering these days is the number of misconfigurations created when developers use tools like Terraform to provision infrastructure on their own. Those misconfigurations often lead to, for example, ports being left open and cybercriminals exfiltrating data. In the wake of a series of high-profile breaches, more organizations are now reviewing the security of their software supply chains including how infrastructure is provisioned by embedding policy-as-code into DevOps workflows.

At the same, organizations are also keenly interested in increasing the productivity of DevOps teams during an economic downturn that limits the number of IT professionals an organization can afford to hire, added Duffy.

Pulumi claims there are now more than 5 million users of its platform spanning more than 1,000 organizations. Less clear is the degree to which organizations are embracing GitOps workflows to unify deployments on IaC and application code. As platforms make it easier to make that transition, the number of organizations using a more opinionated approach to DevOps should steadily increase.

In the meantime, DevOps teams are going to be required to find ways to build and deploy safer applications faster. Much of the focus in recent years has been on accelerating application deployments with little regard for cybersecurity. Those days are clearly now coming to an end.

November 2, 2022
Four Causes of Technical Debt in DevOps

Ideally, DevOps should retain a lean footprint, but avoiding technical debt is easier said than done. As such, over half of IT leaders report technical debt is a big or critical problem. Without routinely addressing technical debt, DevOps teams can easily face inconsistencies during deployments. Versioning can get out of hand without consistent upgrades and aging legacy components can be a roadblock to development fluidity.

Simply put, letting too much technical debt pile up isn’t all that great.

I recently met with Kurt Andersen, SRE architect at Blameless, and Matt Davis, SRE advocate, to gather insights on the sources of technical debt in the field of DevOps. According to Davis, “Tech debt is something that you really want to keep chipping away at.” Below, we’ll highlight some common causes of technical debt and suggest ongoing practices to help tame it.

1. UI-Based Deployment Patterns

UIs are great for quickly configuring DevOps, but if your deploys aren’t codified, this reliance on UIs could come back to haunt you in the form of technical debt. A lot of the time, explained Davis, infrastructure teams have a tendency to just use the GUI of their given cloud provider to initiate deployments. Perhaps the team is wrapped up in feature velocity or a deadline to release.

For example, you could use a UI to quickly create a database, but this is likely not a repeatable and robust process. Especially as an organization scales its cloud adoption and uses multiple clouds, you start to produce “snowflake” deployment patterns across teams, says Andersen. This could lead to inconsistencies around different deployment options per region.

It’s not easy to catalog a configuration if it’s not written in code, and the more undocumented UI-based deployment patterns are used, the more silos of knowledge you create. “If not codified with Terraform, you have to document what you did—code gives you that documentation,” said Davis.

Solution: Rely less on the GUI and codify your deployment patterns with infrastructure-as-code (IaC).

2. No End-of-Service Life Cycle

Development is typically more concerned with feature velocity than planning for service deprecation. Yet not envisioning the entire life cycle or setting an off-ramp from the start can produce more technical debt later on. “No one pays attention to planning for retirement, and at the end-of-life, services are hard to retire,” described Andersen.

What you end up with are these “senile services” that aren’t doing that much but remain critical to the business’s operations. These services may be challenging to migrate. Or, they may be the product of unknown shadow or zombie APIs. For example, a marquee customer may depend upon a certain legacy endpoint, making the service difficult to evolve.

Not planning for the end of the service lifecycle can hold up your development from more efficient ways to work. For example, if one “senile service” has never used a particular commit strategy before, it could stall agility when revamping the deployment pipelines, says Andersen. Thus, it’s best practice to set clear deprecation timelines and plan for the future.

Solution: Plan the end-of-service life cycle from the upstart.

3. Lengthy Dual-Track IT

Another area that can produce technical debt in DevOps is dual-track IT. This is when engineering teams maintain one core stack in addition to an experimental structure and shift toward adopting the experimental structure over time. Development is constantly overseeing migrations to new technologies, yet they rarely plan for the coexistence of both stacks.

While splitting an old and new stack into distinct units sounds like a temporary measure, some of these large transitions simply take longer than you think. For example, SoundCloud’s journey to refactor its monolith into microservices was an eight-year process.

Solution: Realize dual-track IT might take longer than you think and plan accordingly.

4. Intermittent Version Upgrades

Third-party software is constantly upgrading to new versions. Especially if you’re integrating many open source dependencies into your stack, you’ll find the cadence of version upgrades to be quite fast. These upgrades are often imperative to plug new vulnerabilities as they emerge in the software supply chain.

But upgrading versions and base images and having a patch management scheme that works from environment to environment requires a lot of work, Davis cautioned. Kubernetes version upgrades, for example, can take a lot of time. Yet, if your upgrades are not matching the pace of change, added Andersen, they can quickly pile up and leave you left behind with a mountain of work. “If you’re not upgrading monthly, you’re in danger.”

Solution: Have a continual upgrade process in place.

Technical Debt: Keep Paying, Cleaning, and Pruning

Technical debt is something you need to grapple with before it gets out of control. But it’s not always apparent where it lives. Looking toward the future, automated tech debt scanning could help discover where technical debt is and provide remediation points as part of ongoing software stack auditing.

You don’t want to get to the point where you accumulate and accumulate technical debt, only to burn it all to the ground in one disastrous fell swoop, said Davis. Similar to the way you slowly pay off a mortgage each month, technical debt is something you must continually keep an eye on and make reducing part of your team’s continuous workflow.

Or, consider the old camping mantra of always leaving the campground cleaner than you found it. You pick up a little bit of extra trash and take it out with you, making the environment better for everyone. Or, consider how a gardener must prune the branches of a fruit tree to help it grow. Pick the analogy that works for you and see if you can apply it to continually reduce tech debt in your DevOps processes!

October 20, 2022
Continuous Delivery Advances Come at Cost of Testing

A survey of 820 application development professionals conducted by Applitools, a provider of an application testing platform, found well over a third of respondents (38%) working for organizations that now deploy daily changes to production environments.

The challenge is that application testing is not able to keep pace with that rate of continuous deployment (CD). The survey found organizations are seeing test suites covering less than 50% of applications.

Applitools COO Moshe Milman said a major reason test coverage is so low is that more than 40% of respondents said their continuous integration (CI) test suites can take more than an hour to run, which often resulted in tests either being postponed or possibly not run at all, he noted.

Most organizations are also still relying on manual testing processes that can’t keep pace with the accelerated rate of application deployment and subsequent updates, added Milman. More than half of organizations (55%) automated less than half of their testing, according to the survey. More than two-thirds of respondents (67%) are embracing test automation to improve their test coverage, the survey found.

In general, the percentage of organizations that are now employing CD processes to accelerate the pace at which applications are deployed is steadily increasing, said Milman. Manual testing processes, however, slow down the rate at which organizations deploy code. Many organizations routinely make trade-offs between application quality and a need to deploy code faster. Inevitably, there will be issues that can be traced back to decisions to not adequately test code prior to deployment, noted Milman.

Overall, the survey found 90% of respondents building and deploying web applications, followed by 52% that were building and deploying mobile applications. Only 30% are building desktop applications. Testing of the applications, however, tends to vary widely. Nearly two-thirds of respondents (65%) said they tested applications on an end-to-end basis, with 55% testing integration followed by unit testing (53%) and application programming interface (API) testing (51%).

However, only 26% tested application performance, followed by 13% that tested security and 12% that tested load, the survey found.

Part of the issue, of course, is that it’s not always clear who is responsible for application testing. In theory, developers are accepting more responsibility for application testing as they take more responsibility for managing applications on an end-to-end basis. In practice, developers are often rushing to make application development deadlines. It’s not uncommon for developers to skip or limit testing in the name of productivity. Many of them have also embraced agile development methodologies that, at least theoretically, allow them to address any issues that might arise via the next application update.

The issue that arises is that end users have become less tolerant of experimentation. If an application doesn’t initially work as expected, it’s exceedingly difficult to regain user confidence. Many will simply evaluate another similar application. The irony, of course, is that in the race to deliver code faster, application experience winds up being sacrificed because of the lack of time to properly test an application before deploying it.

July 29, 2022
VW CEO Fired for Dev Fails | Fiber Shortage Hits | Google Fires Blake Lemoine

Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

This week: Herbert Diess is out at VW because software is hard (yo), fiber optic cable is hard to find, and the guy who said LaMDA was sentient is dumped by Google. (more…)

July 26, 2022
Top 5 Must-Haves for IaC Automation Tools

These days, there are a lot of different DevOps tools to accomplish a lot of different jobs. Almost daily another startup comes out with a new and innovative product or a newer (maybe even better) version of existing tools. One of the biggest innovations has been infrastructure-as-code (IaC). Giving infrastructure admins and developers alike the ability to create, manipulate and destroy infrastructure using code files changes the way we administer our environments. Though these tools are revolutionary and can solve many problems, sometimes they come with a whole new set of issues—as can any type of tooling, really.

Generally, IaC tools are run locally at first. When an infrastructure admin or developer is just starting out with it, they store the files locally; maybe in a repository, then clone the files to execute locally. This is perfectly acceptable and a very common practice. The issues arise when it comes to scaling this practice to a team; for example, with regard to visibility and governance. If everyone is running their deployments locally, the team doesn’t have clear visibility into who is deploying what, when and with what variables. This is where IaC automation tools come into play.

There are many different types of tools to solve these problems. Some are better than others. CI/CD pipeline tools, custom-built automation platforms, purpose-built IaC automation platforms—each has their pros and cons. Getting into all of that can be overwhelming and we could go on for hours about each option. But, we’re here to talk a bit more generically today. So, we’ll save all of that for another time. For now, let’s take a step back and talk about the top five must-haves for any infrastructure-as-code automation platform you’re considering. Because each company and use case is different, we’ll talk about these considerations in no particular order.

Role-Based Access Control (RBAC)

This first factor is pretty important. I know I said I wasn’t discussing these in any real order, but this one should really be close to the top of the list for anyone looking into IaC automation. We talked about how automating IaC can cause visibility issues when you scale across a team. You may want to know who deployed what, when, where and why, for example. Usually, the reason you want that kind of visibility is that you would like to control all of these aspects. That way you can help limit waste, budget problems, security concerns and the like. You may want to provide self-service deployment access, but only within certain parameters; maybe only to certain people or teams. Having granular RBAC can help you design a security policy that makes the most sense for your organization and its needs. Some tools have this baked in, some take a bit of work to make it work. The point is that you should have some form of control over your deployment process.

Self-Hosted Agents/Runners

Following the security thread: A lot of these IaC automation tools are SaaS platforms. SaaS can be very secure. But sometimes, for compliance or regulatory reasons, you may want to retain a bit more control over your deployments. This is where self-hosted agents or runners come in. This type of design allows you to keep your “secrets”—such as your cloud credentials and other sensitive variable information—secure in your own way. You can use whatever secrets management solution you want, be it something from your cloud provider or another third-party solution. It also allows you to keep your code secure. All of these tools, when deploying your code, have to get a copy of that code to deploy. So, they’ll do a ‘Git clone’ or some kind of file copy process to get your files from where they are stored, to execute them. If this is a SaaS solution, then they will essentially have access to your code. Again, this may not be a huge issue for some, but it very much is for others. If you don’t want to give a third party this level of access to your code and secrets, a self-hosted agent or runner can help keep it all secure and under your control.

Plan on Pull Request

This feature is a big one when it comes to workflow. Some may not even know how important plan on pull request (PR) is until they realize how easy it can make their life. This feature may be called something different depending on the platform. Plan on PR, PR plan, speculative plan, etc. Essentially, this means that whenever you open a pull request against a branch, your automation platform will complete a deployment just up to the “plan” phase so that you can see exactly what this code change will do. For example, if you accidentally add an extra 0 to a variable or code file, you may end up with 100 instances instead of 10. Ideally, you’d like to know what was going to happen with a change before it is deployed, giving you the option to quickly fix a potential mistake before it actually goes through. In DevOps, you always want to ‘fail faster’ and close the loop so that big mistakes are caught before they happen. That is why this feature is so important. And if the automation platform you choose is doing this correctly, when it runs this plan phase it’ll automatically update the pull request directly with the possible changes as a comment. This way, your developers can see what is going to happen before they try to merge the deployment back to the main branch for deployment to production. And if they take it a step further by adding service checks to the pull request, you can even configure it so that if the deployment plan fails for any reason, developers are actually blocked entirely from merging the PR back until it is resolved. All of these features help to enable the ability to work in a GitOps methodology. Don’t get me started on the wildl misuse of the term GitOps—it is very specific. While one of the central pillars of GitOps is using your repository as your “single source of truth” for what should be deployed, it’s not just using Git to store your Terraform files; it is an infrastructure administration via pull request methodology. If you want more information about GitOps, you can read more here.

Continuous Deployment

This feature is essential for most folks. But, depending on the workflow, it can be less important. Even if it isn’t a must-have now, it may be down the road when you evolve and grow into new workflows. Continuous deployment is the ability for your automation platform to have a webhook or other connection to your source code repository so that it can automatically trigger a deployment process whenever you commit or push to the repository. This feature goes hand-in-hand with the plan pull request feature we just spoke about. To make that feature function properly, a CD feature would be needed, as well. Now, I want to make sure I clarify. Just because you have continuous deployment enabled doesn’t mean it is just going to push to production in the middle of the day every time a commit or push is done. You should still be given the ability to pause the deployment process after the plan phase. This way you can validate that the plan looks acceptable before you approve the deployment. For some use cases, like developer sandboxes, you may “auto-approve” the deployments because you don’t care about having to get validation first. It all depends on your workflow and processes. A good platform will give you the ability to control all of this.

Shift Left Extensibility

This one is a bit of a wild card. It can mean a lot of different things depending on the situation. Essentially, what I am talking about here is the ability to integrate—or at least be interoperable—with a multitude of other tools during the deployment process. There is a concept called continuous verification which means to shift some tools and processes left into the deployment process so that you can stay on top of problems before they start. In the PR plan feature section earlier, we talked about catching issues before deploying. This isn’t just for resource configuration issues. What about security issues? Compliance issues? Performance issues? Or even budget issues? If your IaC automation tool gives you the ability to integrate the processes and tools you are using to validate these things into the deployment process, they no longer have to be an afterthought. You never have to worry about going over budget if you are constantly validating the budget each time you deploy. You don’t have to worry about a security issue if you’re validating security during every deployment. This type of functionality can look and be named very differently between tools. But as long as you have the ability to get all of these other groups (security, finance, etc.) involved sooner, everyone will be better off. Like we talked about before, in DevOps, it’s always better to fail faster. And integrating these checks into your deployments can help you achieve this.

Conclusion

We’ve made it through all five! We’ve talked about some basic security features. We talked about some automation workflow must-haves to make your life easier. And we even scratched the surface on some next-step-type things to look into such as continuous verification. That was a lot—hopefully this has been helpful for anyone looking into IaC automation platforms and you now have an idea what to look for. Some features may be named differently, but the ideas behind them are the same. If your platform can’t do these, it may not be the right one for you. Or, maybe it is and that feature just isn’t that important to you. As I mentioned, your mileage may vary. Every organization is different.

To hear more about cloud-native topics, join the Cloud Native Computing Foundation and the cloud-native community at KubeCon+CloudNativeCon North America 2021 – October 11-15, 2021

October 14, 2021
Small Teams Focused on Delivering Outcomes Produce the Best Results

The days of big team consulting are over. Technology organizations are aligning their budgets to get projects done faster and more agilely. More applications are being built by small teams of cross-functional engineers moving fast, providing higher quality code, designing away waste and cutting out processes that don’t add value.

It’s becoming apparent that engagements incorporating smaller footprints and shorter iteration cycles are more common. The best outcomes are where an engineering team is small, lean and iterating many times faster than much larger teams. That means that enterprises get their applications to provide value earlier and at lower cost, and they cost less to maintain over time, starting right away. It means applying higher quality and better managed teams using core engineering principles. It means cross-functional technologists, not cross functional teams working in silos in their specialties

The catch is: How do you incorporate that at enterprise scale? It’s not easy to do because there is a lot of internal resistance and inertia. And many organizations will not achieve this.

Agility and priorities are key here. Future thinking organizations don’t waste this effort on trying to knock out backlog. They make sure they’re applying themselves in a targeted fashion and they drive velocity higher on things that matter. It’s not about providing spot fixes. Don’t start with building a CI/CD pipeline or an AWS migration. Work with your engineering team on driving down the overhead that’s slowing down the delivery of your platform.

It’s not about using this tool and that tool, and this programming language and that platform to host it, and that it will integrate a certain way, and you are now able to say that you are delivering in a continuous integrated fashion. You should get continuous delivery set up. However, continuous deployment is one step. What you really want to do is reduce costs by 20% and reapply that to a feature that you’re looking for next quarter, instead of waiting till next year. You can restructure the conversation because from the get-go you front load the delivery strategy with those kind of performance terms. And this is real money. We are applying these processes where customers are paying for software to be delivered.

What doesn’t work is “Let’s do some DevOps over here, and then maybe a little over there and make incremental progress.” Spend resources doing principled engineering all in the same application—not bits and pieces in various places and hope to bring it together at some point

One way to know if you are doing DevOps—or any cloud related engineering—better is that you have a better grasp of Amazon’s APIs than Amazon does. Literally, the best companies release updated APIs for Amazon before Amazon releases their new APIs. That’s how good they are.

Different organizations move at different rates with an understanding of DevOps. Some people still say “Get me a DevOps engineer.” Others say “Get me a DevOps’ culture change,” which is OK. But, really smart organizations are asking for small engineering teams to build high performance, high priority platforms quickly, at the lowest possible cost, using the best engineering principles available.

One other way to know that you are doing good engineering and DevOps is when you need DevOps engineers and you talk to a big consulting outfit and they say, for example, that you need 65 SREs with different skill expertise, four project managers, a bunch of data scientists and product people and client partners. They say it’s going to be really expensive and that they’re going to be here forever, and that this is an engineering challenge. Let’s address it from that POV. The engineers are running it, the engineers are closest to the work, they understand the issues and the opportunities, and they just need 10 really good full stack engineers.

The name of the game now is applying engineers that clearly have a specialty, but are more importantly good engineers with software and know how to write code well. You might have a platform engineer, which is going to be like your infrastructure SRE but he’s going to be knowledgeable and know how to write code at a basic level. That will work. You might have an engineer that isn’t the best software developer, but knows platforms, what clouds are available, how to offer it across those programmatically, etc. That will work, too.

The bottom line is moving initiatives that are focused on building revenue or saving a lot of money and driving them forward at a higher velocity, with higher quality delivery at their product launch through foundational concepts, such as continuous integration, continuous delivery and continuous deployment. Leading organizations are focused on doing that in a way that is future-forward, focused on stateless and serverless infrastructures.

This article was co-authored by Jack Keck. Jack is a seasoned technologist leading a cross-functional engineering organization at a global investment bank. He spearheads transformative initiatives with the goal of raising revenues through the use of future forward technologies.

— Larry Gordon

March 25, 2020
The Definition of Faster in the Age of App Capital

Businesses want speed. One of the impacts of digital transformation—and the associated pressure to succeed in an app economy—is the desire to move at speed. Nearly half (48%) of organizations are driven by digital transformation to deploy faster according to our State of Application Services 2019.

But it’s not all about deployment. It’s also about development and responsiveness to threats and changes in demand.

Organizations want to develop and deliver apps faster. They want to adapt to unpredictable changes in business conditions faster. They want to respond to attacks faster.

So, how are they going to get it?

Faster Development

Deploying more frequently implies you’re first developing faster. That often means adoption of speed-oriented Agile methodologies. GitLab’s 2019 Global Developer Report: DevSecOps found that the majority (54%) are choosing Scrum, while 37% go for Kanban.

Componentization

Methodologies aren’t enough if application architectures aren’t well-suited to development styles. Smaller, focused teams sprinting toward frequent releases isn’t really a good fit for developing new features or fixes for traditional and monolithic applications. Componentization through the adoption of microservices and more distributed forms of architecture fit more naturally with modern architectural and operational principles. It’s no surprise that, on average, more than 80% of a modern application is made up of third-party components, heavily weighted toward open source.

APIs

Adoption of APIs, as well, is no surprise amongst those seeking faster development through componentized application architectures. APIs decouple implementation from interface and allow teams to make modifications to processing without impacting the use of APIs from other components or applications. This is a common pattern, with an impressive 64% of organizations creating APIs today for use in either internal or external use cases. Nearly 50% of those organizations rely on APIs to get ideas to market faster according to Jitterbit’s State of API Integration 2018.

Automation (CI/CD)

The build pipeline, too, has to keep up if frequently deployment goals are to be met. That means CI/CD tools that seamlessly move code from commit to test to release. The most popular CI and build tools in GitLab’s survey were, unsurprisingly, GitLab (61%) followed by Jenkins (36%) and Travis CI (12%). Notably, our own research finds Jenkins in use by 16% for network automation as well—a promising result if organizations are seeking to extend DevOps beyond delivery into deployment.

Faster Deployment

Developing faster doesn’t get ideas to market faster. That requires deployment. While born-in-the-cloud technology companies have mastered the delivery-deployment divide with ease, many established enterprises find this transition challenging. Existing organization structures as well as the continued need to support traditional and monolithic applications introduces conflicts that can be challenging to integrate with modern operational demands. But there is no doubt that organizations must overcome these challenges for those applications requiring faster and more frequent deployments.

Continuous Deployment

Don’t be fooled; established organizations are embracing automation and applying it to the deployment pipeline. The issue is often that traditional IT structures introduce inconsistent automation and self-service efforts. We see this play out in our own research, where strongly siloed team structures continue to impact automation efforts across IT.

Team structures matter and organizations whose compass is pointed at automated pipelines will need to address the cultural aspects of continuous deployment if they want to consistently deploy faster.

Public Cloud

Traditionally, the inability to achieve continuous deployment has pushed developers and app owners to turn to the public cloud, because it eliminates the obstacles in the slower, enterprise deployment path. The disparity in opinion on frequency of deployments is, in part, a contributor to the issue. Our 2018 NetOps/DevOps survey found that while 55% of DevOps and 52% of cloud architects felt their organization did not deploy frequently enough, only 30% of NetOps and the same percentage of operations agreed.

But it’s not the only driver. Digital transformation is certainly a factor; 33% of respondents in our State of Application Services indicated they are intentionally moving to deliver apps from the public cloud as a result of embracing digital transformation initiatives. The ability to easily integrate application services and subsequently automate operation is a huge boon to organizations seeking a faster route to frequent deployment.

Containers

In the cloud—but also on-premises—continuous deployment often requires the ability to adopt per-app pipelines and application services that support the model. Increasingly, organizations are turning to containers because of their ability to support rapid updates and operate seamlessly in highly volatile environments. We see containers in demand not only to support modern application architectures such as cloud native—the top container use-case at 33% of respondents to Diamanti’s 2019 Container Adoption Benchmark—but also for infrastructure. The demand for on-premises application services in containers has consistently grown year over year in our own research, rising from just 4% in 2017 to 15% in 2019.

Faster Responses

Not everything is about delivery and deployment. Organizations also need speed when it comes to security and operations. In today’s world, where more than half of all app interactions are conducted by bots, it’s important for organizations to respond quickly with a reject message to avoid falling prey to exploits or infection.

Real-Time Threat Analytics

One of the ways organizations are seeking to speed up their ability to respond to attacks is through the adoption of real-time threat analytics. With security continuing to be a top priority and consistent challenge, it was no surprise to see this category ascend into the top five strategy trends and technology in 2019 with 41% of respondents tagging it.

The availability of intelligent security application services will accelerate as machine learning and automation continue to apply their considerable capabilities to the problem of identifying troublesome traffic faster.

Orchestration Included

Machine learning isn’t the only way in which infrastructure and application services are getting smarter. As the need for faster responses to capacity and processing grow along with users, infrastructure and application services are evolving to include orchestration as a core capability rather than an add-on. Application service platforms with integrated orchestration layers will mean services that scale along with demand—on-demand. While such capabilities exist today—indeed, they are part and parcel of cloud and containers—the capabilities to automatically scale up and down based on application and user defined requirements is not native to most systems today. But they will be.

Faster Identification of Bad Actors

Finally, organizations need to combat bots for both business and security purposes. Whether organizations need to stop scraping—a real threat to business—or prevent bots from probing for vulnerabilities, identifying bad bot actors faster is critical today. Checking a box (I’m not a robot) isn’t enough today as bots are growing more intelligent and are able to foil such primitive techniques.

Organizations are turning to Bot Defense services capable of applying more modern, successful techniques to quickly identify and block bad actor bots. We’ve seen the use of bot defense services grow quarter over quarter in response to this need, and expect this trend to continue.

Faster Development, Faster Deployment, Faster Responses

When it comes to business’ need for speed today, we wind up looking at processes. Development processes are automated to gain speed with CI/CD. Deployment processes are automated to gain speed with continuous deployment and cloud. Scaling and security processes are being automated by systems and the introduction of orchestration as native to application service platforms.

Faster, it turns out, relies on an organization’s ability to automate processes across development, deployment and security.

— Lori MacVittie

February 14, 2020