Category: IT as Code

Infrastructure-as-code (IaC), often embodied by open source Terraform, is an essential ingredient to cloud and cloud-native strategies. But without the ability to scale, secure and manage IaC, you very quickly experience drift. Tim Davis, DevOps advocate with env0, pronounced “N zero”, discusses establishing a single source of truth and reigning in drift by updating and managing infrastructure as code. The video is below, followed by a transcript of the conversation.

Announcer: This is Digital Anarchist.

Mitch Ashley: I have the pleasure of being joined by Tim Davis. Tim is DevOps Advocate with env0, and so, we’re going to be getting into some infrastructure as code type discussions, so, looking forward to that. Welcome, Tim—good to be talking with you.

Tim Davis: Same for you, Mitch. Great to meet you. Thanks for having me.

Mitch Ashley: You, too. I love your cloud off to your right shoulder, by the way. [Laughter] Anyways, so—introduce yourself, tell us a little bit about env0.

Tim Davis: Absolutely. So, I am the DevOps Advocate with env0. dnv0 is a TACoS platform, if you will, the Terraform Automation and Collaboration Software space. We really focus on infrastructure as code, automation, teams and governance, and other tools to help you with the life cycle of your infrastructure as code environments.

Mitch Ashley: Cool. Good. Well, you know, it’s a great conversation to have, because something that I observed always happens in our industry is, you create a new technology, it’s kinda easy to get started and starting to use, sometimes it isn’t always easy, but the complexity comes—alright, how do I scale this? How do I do this across a larger development team or multiple development teams? How do I operate it? How do I secure it through the day two kinda stuff?

Tim Davis: Right.

Mitch Ashley: And I wouldn’t be surprised if you told me the same is true for infrastructure as code, and in fact, I know that to be true.

Tim Davis: [Laughter] Exactly right.

Mitch Ashley: Let’s talk about that. What are some of the challenges once people get into doing infrastructure as code, particularly with Terraform? What are some of the challenges that you run into?

Tim Davis: Yeah, and most of those kind of lie around the visibility and kind of the control space of that. Infrastructure as code is great. It is a fantastic way of speeding up, kind of pushing yourself into the future for DevOps, getting into that real GitOps life cycle. But with that, you know, if it usually just starts out with either one developer or one infrastructure person that starts running it locally on their laptop and figuring it out, it’s great, it does what they want.

As soon as you start to scale that methodology and doing it locally to multiple people, that’s when you start losing that visibility of, “Where are my state files stored? Who just deployed something into the cloud? What variables might they have used for that?”

So, it really becomes kind of an, “I don’t know who’s doing what in my cloud anymore.”

Mitch Ashley: Mm-hmm.

Tim Davis: So, really making sure that you’re staying on top of that, kinda centralizing everything to maintain that visibility is very important when you’re scaling it.

Mitch Ashley: Let me ask you, then—is it fairly typical that within one software Dev team you have a person or sort of a close enough coordinated group of people that are doing the software as code, infrastructure as code. And then the confusion may exist as you start to go across multiple teams, or does it also occur within one team, just sometimes, it’s every developer for themselves and so—

Tim Davis: Yep.

Mitch Ashley: – you could have a wild, wild West kinda situation before you know it.

Tim Davis: [Laughter] And the answer of that, of course, is yes, I mean, it’s all of the above. It could be even just two people on a small team that start to use these tools and they start to step on each other’s toes by having deployments that kind of counteract each other. Or you have it where somebody is deploying out into the cloud using your infrastructure as code and somebody else is just going into the cloud and clicking around and deploying stuff manually where you get something called drift, where what the infrastructure as code says is out there isn’t necessarily matching with what’s actually there.

Mitch Ashley: Mm-hmm.

Tim Davis: So, it can be across a single team, multiple teams—any time you really scale it past a single person on a laptop is really where you start to see those issues.

Mitch Ashley: Yeah, okay. It’s a familiar problem, right?

Tim Davis: [Laughter] That’s exactly right.

Mitch Ashley: We’ve had this in other things, too. Well, what are some of the best practices or kind of good infrastructure as code hygiene to keep your worlds at least sane to start with so you—the first question is always, “What’s there?” If you don’t know what’s there, then—

Tim Davis: Right.

Mitch Ashley: – everything is up for grabs, or you question everything because you just don’t feel like you have a good grounding of what you’re looking at and what you’re dealing with. What are some of those best practices?

Tim Davis: And I really—I try to stay away from the best practices thing, because sometimes what’s good for somebody isn’t great for somebody else, but that—

Mitch Ashley: Well, and I say that as best practice is not universal, right?

Tim Davis: Absolutely. And there are some things that you can do to help yourself out, for sure. If you’re using infrastructure as code, having the single source of truth can be very helpful for everybody, essentially stopping that drift that I just mentioned. So, if you are using infrastructure as code, save them in a repository somewhere. Make that the single source of truth where whatever’s in there, whatever’s in those infrastructure as code files, that’s what’s there.

So, whenever you decide to go for infrastructure as code, stop using the cloud UI, stop going around and click, click, clicking and adding resources and things. Do it by updating the infrastructure as code file and then re-deploying. That can really help you make sure that you’re not causing any problems, nobody’s stepping on anybody’s toes, and you know if I see it in the files here in this one spot, that’s what’s out there.

Mitch Ashley: Mm-hmm. Yeah, I totally agree with you. That makes a lot of sense to me. What are some of the complexities, then, you have to deal with? Because now we’re working an environment where infrastructure, the stack, the distribution of that, the locations, whether geographic or service provider really can change instantly without necessarily the full development team, which is an advantage, right? You want that sort of abstraction.

Tim Davis: Exactly.

Mitch Ashley: But it also can be—introduce new variables and maybe even new problems.

Tim Davis: Yeah, and it could be new problems, it could be the same old problems, or it could be new problems that you kinda deal with the same, old way. And, you know, adding controls of making sure you’re adhering to policy, like, you’re only allowed to deploy to certain regions or you’re only able to deploy a certain number or size of instance, making sure that you have some form of role based access control there to know, you know, this person is allowed to deploy straight out to development, no problem, but if it’s going to production, it needs to wait for approval and somebody from either the DevOps team or the SRE team needs to validate everything first before we actually make that push.

Mitch Ashley: Mm-hmm. I think something else that always happens, too, is in—you know, I’m a security guy as well as a software guy, so, I always worry about the security side of it. Now, automation can make that, you know, much better.

Tim Davis: Exactly right.

Mitch Ashley: Much more reliable, less human error. There’s a lot of good things that come with infrastructure as code from a security standpoint, but it also can be, it’s sort of security is in the eye of the scripter and the person who’s—

Tim Davis: Right. [Laughter]

Mitch Ashley: – doing, you know, configurations or doing the scripting, too, which can be good or bad. How do you address security in this kind of a world?

Tim Davis: Yeah, and this is just one of those things where we’ve all heard the term shift left, and shifting all of these different things—security, performance, you know, billing—everything, if you shift it left into the deployment process, it really helps if you foster that communication. I mean, we know that old school term of silos where one hasn’t isn’t talking to the other hand. If everybody gets involved, if the security team is involved with writing the policy, they’re involved with double checking that it is implemented into the deployment process so that you don’t have to go and fix a problematic deployment later, it’s just stopped before it starts. That can really make sure that everybody kind of gets what they want, but security folks still have to be involved. They are still in charge of the overall process and procedure and policy, it’s just, it’s kind of a new way of getting that implemented and at what stage of the life cycle you’re implementing it.

Mitch Ashley: Do you think it shifts—and this is actually, I just did a talk on this about shifting left of, we wanna shift everything left, which is generally a good idea, right?

Tim Davis: Right.

Mitch Ashley: To be able to address some of those things early and design it in, design security in, et cetera. Oftentimes, it’s interpreted as, “Let’s put that on the developers, too. Let’s have them—the developers will worry about that. The developers will worry about all those.” Now, it’s pretty much, it could be everything, sort of a ridiculous request, right?

Tim Davis: Right.

Mitch Ashley: Because that’s not a developer’s expertise to necessarily know all those things. What is your—when you say shift left or infrastructure as code—

Tim Davis: Right.

Mitch Ashley: – in practical terms on a software team, what does that look like?

Tim Davis: Yeah, and a lot of folks think that that just means, “Hey, I’m taking away your job as the security guy and I’m gonna give it to the developer” and that’s not the case. The developer, they may be a little security conscious, they may understand the infrastructure or the networking piece, but nowhere near as much as the career security guy or the career networking and infrastructure guys. It just kind of brings them in and fosters that conversation.

I definitely think the developers are going to be part of the conversation, because it is bringing it into their tooling, into their languages, into their life cycle, but it still requires that expertise to be able to make sure that it is done correctly, it’s implemented correctly, and that it’s being checked the way it needs to be done.

Mitch Ashley: Mm-hmm. So, does it look like security engineers, to use that as a term, they’re sitting down with the developers, saying, “Okay, how are we configuring the environment?” and, “Here’s how you’re building your Terraform configurations”—going through that with them?

Tim Davis: I think so.

Mitch Ashley: Is it saying, “Here’s the principles we want you to follow”—

Tim Davis: Yeah.

Mitch Ashley: – “and as you’re creating this, please do these things, and then we don’t have to sit on your shoulders”?

Tim Davis: I definitely think there’s lots of different ways you can do that. I mean, obviously, you know, having your standup meetings, making sure that they are kinda telling them, “This is what the policy should look like, however you want to implement it is fine, but this is the kind of processes and procedures, these are the rules that we need to check.” So, it can definitely be done however works best for everybody, but as long as you’re having that communication and you’re making sure that everything’s going through—yeah, you know, you can always do it different ways and be successful, still.

Mitch Ashley: Mm-hmm. Interesting. What are some of the things that you’ve seen change? You know, there’s been a lot of acceleration over the last 12 to 18 months.

Tim Davis: Right.

Mitch Ashley: And, you know, I’ve heard people talk—of course, you always hear about the, as much digital transformation happened in the last year that was planned for five years.

Tim Davis: Yeah.

Mitch Ashley: And there is some truth to that, and I’m sure some exaggeration, too. But there’s a strong belief that, you know, people are now thinking they can deploy applications much, much faster and they’ve proved it in this last window. Have you seen that happening?

Tim Davis: Yeah, absolutely, and you know, technology changes. They used to say it doubles every six months or what have you, and really, we are iterating and changing and kind of moving forward very, very quickly. A lot of times, we get a new tool and it’s great and then people are like, “Alright, I’m gonna pull this tool in and automate it” and then they figure out, “Oh, there’s some extra complexity here and we’re having to go back through and figure it out.”

I definitely think there’s a lot changing, you know, even just the shift from on prem legacy, I guess, enterprise architecture through to new, like, cloud native architectures and things like that. We’re moving fast, we’re able to change and pivot and do what we need to do, but as long as we make sure that we’re kind of keeping grasp of what we’re supposed to be doing from either a security or an infrastructure construct, we can make sure that we can iterate and change and adapt as fast as possible without causing any major issues down the line.

Mitch Ashley: Mm-hmm. Cool. What do you think the next 6 to 12 months look like? If you had to put on your crystal ball, your Magic 8 Ball, that says, “Ask Again”?

Tim Davis: [Laughter] Yeah, I love to look into that. Obviously, it’s good to see, you know, what am I gonna have for lunch next week, but also, you know, looking into the industry of what’s gonna happen. We’re seeing more and more X as code. We’ve got infrastructure as code, you’ve got security as code, you’ve got policy as code tools. I think we’re gonna see more and more and more, you know, performance as code and things like this that just are able to bring those day two plus operational tools closer left into the deployment cycle and have all of these single source of truth in the repositories of, we know exactly what our infrastructure is supposed to look like, we know how it’s supposed to be secured, we know how it’s supposed to perform. All of this is just declaratively set up there. I think we’re gonna see a lot more of that going forward.

Mitch Ashley: Mm-hmm. Well, certainly, as you said, it follows the “everything as code,” right?

Tim Davis: Exactly right.

Mitch Ashley: And if we’ve learned nothing else, I think that’s been a lesson of the past 12 to 18 months, right? It is software, it’s all about software.

Tim Davis: It absolutely is.

Mitch Ashley: What do you think some of the skill challenges are for people as we continue to evolve this and move into this rapid world, shifting things left, doing more infrastructure as code, and maybe asterisk as code—everything? You know, it’s always the self-learners can be at the edge of the curve, other folks have other ways of learning. How do we bring everyone along in terms of their skills and their learning, their development?

Tim Davis: Yeah, and that’s definitely something. I mean, I ran into myself. I’m an infrastructure guy at heart, that’s where I’ve been, you know, my whole career, and eventually, someone was telling me, “Hey, this Python thing might take off” or, “I’ve heard of this thing called Terraform and it’s basically like code, but it’s our job.” And, you know, just realizing that nobody’s trying to take away your job as a security guy or an infrastructure guy or girl or what have you. They’re not trying to take it away from you and move it over to a developer, it’s just one of those things where you need to adapt to move forward faster.

Now, it’s not just infrastructure folks that are learning to code, it’s also developers that are kind of learning infrastructure and figuring out how can I optimize my application against this infrastructure a little better? How can we work together to move forward faster? So, it’s kind of a melding of skills from different, you know, old school silos now to work together to solve the problem.

Mitch Ashley: I made that recommendation to a network and security engineer about 10 years ago. It’s like—learn Python, and he looked at me like I had six eyes instead of two.

Tim Davis: [Laughter]

Mitch Ashley: And he came back from the Cisco conference, he goes, “Now I know why you said that.” [Laughter]

Tim Davis: Exactly right. And that’s a great piece of advice even today in 2021. I mean, if you’re in infrastructure or you’re a network tech or you’re a security person, look into Python. It’s not as hard as you think and you can do so much with it. It’s a fantastic way to kind of break into that DevOps automation type space.

Mitch Ashley: I like that, too, and not that we’re trying to make everybody a developer, but if it’s not a skill you have, it sort of leaves out this whole domain of things that you can either do to understand what’s happening—

Tim Davis: Right.

Mitch Ashley: – or do to automate or et cetera. It’s sort of like you’ve got one arm tied behind your back, why don’t you have two out in front where you can fully work?

Tim Davis: Yeah. And it also helps from a cohesive understanding between groups. Back when I was a dedicated infrastructure person, myself and the development team didn’t care for each other, because we all were trying to get in each other’s way, it seemed. We were trying to solve the same problems, just going about it in different ways, because we didn’t necessarily understand the methodologies or how those people were trying to get their work done.

If you as an infrastructure, security, or other operations person start to kind of learn the development methodologies and figure out how things work, it’ll help you to communicate and solve problems across teams better, simply because you kinda see where they’re coming from with their tool sets and the way that they do business.

Mitch Ashley: Great. Well, it’s been a great conversation. I’m glad we got to explore this path together.

Tim Davis: Absolutely.

Mitch Ashley: Where can folks find out more about env0? I know you have a kind of funky spelling of env0.

Tim Davis: [Laughter] Yes. We are at env0, E-N-V and the number 0.com. You can find us on Twitter @E-N-V-Z-E-R-O and on YouTube at E-N-V-Z-E-R-O.

Mitch Ashley: Great. So, environment zero, right?

Tim Davis: Exactly right.

Mitch Ashley: Perfect. Well, hey, it’s been fun talking with you and hope you’ll come back again.

Tim Davis: That sounds great. I look forward to it. Great to meet you, Mitch. Thanks for the time.

Mitch Ashley: Alright, take care.

[End of Audio]