devopsy.kinsta.cloud

Tag: InSpec

Using Inspec to Validate Infrastructure

So, here we were, at one of the bigger financial corporations in New Zealand, working on an elaborate infrastructure built on Docker containers. It had configurations, SSL certificates, DNS entries and all the other condiments required to spice up a tasty infrastructure curry. Though we were able to stand up our complete infrastructure environments from scratch using automated configuration management tools, we weren’t really sure if it was indeed working correctly, or rather the way it was supposed to. The question staring at us was, Is there a way we can declare the desired state of our infrastructure and then validate our infrastructure against it?

Inspec to the Rescue

The open source tool Inspec has been used with considerable success over the last few years to achieve security compliance for infrastructure. The syntax/constructs happens to be unbelievably simple. For instance, let’s consider the code snippet below. It doesn’t really require a lot of programming knowledge to make sense of it. Isn’t it?

This made us wonder, what if we use Inspec to validate our infrastructure setup? What if we map our infrastructure configurations in an easy-to-read declarative syntax? What if this actually solves our problem?

In line with the best practices of DevOps, we started with a simple experiment; an experiment that could solve a tangible problem. We had a Docker host which was supposed to have a particular docker container running inside it, but it used to fail quite often, for some reason. We decided to write a test in Inspec to check whether this container was running on this host:

… and tried executing the test.

This was just what we wanted!

Excited with this success, we started writing a battery of tests that validated each and every component of our infrastructure. Within a single day, we churned out 192 tests. The number of tests wasn’t as important as the scope and breadth of the test coverage. Once the tests were there, we plugged them on to our infrastructure pipeline (which was built using Jenkins) to ensure the components align with the desired state. We called them ‘component-readiness’ tests.

So far so good.

Now that our components match the desired state, are we still confident that our platform would work the way we intended? Perhaps we need some “platform readiness tests” as well. Can we use Inspec for that, too? The answer: Yes!

Now that the platform readiness tests were ready, we extended our infrastructure pipeline to incorporate them as well.

Setting our component and platform readiness tests using Inspec had one more fantastic offering: live documentation for infrastructure. Inspec allows the audit results to be displayed in a variety of report formats, including JUnit, JSON, HTML and others. We used the JUnit report format and displayed them using the Jenkins JUnit Plugin. With this, we were able to have human-readable JUnit reports that doubled as live documentation for our infrastructure.

Needless to say, this infrastructure testing with Inspec provided that (much needed) confidence on our setup and readiness. Our pipeline is designed in such a way that whenever any infrastructure component changes in our ecosystem, the pipeline is triggered which reconfigures and (if required) rebuilds the ecosystem. These Inspec tests are triggered with this infrastructure pipeline and validates the newly configured infrastructure setup. Thanks to this, we are never in a situation in which the client application running on this infrastructure complains or aborts because of incorrect configurations. Moreover, as described above, the living documentation helps document the expected configuration of the environment.

So far, the Inspec implementation I have referred to is using the freely available Inspec tool and the Jenkins platform. Inspec can also be combined with Chef Compliance to produce more descriptive reports and charts. Chef Compliance also comes with out-of-the-box CIS profiles that can be used to jump start an Inspec implementation for security compliance and audits.

Inspec is an easy tool to master with simple and intuitive constructs and one could get up and working quite easily. However, this simplicity has a downside. It’s very easy to speed ahead and gloss over the documentation while missing out on the best practices and tips/techniques to implement the solution in a better way. One way to master this is to have a crack at the newly released Auditing with Inspec certification, which provides the impetus to go through the documentation in detail while preparing for the certification. And, you also get to show off a shiny badge for your efforts!

In Summary

Infrastructure validation is quite an important step in any DevOps implementation. Apart from being very useful for security compliance testing, Inspec can be used to validate infrastructure using a simple and easy-to-understand declarative syntax. Moreover, it has a small footprint, a vibrant community looking after it and up-to-date and clear documentation. It definitely is a tool with the potential to provide quick and effective results.

— Mrinal Mukherjee

April 3, 2018
DevOps Chat: Chef Update with Ken Cheney, Chef CMO

There are always new developments going on at Chef. We caught up with Chef CMO Ken Cheney to give us an update. Last time we spoke with Ken was right after ChefConf. Since then, Chef continues to move forward on several fronts including Habitat, Automate and INSPEC. We speak to Ken every other month or so, as it is always good to know “what’s cookin’” at Chef.

As usual, the streaming audio of our conversation is immediately below, followed by the transcript of our chat. Enjoy!

Audio

Transcript

Alan Shimel: Hey everyone, Alan Shimel, editor in chief of devopsy.kinsta.cloud here, for another DevOps Chat. Today’s guest on DevOps Chat is our friend Ken Cheney, CMO of Chef. Ken, welcome back to DevOps Chat.

Ken Cheney: Alan, it’s great to be here. Thanks for having me.

Shimel: Thank you. Ken, I apologize, between your schedule and my schedule, it’s been a while since we’ve had a chance to catch up. I think the last time we left off it was – was it right after Chef Conference, ChefConf?

Cheney: It was actually right before ChefConf. So it’s been a couple months, right? Given that was at the end of May.

Shimel: Ken, I don’t think I’ve been home for more than two or three days at a time since then. So yeah, it has been. But for our listeners who maybe aren’t lucky enough to be down in Austin and ChefConf this year, can you fill us in a little bit about maybe what we missed?

Cheney: Yeah, so ChefConf this year, as you mentioned, was in Austin, Texas. And you know, I think the highlights of ChefConf, on the product side, were some announcements we made around Chef Automate and bringing in our open-source tooling fully under that umbrella of Chef Automate. So Chef Automate is our commercial offering. And first and foremost, we brought in InSpec fully, so now, from one spot within Chef Automate you can see both how you’re doing in terms of your infrastructure and how your infrastructure automation is going with Chef, but also from a compliance perspective you can see what’s happening in terms of policy through the Chef Automate UI. So it’s really – the product has evolved from how it was working before, where we essentially had two servers for and now you have a single, common UI for looking across all the various data points

In addition we brought in the Habitat data so now you can see the state of your Habitat instances that are running in the wild. And you know, what I find really interesting about what’s happening with the compliance front is that, you know, we’re really evolving quickly from just being focused on sort of core compute elements to now expanding InSpec out to cover up on cloud-based APIs. So we can now go at AWS and Azure and be able to scan those, as well as VMware environments. So you know, we’re expanding InSpec to be a language that’s not just appropriate for compute, but also for broadly managing all of your infrastructure as code. And so that was announced as well at Chef Conf. And I think that’s a bi move forward. You’re going to see that continued expansion of that InSpec language and capabilities to go after a broad set of other types of infrastructure in the near future.

And it’s an area where we see just a tremendous interest and uptake from our customers and users. So anyone who hasn’t tried out InSpec, I really encourage them to try it as part of Chef Automate if you want to actually see that InSpec data roll up into common policy views and really understand how you’re doing at an infrastructure level, both in terms of the state of your infrastructure but also how your infrastructure is conforming to policy around compliance and security. You get that from one place. And in this world of WannaCry and other exploits that are happening, it’s amazed me how many organizations, when tools like InSpec exist, still have giant vulnerabilities, right? And here we have this open-source tool that can solve this problem of really understanding what is your exposure. And then you can use a tool like Chef to actually remediate and fix it. It just blows my mind that it’s still a common occurrence.

Shimel: It blows your mind, Ken? For 20 years it’s blown my mind. For those of us in the security space, it’s no secret that well over three-quarters—actually, probably over 80 percent or 85 percent of attacks and successful attacks actually take place not because of what we used to call Zero Day or unknown vulnerabilities, but because known vulnerabilities were not patched.

Cheney: That’s exactly right. That’s exactly right, yes.

Shimel: And that’s the key. I mean even today –

Cheney: Yeah, and –

Shimel: It’s crazy. I’m sorry, go ahead, Ken.

Cheney: Oh I was just going to say, one of the common things, you know, as many organizations haven’t really fully automated – and so they reintroduce the vulnerabilities into the wild all of the time. And what I love about what you can do with InSpec is you can actually – because you’re managing as code, every time you test your software, every time you push a configuration with Chef, you can verify that you’re not reintroducing to file that or something on a keyboard the reintroduces that potential hole in your organization.

Shimel: Yep, you know – I’m sorry, can you hear?

Cheney: Yeah, yeah, go ahead.

Shimel: I’m getting a little audio back. I hope this is recording well. Ken, we did a survey a while back where only 75 percent of organizations are actually scanning their code for vulnerabilities prior to deployment. And you know, what are the other 25 percent thinking?, is kind of what I think. But anyway, on the topic of security, Ken, I wanted to quick mention that Matt Ray of Chef is actually speaking at a conference I’m doing as a part of RSAAPG on DevSecOps next week. And Matt, I think, is based in Australia, correct?

Cheney: Yes, he is. Yeah, Matt is based in Australia and he will be talking about InSpec, and folks who are there can attend his session and learn more, for sure.

Shimel: So maybe it’s because I’m traveling to Singapore and so it’s kind of right in my face, but to me, it’s this global – I mean, DevOps has really become a global – I don’t want to use the word “phenomenon” – but a global force. And we’re seeing it

Cheney: Yes, it is.

Shimel: – you know, certainly in EMEA, but we’re seeing it in AP. And then just this week, Ken, I think Chef just announced their first Latin American partner, right?

Cheney: Correct, correct. Yeah, we are definitely seeing DevOps at a global level and seeing broad interest. And what’s interesting is that – Alan, I’ve been in the industry a while, and it used to be a little bit of a lag – you know, a several-year lag before you’d see markets such as Latin America or parts of Asia pick up technology that would originate in the United States. Or process ideas. But things move much more quickly now and you’re seeing global companies with operations in India and operations in China. And so we as a company do a lot of activity at a global level. You know, if you look at how we’re doing meet-ups and where we’re doing meet-ups, they’re happening all over the place. And our partner ecosystem, which is well over 300 partners, covers the globe, because we see that demand.

Now how we enter any given market varies depending on what’s going to work best in that market. So some markets we address through partners and some markets we actually put our own physical presence in that market.

Shimel: Yeah, we see the same thing at devopsy.kinsta.cloud, Ken. You know, our readership when we first started was almost three-quarters U.S.-based, and now it’s only about 37 percent, with the rest being global. But it’s more than just spreading through the world. It’s what you said – there’s not as great a lag time. At one point, Latin America was three – we used to tell people, “Oh, it’s three years behind what we’re doing here.” And that’s not the case anymore, certainly.

Cheney: Yeah, I’ll tell you, the highest volume of leads that we get come from the Asia Pacific region.

Shimel: Really.

Cheney: Yeah, so there’s just a tremendous amount of demand coming out of there. I’ll tell you, if we do a meet-up in Bangalore or Delhi, we’ll have 300 people show up to that meet-up.

Shimel: Wow, that’s fantastic. You know, we’ll talk more about that off-mic. But Ken, a lot of what’s driving strong numbers like 300 people at a meet-up and stuff like that is training and availability of training, right?

Cheney: That’s right.

Shimel: And that’s something Chef has always done really well. But you guys recently rolled out – not Pilot, excuse me – what is the name of the –

Cheney: Chef Automate. The Chef Automate Pilot, yeah. We actually rolled out a whole new online learning site called Learn Chef Rally and it’s gotten very fast adoption and has a lot of users hitting it every day. And as part of that we have what’s called the Chef Automate Pilot, which is really targeted at an individual who wants to really understand Chef over the course of maybe a lunch, right? So that kind of lunchtime evaluation where they want to get a feel for how Chef works and what it does for them. But as a whole, Learn Chef Rally constitutes a whole set of modules that will walk you through the various parts of our technology. So if you want to learn about compliance, there’s an InSpec module and a Chef Compliance module. If you want to learn even about DevOps concepts, you can go and learn about them there on the Learn Chef Rally site.

And we’ve had such strong uptake. You know, we’ve seen a lot of customers who would like to bring that learning site in-house and customize that learning environment for their own company. There’s been such a strong interest. So it’s a great resource and I encourage people to go look at it.

Shimel: Sure. Ken, well, the logical question then is, how do they get there?

Cheney: Yeah, so it’s very easy to get there. You can just go to learn.chef.io and you can get started.

Shimel: Excellent. That’s a good one. So that’s Chef Rally and it’s highly recommended. It sounds like there’s a lot and I’m sure you guys are planning to add more into that. So Ken, we are kind of in the dog days of summer here but it’s certainly not slow, it doesn’t seem, anywhere. Any other news, events?

Cheney: Yeah, well we just passed the one-year anniversary of Habitat and I’ll note that, you know, we at ChefConf made some Habitat news. So we announced a Habitat build service and we made improvements to the Habitat scaffolding, which are essentially build packs to make it easier for people to package up their applications, as well as we’re writing a set of enterprise-ready Habitat plans or common languages, common platforms, such as like BigData, Cassandra, Spark and real common kind of web apps. And so Habitat continues to evolve, and just in the past couple of weeks we announced the pre-release of Habitat for Windows. So along with what we had for Linux support, people can now use Habitat to package up their Windows-based applications. And from our view, Habitat is now the premiere packaging solution for containers. It’s the fastest, easiest way to package an application and have it run in a container and be able to manage it across the life cycle.

Shimel: That rocks. I mean, because certainly, Ken, the one thing that we’re seeing here is the – you know, we talk about the Internet time crunching or compressing time. The pickup on containers? I’m old enough, and probably you are too – remember when VMware first came out, right?

Cheney: That’s right.

Shimel: And through virtualization, it went really quick. The move to virtualization went a lot quicker than let’s say the move off of client server or something like that, from peer-to-peer, or peer-to-peer to client server or what have you. But the move to containers – man, I’ve seen several surveys recently that showed container in production numbers upwards of 50 percent of organizations. And I thought frankly this wasn’t true, but it says something. Right?

Cheney: Yeah well, you know what’s interesting though about that move? What’s interesting about that move, Alan, is that the majority of containers that are running out there are being used almost like a VM. You know, 75 percent of containers right now are running a full OS.

Shimel: Yep.

Cheney: And they’re basically just using it as a way to kind of attempt a higher level of abstraction and remove a little bit more of the infrastructure complexity that they were encountering when they were using VMs. So what I think the next wave is going to happen, I think people are going to actually figure out how to actually better package their applications to actually take advantage of what they can do to remove unwanted dependencies and remove a lot of the weight around the application that they’re currently doing today when they just package up the full OS and all of the broad set of unassociated dependencies that they’re packaging right now. So I expect that’s going to be the next wave and we’re there to help them when they go there.

Shimel: Agreed, agreed. Ken, any shows or conferences coming up, other than of course the DevSecOps that I mentioned, where Chef will be presenting or a part of?

Cheney: Yeah, well yeah, we will be at VM World, coming up. We’ll be at Microsoft’s big event that’s coming up. And of course as we look toward November we have our own community summits that happen in the fall timeframe. So those are a great way that we bring our community together. So the first one is going to be in London Oct. 10 and 11. So that’s coming right up as well. So expect to see Chef and our brand showing up at a lot of different places in the coming months.

Shimel: Great. Very cool. Well Ken, we’re about out of time here. 15 minutes goes quick. But hopefully it won’t be a few months before you’re back on –

Cheney: Yeah.

Shimel: – and we can pick up where we left off here.

Cheney: Yeah, Alan, enjoy your travels and thanks so much for the time today.

Shimel: Not a problem. Ken Cheney, CMO of Chef, here on DevOps Chats. This is Alan Shimel for DevOps Chat and devopsy.kinsta.cloud. Thanks for listening, everyone, and we’ll see you on another DevOps Chat soon.

— Alan Shimel

August 21, 2017
Chef Applies IT Automation to Meet Compliance Mandates

Much of the productivity waste associated with compliance requirements stems from the fact that most developers and IT operations teams aren’t aware how an application might be non-compliant until it’s too late. Aiming to eliminate that fundamentally inefficient approach to meeting compliance mandates, Chef this week tightened the integration between its Automate continuous automation platform and InSpec, an open-source project lead by Chef for specifying compliance and security policies.

At the same time, Chef enhanced support for Docker within InSpec and released InSpec-AWS, InSpec-Azure and InSpec-vSphere as incubation projects that IT operations teams can employ to test and audit for compliance. Chef also pledged to extend InSpec beyond operating systems to include support for middleware to address compliance issues across an entire platform. The announcements were made at the ChefConf 2017 conference.

Chef CEO says Barry Crist says that in much the same way Chef automation software turns infrastructure into code, InSpec turns compliance into code. Tighter integration between the two platforms should enable IT operations teams to automate compliance testing before code gets deployed in a production environment. In effect, Crist says, compliance becomes just another continuous process automated by Chef. This approach not only leads to faster application deployments, it also reduces the amount of time auditors must spend checking on where production applications comply with a particular mandate, as all the documentation has already been generated.

Compliance mandates are, in many ways, the enemy of innovation. In fact, it often takes longer these days to navigate compliance issues than it does to write a piece of code. Rewriting code to comply with multiple compliance requirements over several different manual tests is a waste of time and money. InSpec makes it possible to make compliance testing a part of an integrated DevOps process in much the same way security testing is fueling DevSecOps.

As the number of platforms that IT operations teams now need to support has expanded, the compliance testing process has become even more complicated. Crist notes that for DevOps to be truly effective, it needs to encompass not just the deployment of code but also making it possible to make compliance and security testing an integrated component of the application deployment process. Of course, as an open-source project there’s no reason developers can’t incorporate InSpec into any continuous integration/continuous deployment (CI/CD) framework they choose. Chef is just moving to make InSpec a more natural extension of its IT automation framework.

There’s a lot of commonality in the controls implemented across various compliance specifications. That creates opportunity to automate controls to reduce the need to rework an application, either before it gets deployed or, worse, after it has been deployed into production. Longer-term, the level of commonality across compliance controls also creates an opportunity to leverage machine-learning algorithms to automate implementing those controls.

Of course, there will also be controls that will be unique to one compliance standard or another. But the days when IT organizations need to manually address those compliance controls one application at a time are coming to an end.

— Mike Vizard

May 25, 2017