Category: Enterprise DevOps

  • DevOps Chat: IBM z14, Built for DevOps from the Ground Up, with Hayden Lindsey

    DevOps Chat: IBM z14, Built for DevOps from the Ground Up, with Hayden Lindsey

    IBM recently GA’ed perhaps its biggest zSystems mainframe release in the last decade or longer. The z14 mainframe from IBM has also been designed from the ground up to be DevOps-empowered. From APIs and other integrations to language support, the z14 is IBM’s most advanced mainframe ever. I had a chance to speak with Hayden Lindsey, VP of DevOps Enterprise Systems for IBM, about the z14.

    Hayden gives us the low-down on how the z14 has been designed to work in hybrid environments. It also works well with modern CI/CD tools such as Urban Code and others. Additionally, APIs give it the ability to integrate with many different environments, tools and apps. It is truly a mainframe for today and beyond. Hayden does a great job of describing all of this.

    As usual, the streaming audio of our conversation is below, followed by the transcript of our conversation. Enjoy!

    Audio

    Transcript

    [su_pullquote align=”right”] [su_custom_gallery source=”media: 117632″ width=”130″ height=”130″] Hayden Lindsey

    Hayden Lindsey, Vice President & Distinguished Engineer – DevOps for Enterprise Systems. Hayden is responsible for the development, product management, sales enablement, and services of this portfolio. He has worked at IBM for 28 years and has held a variety of positions, including software developer, product architect, product manager, co-leader of the Eclipse tools platform, and director of the WebSphere Studio/Rational modeling and construction tools. He has a bachelor’s degree in Mathematical Sciences from the University of North Carolina at Chapel Hill.[/su_pullquote]

    Alan Shimel: Hi, everyone, this is Alan Shimel, devopsy.kinsta.cloud, here for another DevOps Chat. Our guest on this episode of the chat is a good friend of ours, Hayden Lindsey, VP, Enterprise DevOps at IBM. Hi, Hayden, welcome to DevOps Chat.

    Hayden Lindsey: Hi, Alan. Good to be with you again.

    Shimel: Good to have you here. Hayden, it’s a big day—big day for the folks over at the IBM Enterprise team. It’s the general availability of z14 today, isn’t it?

    Lindsey: It sure is. Yes, we announced this back in the middle of July and we’ve now been making our products available. A lot of mine were available the latter part of last week, and now the box itself, GA, today. So—a big day.

    Shimel: Fantastic. And Hayden, just—you know, our audience being so focused on DevOps, they may not realize the full scale of what the z14 release is about. What it is, it’s a combination of hardware—as you mentioned, a new box. There’s a ton of software, there’s a tone of integration—it’s just chockfull of all kinds of goodies. Can you give us kind of a—at a high level, the scope of z14?

    Lindsey: Sure. And I think what you see is the result of pulling software, both middleware and tools, from the various divisions that we used to have like Rational and WebSphere and moving those into a single z or mainframe focused brand.

    And so, in contrast to most prior hardware releases where you’d be hearing about the speeds and feeds and, you know, the z13 had the fastest processor of any box on the planet—this time, the focus is more on the entire solution that involves both hardware as well as software and tools. And so I think—we worked on that, we did design thinking, we had sponsor users, we did the whole nine yards form the hardware up through the operating system, middleware, and tools, and I think that made a big difference.

    And so this time, for example, we have something called pervasive encryption. We all know how important security is, we all hear about these breaches that are happening—Experion being the most, or Equifax being the latest one. And, of all of the records that have been stolen, and I think it’s something like 9,000,000,000 since 2013—only 4 percent are encrypted. And this new mainframe lets you encrypt 100 percent of the data with no impact to performance. So, that’s just—that’s one example. Clearly, there’s hardware stuff there, but there’s also middleware that had to be involved in making this solution. So, that’s an example—it’s really cool.

    Shimel: That’s actually a big example, Hayden, because you know what? Encryption has been around—I mean, I spent 20 years in the security space and encryption has been around for most of that. The problem was the hit you took on performance when you encrypted and then had to decrypt and re-encrypt just the latency and—

    Lindsey: And the programmer had to do it. [Laughter] Or the database administrator had to set it up for the database, but then what about when it was being used in the application? And so that was—this is what is being solved, where you don’t have the time inhibitor of people doing it, and then you also don’t have the performance impact where it makes a system unusable.

    So, I think this could be really groundbreaking for the companies and the government agencies around the world that deal with sensitive data.

    Shimel: Sure—and we all do now. Hayden, you said something else, though, that I wanted to key in on, and that is sort of moving away from the speeds and feeds, if you will, metrics of—you know, to measure the value of anything. And I worked with a woman once, and she used to call that bag diving. Right? You know, when you gotta start getting into the speeds and feeds to show the value of your product, you’re diving into your bag instead of talking the true value.

    Lindsey: Right.

    Shimel: So, I’m happy to see that that’s kind of the path that IBM is taking, talking about the value, but I also think it’s a result, as you said, of so much being added into here, right, from the design thinking around the entire release, if you will, to the integration with things like cloud, with the DevOps entire tool set and methodologies that IBM and others have developed.

    This is so much more than just, “Oh, we put—this is the next iPhone with the faster processor.” This isn’t the next z with the faster processor, it’s much more than that. As I think I—maybe you said it earlier or maybe we talked about it off-mic. This is probably the most substantive release for z in at least a decade.

    Lindsey: I think it is, and of course, the market will tell us if our aspirations come to bear or not, but I certainly think the encryption element is just—well, I think it’s phenomenal, and it could be very, very, very significant for virtually any company out there.

    And then we’ve got more on machine learning and analytics so that you can do in transaction analytics rather than doing it after the fact and having to try to back out or just write off fraudulent transactions. And, of course, we’ve done a lot with our DevOps portfolio compilers. But I really do think, maybe, since we put Linux on the platform, it’s that significant.

    Shimel: Wow. Huge. So, Hayden, we are about DevOps here, so let’s focus in a little bit. I mean, one of the nice things is, I’ve seen some briefings around how you can use the new z14s with the IBM Enterprise DevOps solution and really sort of leverage—I don’t wanna use the term bimodal multi-speed nonsense. Leverage integrating your z system with cloud systems into one coherent business strategy. How’s that sound? [Laughter] So, talk about it.

    Lindsey: Yeah, so one of the three themes for the release is open and connected, and that obviously ties into a hybrid cloud strategy. It ties into, you know, a digital economy or digital transformation. It ties into API economy—whichever angle you want to take, you have to be able to make the data that resides on the mainframe, which is estimated that roughly 80 percent of the Enterprise data in the world resides on and is managed by the mainframe, and about 70 percent of the business transactions.

    And so, if you want to, though, make that available to your new iPhone X app, as we saw introduced yesterday—you know, or 10 or whatever they’re gonna call it.

    Shimel: Mm-hmm. Yeah, they’re calling it 10.

    Lindsey: They’re calling it 10 using a Roman numeral.

    Shimel: Yeah.

    Lindsey: But, you know, if you’re gonna do that, you have to make it easy to access the data and transactions, find those transactions, understand them, because very likely, they were written by somebody who’s not around any more or had moved on to other jobs. And so, you need to have tools that make API enabling and opening up and connecting to the mainframe easy and efficient, and just as easy, just as efficient as you would if you were opening up something on the Linux platform—although, of course, you have Linux on z—or a Windows box or anything else. So, that’s—it’s essential that we allow z to participate in the broader digital economy, API economy, and that’s what my team has been focusing on, along with other teams in IBM.

    Shimel: Sure. So, now, Hayden, in that same vein, you know, there’s still that mainframe attitude where, you know, we don’t open our mainframe up to all of these APIs, and maybe we don’t want to hook into our systems of engagement, you know, up in the cloud or what have you.

    For clients listening out there—and a lot of our listeners are Enterprise class, they work in Enterprise class organizations—how important is it that they really consider opening up, you know, their mainframes to this API economy, as you called it?

    Lindsey: Well, I think increasingly, certainly, the IT and line of business executives, the clients with whom I work, are realizing they have tremendous assets that they could leverage if they were to make them available. Nobody—at least no companies I know of, or government agencies, have the resources to just recreate everything from scratch. And, I think if you look back at what the analysts were saying back in the SOA era, they said, you know, it was five times faster to create a service leveraging something that exists that already does the same thing or something very similar than it is to create it from scratch—not to mention the fact that, if it already works, you’re likely to have higher quality as well. And I think the exact same analysis would prove to be true now that we’re talking more about RESTful APIs.

    So, I think more and more clients realize they need to expose that value. The key is, just, it’s sometimes daunting when you’ve got—you know, the estimate is like 250,000,000,000 lines of COBOL, not to mention PL/I and Assembler and 4GLs and so forth. It’s getting your arms around applications that were probably created in a more monolithic, less well-structured fashion. You need to do refactoring. You need to discover what is there, and then put together a plan for doing that API enablement.

    But I think people know they need to do it, it’s just, sometimes it’s viewed as a daunting task and they have to figure out how to get started, and we have something called the digital transformation model that we’re rolling out to help our clients put a plan in place.

    Shimel: Got it, and I agree. So, you’re just setting me up for my next things here, but—so, that’s the biggest question we hear at devopsy.kinsta.cloud, Hayden, which is—hey, this is all great. How do we get going? How do we roll this? How do I get executive buy-in? How do we—is there a one, two, three? Is there a blueprint I can follow.

    So, z14, let me throw it right back to you—where do we start? How do they get moving with this? How do they—and specifically, how do they start incorporating the DevOps and API stuff into this?

    Lindsey: Yeah, it’s the question that I hear from virtually every customer I talk to, and I’m getting ready to do a tour in Europe next week, and I’m sure it will come up in every single meeting. And the fact is, every customer is different, so there’s no single answer. And, within any given client, they have applications that span the gamut from nearing end of life to ones that are under active or brand new development. And so, what we’ve—but, of course, there are patterns and similarities across clients and application types, and there are best practices that we have gleaned by working with clients through the years, and that’s where I mentioned this thing called the digital transformation model.

    We’ve shifted to this way of working with our clients to focus on applications rather than one answer for the Enterprise—hey, you should start with deployment automation, or you should start with test automation. That answer should not be an Enterprise answer, it should be for this suite of applications because of what their current state is, the as-is state, and what you desire the to-be state to be. If you just need something to run efficiently and you’re gonna replace it in a few years or retire it, all you want it to do is run efficiently.

    So, use our latest compilers, but you know, the fact of the matter is, putting in a fully automated pipeline is likely a wasteful effort. Whereas, on the other end of the spectrum, for the things under very active development, that’s where you want to do DevOps in its fullest, and really, it’s just a matter of what steps do you take to get there.

    And so, this model that we’ve come up with has four levels—the first one I described about, just do your compiling, is run—just run it. Just run it efficiently. The next is maintain, the next is expose, the next is evolve. So—I mean, and I don’t know if we’ve got the best names in the world, but the fact is, you really need to think about, for the business, what degree of DevOps adoption do you need for a given application or application suite. So, that’s the more, I think, realistic answer that we’ve been using with clients this year and a tool that we’re using that’s different from what we’ve done in prior years.

    Shimel: Excellent. So, Hayden, I mean, just to illustrate this, I remember back at Interconnect last year, you know, we had the whole little DevOps TV booth set up, and I was interviewing some IBM clients. And one of them, it was a bank, and I forget if they were from Poland or the Netherlands now, or maybe even Australia. They were IBM’s z System customer and they had just updated, I wanna say, from COBOL 4 to 5—maybe it was 5 to 6, I don’t remember now.

    But they had just updated their version of COBOL, and it made a huge, huge difference for them in terms of performance and—

    Lindsey: Yeah. Yeah, exactly. You know, we’ve—one of the things that our compiler team does every single release, and this starts from the very earliest days of, we were just imagining what the new mainframe or new power platform is going to be. We were working hand in hand with the hardware folks.

    And, you know, with COBOL specifically, where something like 70 percent of the world’s business transactions are running COBOL, obviously, performance does matter, and so this is where you can get into a little bit of feeds and speeds. Mainly because, not because—some of the numbers aren’t eye popping, but it’s because you can be more responsive for your clients. But with some compute intensive apps, they run 16 times—not percent, times—faster on the latest compiler, which is 6.2, versus 4.2, which may be where that client was coming from. So, that’s skipping the version 5 things, but we still have the majority of our clients on the 4.2 version.

    So, it’s tremendously faster, but this is by doing this joint development. You know, if you put new instructions or new vector support in the hardware and the compiler doesn’t generate these new instructions, well, then, obviously, no client is gonna be able to take advantage of it unless they want to write their own assembler. So, you know, our compilers are delivering tremendous value, and I gave you a COBOL example, but you get similar results with PL/I and with our automatic binary optimizer that basically just does the, tries to do the work that the compilers do, but just working on binaries rather than having you go back to the source.

    So, it’s a big deal. You know, and the less—it’s maybe a bit more mundane. Yeah, the stuff runs faster and, as you said, diving into the bag. [Laughter] But, you know, when you’ve got numbers like 16X or 5X or what have you, that’s huge.

    Shimel: It sure is, it sure is. So, Hayden, you know, I mentioned Interconnect, too, and as we’re running low here on time, I just wanted to quickly prompt you on that and let our listeners now.

    So, this year, Interconnect is part of an even bigger kind of unified conference that IBM has put together called Think—Think 2018, I believe it’s called, and if I’m not mistaken, it’s March.

    Lindsey: Yes.

    Shimel: Go ahead.

    Lindsey: Well, you might see a trend here. [Laughter] Because, you know, Interconnect was the combination of Impact and Innovate. Innovate—I think that’s the name it was. I’ve kinda forgotten, but anyway, we had a Rational conference, we had a WebSphere conference, we had IoT stuff and all that. And now we combine those into Interconnect—at least the WebSphere and Rational stuff. And it got bigger, and now we’re doing something that is more or less IBM-wide and it’s called Think. It’s gonna be the week of March the 19th at the Mandalay Bay in Vegas, so still in Vegas like most of our other conferences have been of late.

    And so, I think there’s pros and cons to having a more broad, where we can tell the full IBM story, and that’s the focus, and you’ve got SMEs from all the different areas of the country, be it cloud, cognitive, Zpower, what have you. But I think that’s great, but because you do have those experts, and we will have the breakout sessions or keynote sessions on specific areas, be it DevOps or z or compilers or deployment automation like UrbanCode.

    So, there will be still those specialty sessions, and yet you get the big picture. And the one thing that I would like to ask your listeners is, the call for papers is open, and you know, we are shooting for having 100 percent of the talks in the various tracks either be a customer or partner only, or being customer and/or partner jointly with an IBM-er, but we’re just really trying to get away from IBM-er talking head telling a story without a partner or a customer there saying what they’ve done in real life with a given solution in a given area.

    So, we’d really love to have as many partner and client nominations as possible.

    Shimel: I love that idea. You know, I shared a panel the last two years, Hayden, on Dev—I think they were on, well, it was on DevOps and then SecOps. But, you know, I tried really hard to balance it with—yeah, we had one person from IBM, but we also had an analyst and then we’d have a partner and then a practitioner. And I think that sort of balance gives it, it makes it—

    Lindsey: It adds some legitimacy to the whole thing and different perspectives.

    Shimel: Yep. Just real quick, for anyone listening, if you go to IBM.com/events/Think, you can get all the scoop there and follow the links to submit. And as Hayden said, if you’ve got something to say and share with people, it—having been to three or four of these, or more, it’s a great platform. So, I’d encourage everyone who would like to participate to do so. And even if you don’t want to speak, but if you want to go really understand what’s happening, whether it be with Watson or z Systems or hybrid cloud or Bluemix or any of those things. You know, if any of those are important to you, I think it’s a great show to attend.

    Hayden, we’re about out of time—we’re actually over time, but that’s okay. Thanks so much for taking the time on what is a busy day with the general GA of z14. Best of luck with it. Let’s check in, I don’t know, maybe before the end of the year, and see how—you know, what kind of market reception a lot of these features and capabilities are receiving and we’ll see where the rubber is meeting the road.

    Lindsey: Sounds good, Alan, and thanks, again, for your time.

    Shimel: Thank you for your time. Hayden Lindsey, VP, Enterprise DevOps with IBM on GA Day for z Systems z14. Thanks, Hayden, have a great.

    Lindsey: Okay. Bye for now.

    Shimel: This is Alan—thank you. This is Alan Shimel for devopsy.kinsta.cloud. We’ll see you soon on another DevOps Chat.

    — Alan Shimel

  • All Day DevOps: Taking Lambda to the Max

    All Day DevOps: Taking Lambda to the Max

    Lambda is the 11th letter of the Greek alphabet, and it is also the name of the Amazon Web Services (AWS) service that lets you run code without actually configuring a server. Amazon chose the name Lambda because—well, okay, I have no idea. Comment here if you know.

    Thankfully, you don’t need to know that to understand how to utilize Lambda. Matt Williams, DevOps Evangelist at Datadog (@technovangelist), outlined how to set up AWS Lambda at the 2016 All Day DevOps conference in his talk titled, “Hold the Infrastructure: Getting the Most Out of Lambda.”

    First, to understand what Lambda is, we must look at the evolution of servers.

    williams1.png

    We started with physical boxes—we had to buy and maintain these boxes, and no matter how much we used, we had to pay for all of it. We then evolved to multiple virtual machines on a physical server so that we could maximize the resources of physical boxes. Now, AWS and similar cloud services are common. We create a virtual machine that we need, and the service provider maintains the physical servers. We also have containers to keep environments identical across machines and, finally, we arrive at Lambda, which is service that runs code but doesn’t require you to set up the server. It takes care of all of that automatically, and you only pay for the time that the function is actually running. So, if your function takes 500ms to run, you pay for 500ms to run it once.

    Lambda is trigger-based. When it receives a trigger, it executes a specified function. The trigger can be a variety of actions with AWS services, such as when a file shows up on S3 or an Alexa skill is requested.

    Lambda is serverless, as far as you are concerned.

    Lambda is stateless. This means you can’t rely on local states, so, if you need to retain information between function calls, you have to save that data somewhere else.

    Lambda executes functions. Each function:

    • Is code in a container on LXC on AMZN Linux
    • Does not have restrictions on what it can do
    • Is priced based on exec time, quantity and memory

    Lambda links to various AWS services, such as:

    • Files (S3)
    • Database Updates (DynamoDB)
    • Streams (Kinesis)
    • Messages (SNS, SES)
    • IoT (Echo, API Gateway)

    Matt mentioned several good use cases, such as bootstrapping (the first 1M requests per month are free), cron jobs, file manipulation, API translation, social media stream processing, scaling infrastructure, mobile/IoT backends and Alexa skills, to name a few.

    He also mentioned three specific examples to explore:

    Matt then outlined how you can get started and noted that setup is simple as long as you think of application development as simple (write the code, set up on AWS, and debug). He walked through a specific example of automatically resizing images on S3 when they are added. You can watch his full presentation with screenshots here.

    Matt also provided a list of some AWS Lambda Tools:

    Matt rounded out his 30-minute presentation with a discussion regarding the need for monitoring and how to use Lambda for monitoring.

    You can watch Matt’s entire talk online here, where you can dig deeper into this subject. If you missed any of the other 30-minute presentations from All Day DevOps, they are easy to find and available free-of-charge here.  Finally, be sure to register you and the rest of your team for the 2017 All Day DevOps conference here.  This year’s event will offer 96 practitioner-led sessions (no vendor pitches allowed). It’s all free and online Oct. 24.

    — Derek E. Weeks

  • From Waterfall Incident Management to Collaborative IT Operations

    From Waterfall Incident Management to Collaborative IT Operations

    What can Ops learn from Dev in the transition from sequential, reactive IT support to real-time DevOps?

    One of the signature ideas of DevOps is right there in the name: the blurring of the boundary between the historically separate worlds of Development and Operations.

    On the face of it, this seems like a great idea. Developers have always grumbled about how Operations is preventing them from deploying as rapidly or as frequently as they would like. Meanwhile, Operations people have harbored a dark suspicion that developers might code with more attention if they were the ones getting paged at 3 a.m.

    Another of the great things about DevOps is that it is pretty easy to adopt in one area or project without needing to get the entire company onboard before you can get started. This means that you can quickly produce some interesting results in terms of accelerated development times and release frequencies, faster resolution of problems and more time spent innovating instead of firefighting.

    Armed with these results, DevOps evangelists spread through the organization, telling their story and recruiting other departments and teams to join them in the bright future.

    This is Where the Problems Start

    Complex enterprise IT environments are, well, complex. Different teams have adopted different tools and methodologies to do their jobs, and are justly resistant to throwing out something that works for them. Specialists in different areas require their own specialist tools to be able to do their jobs. Because of this, when an incident occurs, all of that agility and collaboration that DevOps brings goes out of the window in favor of something that looks much more like a classic “waterfall” approach:

    • When an event is received, it is routed to one particular team
    • The event is placed in a queue
    • Eventually, someone from the relevant team takes a look at the event and makes a determination of whether it falls in their area or not
    • If it’s in their area, they begin working to resolve the issue within their team
    • If it’s not (believed to be) in their area, they reassign it, either back to the front-line team, or to whichever team they believe to be responsible (often, this is the network team)
    • Repeat until either someone takes ownership and fixes the issue, or it becomes a Major Incident and everyone gets paged at 3 a.m.

    Just to complicate matters, there may be multiple instances of this process going on at any one time, as two or more different teams grapple with what eventually turn out to be different aspects of the same problem.

    Why Does This Happen?

    In health care, there is only so much that can be done by way of prevention. There is always the need to react to patient’s issues, starting from their symptoms. In the same way, the starting point for Operations is monitoring—gathering the symptoms of the IT infrastructure to be able to return it to health as quickly as possible.

    To understand problems with the Ops part of DevOps, we needed to start at the beginning, with those symptoms. We wanted to do this the scientific way, so we polled attendees of Velocity and Monitorama to understand their top monitoring challenges.

    The interesting fact was that across both populations, the main challenge listed by attendees was “alert noise/fatigue/volume.” Simply put, Operators were drowning in alerts to the point that they had completely lost the ability to understand what was actually important, let alone an overall picture of how those alerts related to the service that IT was supposed to deliver.

    This is another parallel between IT and health care, where alert fatigue has long been a concern, and with potentially far more serious consequences than in IT.

    A big part of the cause of the alert fatigue on the IT side is the proliferation of specialist monitoring tools. In the poll, 66.67 percent of people we surveyed have five to 10 monitoring tools in place, and yet 61.90 percent of them are still struggling with that problem of alert noise/fatigue/volume.

    To be clear, those different tools are each very important. Trying to monitor the network with an APM tool is probably not going to go well for you. They only become a problem when it becomes too difficult to build up an overall picture of what is going on, and sharing that between different people to solve problems faster.

    So What Can Be Done?

    Dedicated Operations organizations have long worked to assemble tools and techniques that can give them a complete view of the environment that they are supporting, and of the services that run in it. The accelerated pace of release that comes with DevOps is causing the old models to break down, but fortunately, it also carries in itself the seeds of a new approach to the problem.

    As development moved from the sequential waterfall model to a more rapid, collaborative, and agile approach, operations teams can learn from that transition to come up with a more agile approach to IT Ops.

    What would this look like? The first thing to do is to get rid of the blame-storming and buck-passing: “It’s the network!” “No, it’s the database!” I don’t know about you, but personally, I’ve been in enough of those meetings to last me for the rest of my life, thank you very much.

    Let’s All Go Fishing in the Data Lake

    Instead, a common perspective is needed, so that everyone can quickly agree on what is going on and what needs to be done.

    This does not mean throwing out all of those specialist monitoring tools! They are there for a reason. Rather, gather all of those symptoms in one place where the whole team can collaborate to make sense of them more easily. These days, it should be easy to get information in and out of the various components of your toolchain (and if it’s not, that’s a glaring warning sign that maybe it’s time to migrate off that particular tool).

    This will, of course, produce a huge pile of data, so it’s not sufficient to stop here with the creation of a “data lake.” Things disappear into lakes and are never seen again—in which case, why did you bother gathering them in the first place?

    What you need is a way to make sense of what is going on in the lake, and show that to the right human operators. In the same way that developers resent sitting around waiting for the Change Advisory Board to meet, operations people get frustrated and bored just staring at a screen waiting for something to happen. If instead you can use automatic techniques to assemble all the related symptoms of a single issue together in one place, that will give you a good starting point to resolve that underlying issue quickly.

    Brainstorming, not Blamestorming

    This is where the collaboration comes in. Again, people are doing this already, but they are doing it in a disconnected way. The tools have evolved, from email and IRC, to Slack and Google Hangouts, or even (heaven help us all) WhatsApp. What has not evolved is the usage model; it’s still ad-hoc and reactive, and requires lots of copying and pasting.

    Operations people can learn from what their developer colleagues have built or adopted. Kanban boards are a great technique for tracking open issues and who is working on them. ChatOps is a fantastic way to integrate human-to-human communication with human-to-machine communication. Integrated collaboration tools capture knowledge that otherwise is stuck in people’s heads or inboxes, unavailable to their colleagues.

    I’m an Ops guy at heart, but I can recognize a good idea when I see it. Our Dev colleagues have come up with quite a few that we could stand to adopt in Ops. Let’s learn from what they already know to work, instead of trying to reinvent the wheel.

    — Dominic Wellington

  • 9 Essential Ingredients for Modern Enterprise Applications

    9 Essential Ingredients for Modern Enterprise Applications

    Digital transformation. It’s the latest in a series of buzz phrases intended to describe how enterprises are embarking on top-to-bottom modernization of their IT environments. It’s easy to see why it’s a hot topic. Emerging technologies including cloud, containers, microservices and distributed applications hold the promise of greater flexibility, lower cost and faster release cycles. The benefits to companies that make this journey can be enormous.

    Unfortunately, making this journey can take years with today’s manual processes and antiquated tools. Not to mention that most IT budgets are at least 80 percent focused on keeping the lights on. Application modernization projects can be extremely complicated since legacy, monolithic enterprise applications are intrinsically dependent on underlying infrastructure and data stores. Cloning complex application environments, including critical production data, for upgrades or testing often can take days or weeks, slowing down cycles and contributing to lower quality releases. Even new applications are too often designed with infrastructure dependencies, trapping organizations onto specific cloud, server or container vendors. This vendor lock-in reduces choice and leaves IT organizations open to unexpected price hikes down the road.

    The truth is that the way we have designed, built, deployed and managed enterprise applications needs to change. Business-critical software is too often built in monolithic blocks that are nearly impossible to move to new infrastructure or storage options without lots of effort and time. The applications are hard to upgrade since updating even one small portion means rebuilding and retesting the entire thing. Managing the various versions of the discrete application modules and the resulting testing permutations can be quite challenging. It’s not unusual to hear from CIOs that they feel trapped in a world where they can only reasonably expect to update critical enterprise applications just twice a year at most.

    Business needs to move faster. There’s no reason why release cycles, even for enterprise applications, couldn’t be every few days instead of months. Application quality should actually be better in this world. Continuous delivery models such as DevOps should be an integral part of the process. And security should not be an afterthought; it should be built-in. To accomplish this, the future of enterprise applications must look very different from today.

    What are the essential ingredients for modern enterprise application platforms and tools in this new world? Here are a few suggestions.

    Complete Portability, Without Code Change

    Enterprise applications must be abstracted and separated from any dependency on underlying infrastructure, including sources of data. By abstracting the functions of an application into components that can run anywhere, it’s possible to then move the application to different infrastructure combinations, without changing a single line of code. It’s only through complete portability between cloud environments, container tools and servers that IT organizations will break vendor lock-in and gain the flexibility required to move their applications to vendors that offer the best combination of price, performance, reliability and features.

    On-Demand

    All enterprise applications should be on-demand, meaning the provisioning of new instances should be hands-free and fully automated. This implies tight integration with orchestration tools that most companies are already using to provision infrastructure. The key difference here is that the provisioning of storage options also needs to be fully automated. This is made possible by the fact that modern applications won’t have any dependencies on the specific data stores they rely on. This on-demand nature is not restricted to technical teams but includes business users that can easily perform user acceptance testing by simply provisioning full-fledged application environments.

    Self-Healing

    Modern enterprise applications should be self-healing, which means applications not only should have measurable data points to define their health but can then react based on these metrics. They must be able to scale up (or down) based on performance characteristics up to a certain cost threshold, restart on a healthy node in case of infrastructure bottlenecks and cache or deploy closer to the user based on latent use performance.

    Composable (DevOps-friendly)

    All enterprise applications should consist of a number of building blocks: APIs, web servers with content, databases seeded with data, security policies, the operating system, network configurations, deployment configurations and application binaries. By breaking apart monolithic enterprise applications into smaller components that can then act as independent services, IT organizations gain the flexibility to upgrade and automatically deploy individual components through standard DevOps processes and orchestration tools. The services can run anywhere desired, giving flexibility of choice in infrastructure or sources of data. And developers can use a point-and-click UI to pull pre-defined components together to build larger functions of an application.

    Model, Snapshot and Clone

    Once an application is separated into its essential components, a catalog can be maintained that makes it easy to create new versions of the application. Developers, test engineers and DevOps teams can then pick and choose whatever combinations of components they need to create or clone complex application environments for testing or deployment. The state of the application environments can be snapshotted, cloned and replicated. This cloning process should be efficient and quick. This dramatically speeds up development and testing processes, which to-date have been largely manual.

    Scalable

    Scaling application and infrastructure environments automatically, on-demand, and in near real-time is one of the essential capabilities that makes modern infrastructure so valuable. Any solution that is used to develop, test, deploy and manage enterprise applications should offer these same capabilities, giving the ultimate in cost flexibility, performance and scalability. As containers are moving quickly from the lab into production environments, look for a container-based solution that complements this strategy.

    Security is Included

    Application security should not be tacked on after deployment. This slows down continuous delivery processes like DevOps and creates friction between development/ops and the security team. Instead, security should be considered an essential component of the overall application environment and treated the same as any other component and baked into the application from the start.

    Push-Button Upgrades

    Upgrading enterprise applications should be made much easier by taking a point-and-click, push-button approach through a modern UI and integration with continuous deployment tools. Once an application is blueprinted into its various components, using a modern UI to choose versions and assemble the final application becomes entirely possible, saving time and reducing manual errors.

    Single-Pane-of-Glass Application Management

    By taking an end-to-end view of the enterprise application development, testing, deployment and management process, it’s now possible to create a “single pane of glass” console that gives development and deployment teams a powerful way to manage what’s been deployed to various infrastructures in the organization. With this approach, it’s also now possible to collect invaluable utilization and cost metrics which can help an IT organization continuously improve their portfolio.

    Enterprise applications of the future are going to look very different from today’s monolithic behemoths. By leveraging new technologies and processes, IT organizations will be able to dramatically speed up their modernization efforts and release cycles, improving competitive position. The inherent cost savings of this approach will be transformational as well, as IT budgets shift toward net-new innovation, not just keeping the lights on. The future looks bright indeed.

    About the Author / Rahul Ravulur

    Rahul Ravulur is the cofounder and CEO of appOrbit, which was founded in 2014. Most recently, he was the Director of Product Management at VMware managing the Availability suite of products. Before that, Rahul managed the vCenter Server and Client set of products with a special focus on scalability and extensibility to chart the evolution of vCenter to a broad virtualization management platform. Prior to VMware, Rahul held engineering leadership roles at BMC Software and Marimba.

  • NetOps vs. DevOps: New Data is Enlightening

    NetOps vs. DevOps: New Data is Enlightening

    Traditionally, we have seen development—and, by extension, DevOps—as having a less-than-cordial relationship with network operations (NetOps). Since the goals of the two groups are sometimes at cross purposes—DevOps wants access to make everything run automated, and NetOps wants to control a production environment so that it is stable—we’ve naturally talked in those terms.

    I’m a developer who learned other skills, including operations, along the way. I can manage to get things done on F5 Networks or Cisco Systems gear, but I’ve sat with true experts while they worked their magic and have a lot of respect for the NetOps crowd. So this topic has always been of interest to me.

    Now, according to a recent survey conducted by F5 Networks, we can see that the animosity we have all been lead to believe is there really isn’t. There is also a press release here, and surprisingly, it offers more information from the survey than the blog does.

    There is a ton of interesting data about rate of automation, views of other teams and deployments in the survey results, making it well worth a read-through.

    While DevOps and NetOps remain at odds regarding how much pipeline access should be available via self-service and automation, for the most part both groups view the other as on the right track. Eighty-two percent of DevOps and 76 percent of NetOps agree that each other prioritizes “the right things.” Clearly, there is common ground across the digital divide in at least goals and focus, if not always in the organizational chart.

    This, to me, says that DevOps is having the proper impact on the organization. That what’s important is getting prioritized on both sides of the pipeline. Or it says that this was always the case, and DevOps has had little impact. Historical surveys, however, seem to indicate that there was not always this level of belief between the two groups.

    Interestingly enough, there has been a lot written about whether your DevOps methodology should be operations- or development-driven, but networking has always been a bit off to the side. Not the type of operations that historically is discussed in DevOps; it sat with teams such as Security and QA working with DevOps but not generally a part of the DevOps teams.

    The results here would indicate that network automation is occurring at a healthy clip (makes a guy want to look at vendor APIs again, and see what’s changed to enable this fact), and if the majority of DevOps people are saying the NetOps team is focused on what’s important, likely the integration into DevOps is further along than pundits have portrayed.

    Though the results indicate that many DevOps teams want more access and control of network operations than they have, this is likely the natural outgrowth of the different views these teams have always had. Network operations’ goal is rock-solid infrastructure. DevOps’ goal is speedy delivery. These two things can be at odds relatively easily.

    I’m hoping that F5 offers us more access to data than a blog and a press release, but honestly what is in those two is enough to get you pondering about the state of NetDevOps (or whatever we’re calling it this week) versus the perception.

    Disclaimer: I am a former F5 employee, and have a positive view of the company and its products. My wife is also F5’s Principal Technical Evangelist, and had a hand in this survey. While this doesn’t impact writing about survey results, you’ve been informed.

    — Don Macvittie

  • All Day DevOps: Modern Infrastructure Automation

    All Day DevOps: Modern Infrastructure Automation

    “We’re no longer an airline. We’re a software company with wings,” claims Veresh Sita, CIO, Alaska Airlines. The success of today’s businesses rests on software. It is an integral part of infrastructure, so we need to always ask, “How can it be better? More reliable? More secure?”

    Nathen Harvey (@nathenharvey) is the VP of Community Development at Chef and the co-host of the Food Fight podcast. He is also an evangelist of automating software operations, which is what he presented on at the 2016 All Day DevOps conference.

    Nathen started his talk by stating that “automation at scale is required for high velocity IT,” making the point that in today’s environment, you can’t wait months between deployments. Enterprises are moving to continuous integration/continuous deployment, and he advocates for continuous automation.

    harvey pic1.png

    To implement automation at scale, you have to start with a dynamic infrastructure by:

    • Provisioning and setting up environments
    • Implementing dynamic scaling of compute resources
    • Migrating legacy workloads to the cloud
    • Deploy in multi-cloud and hybrid-cloud environments
    • Support heterogeneous environments

    harvey2.png

    Several speakers of All Day DevOps talked about automation, and all of them said a critical step is to treat infrastructure as code. According to Nathen, this means that to programmatically provision and configure components, such as servers, databases, firewalls, etc., you also have to treat it like any other code base. This includes version control and automated testing, but also:

    • Automated, full-stack application policies
    • Package and service installation
    • Versionable, testable and repeatable workflow
    • Scalable application policies
    • Management of interdependencies across nodes

    When you treat your infrastructure as code and deliver at scale, you can scale elegantly from one to tens of thousands of managed nodes across multiple complex environments.

    For software to be effective, users have to trust that it will do what they expect it to do. This is true from handheld calculators to self-driving cars, and it is also true when we automate our development and operations. Nathen advocates that we have to build a pipeline to production that creates faith in us that our code is ready for production.  This pipeline must:

    1. Test the code/verify locally
    2. Verify in pipeline with automated testing
    3. Approve with another engineer (aka code review or Nathen’s “four eye rule”)
    4. Build an artifact
    5. Deploy into acceptance environment to automatically test the artifact
    6. Ask someone else, “Do we want to ship this?”
    7. Ship the code with a click of a button
    8. Union (bring together dependencies and run tests)
    9. Rehearsal
    10. Delivered (ship to customers)

    What does the future hold?

    Infrastructure automation is NOT enough. Why? Auditors show up to audit your infrastructure and applications. The problem is, they use different tools—Excel spreadsheets, notebooks, PDFs, etc. This makes it very difficult to audit. Nathen advocates managing compliance documents as code, too. When we translate compliance into code, it can be executed as part of the pipeline so we can test for compliance throughout the entire life cycle.

    Finally, Nathen stresses the need to have application automation, too. Modern applications look like infrastructure as code.

    harvey3.png

    Developers can make applications that operations can deliver and run in production. This makes it easier to consume, run, and keep applications healty. Operable applications need to be:

    • Isolated
    • Immutable
    • Configurable
    • Built on a common interface for monitoring health
    • Rebuildable from sources
    • Reside in common packaging
    • Maintain runtime independence

    As enterprises implement infrastructure, application, and compliance automation, it creates a freedom for people to focus, and people are the key.

    Nathen notes that, “DevOps is a cultural revolution and professional movement, focused on how we build and operate high velocity organizations, born from the experiences of its practitioners.” He underscores that people are the future of automation. Automation places the priority on people, which enables great companies and great products.

    harvey4.png

    You can watch Nathen’s entire talk online here, where you can dig deeper into this subject. If you missed any of the other 30-minute long presentations from All Day DevOps, they are easy to find and available free of charge here.  Finally, be sure to register you and the rest of your team for the 2017 All Day DevOps conference here.  This year’s event will offer 96 practitioner-led sessions (no vendor pitches allowed).  It’s all free and online on Oct. 24.

    — Derek E. Weeks

  • In Determining DevOps issues, Sometimes It’s the Process

    In Determining DevOps issues, Sometimes It’s the Process

    It is pretty much standard DevOps process that when a server instance starts having problems, you simply kill it and start another. This is in line with the idea that servers are cattle and there isn’t a ton of difference between them.

    But it creates a problem that no amount of CI/CD or automated provisioning can overcome. That’s the blindness problem. CI and CD miss some bugs, simply because of the wild variations possible in inputs when humans are involved, or unique issues of hardware or platform that DevOps tries to ignore as much as possible. While our tooling cannot be so over-arching that we have data points on everything, and we can track down problems to the routine that is causing them, a service degrading performance or outright crashing is indicative of one of those blind spots.

    If you’ve been in a highly dynamic DevOps environment, you know this is no simple problem. But it is one we have to resolve, because killing and restarting is simply masking a problem. Indeed, it is this very process that turned Cloudbleed from a simple and understandable programmer error into front page news. They knew their servers would occasionally have memory over-runs, even crashing on occasion; they would just spawn another instance and keep moving along. The ability to create another so easily reduced their desire/need to fix the underlying problem. But the problem was bigger than they knew.

    The problem we face is, of course, how much time and work is invested in monitoring, and how do we know what’s really important. This problem gets worse when you have an instance that stops responding, or starts hogging resources. Getting it offline is imperative, but re-creating the issues that caused it to go off the rails may not be so easy. In fact, often it is not at all easy.

    Long term, we need a way to pick up the exact problem area, create a bug report and get it back into the Dev/CI/CD system that is reliable. Short term, don’t ignore errors in production unless you know exactly what is going wrong.

    Sure, it’s easy to kill and/or restart an instance to “recover” from an error. It’s easy to chuck your app on the public internet without any security, too. Just because it’s easy doesn’t make it a good idea.

    So watch those logs, don’t mask problems and stay on top of it all, even if the complexity sometimes has you reeling a bit. Because application reliability might be impacted—and application availability is, in the end, the whole point.

    — Don Macvittie

  • Move the Needle in DevOps with Continuous Testing

    Move the Needle in DevOps with Continuous Testing

    Shift-left testing, risk based testing, early testing and technical debt reduction are all predicted to be some of the most sought-after trends in product development life cycle. All of these can be achieved with continuous testing through DevOps, as it emphasizes quicker transitioning of quality products to production environment at a reduced business risk.

    Software Engineer in Test (SEiT) or Software Developer in Test (SDiT) was long-spoken earlier and now implemented as part of DevOps culture. As the software side of the organization moves to continuous testing practices, so, too must the product side of the organization. External market forces including competition, innovation quotient, faster time to market, scalability and lean process initiatives definitely play a role in driving the continuous testing trend.

    From a technical standpoint, test environment creation, full test coverage, regression testing and integration testing of multiple endpoints are key parameters to ensure a faultless production environment when a new feature is added.

    Adopting Testing Tools for DevOps Implementation

    Continuous testing implementation not only includes adoption of right tools and test automation, but also a cultural change to incorporate testing at each stage of the agile development life cycle. It will involve changes right from test planning, test scenarios creation, automation feasibility analysis to test execution.

    Teams need to be restructured in an agile setup to align testers closer to developers and operations teams, and have common goals for product/software delivery. This involves changing the culture and organization structure, which generally takes a longer gestation time. Test management tools such as JIRA and TestNG can help expedite this structuring, as both parties would be aware of the product features that need to be delivered and can view the progress at all times. There also are various open-source tools, including Selenium, Appium, Python robot framework/pytest and JMeter, that aptly meet the needs of agile testing.

    Regression testing generally takes up the bulk of the time, as new features need to be tested every time they are added to avoid critical failures. Hence, automating regression tests have become an imperative as part of continuous testing. JUnit, which is a unit testing framework, promotes test-driven development and early code quality improvement. Bamboo can run tests in parallel batches and support hundreds of build agents providing an early feedback to developers. Automation scripts for test data cleanup, load and execution saves the testing effort of the operations team. Selenium test automation framework for web applications is one of the most famous tools for continuous testing, as it works with a host of programming languages including Java, C#, Groovy, Perl, PHP, Python and Ruby. Appium is another tool used frequently for cross platform mobile application testing.

    Ensuring an Effective Product Release

    Testing UI components and functionality of a product through API-based test automation is another important phase in continuous testing. API test scenario creation also generally needs involvement from the ops team. Considering virtualization of devices, along with applications or services, will accelerate test setup environments. Device virtualization using Raspberry pi helps in testing end–to-end use cases in IoT scenarios. Network virtualization for performance testing ultimately helps improve uptime and reduce outages.

    The orientation of continuous testing with continuous integration and continuous delivery pipeline within an enterprise will determine the long-term success and standing of the final product in the market, from a quality perspective.

    About the Author / Urvashi Babaria

    Urvashi Babaria is a Product manager at eInfochips working in new age areas like IoT, DevOps and CloudOps. She is a techno commercial marketer with nine years of experience in product and project management, business analysis and transformation. She has keen interest in areas of data science including visualization, analytics and modeling. Connect with her on LinkedIn.

     

  • The Role of QA in the DevOps World

    The Role of QA in the DevOps World

    The need for faster time to market and superior quality is getting intense in the software development process. Software trends for 2017 suggest that DevOps will grow as a trend and become more widely accepted across enterprises. Organizations are reaping the benefits of DevOps approach and are incorporating the rules within every aspect of software testing. A case in point: DevSecOps—security testing integrated with DevOps—is an emerging concept and is being considered by enterprises to execute application security testing.

    The underlying objectives of adopting DevOps are to make the testing and development process more collaborative, ensure software quality and to help launch the product faster to the market. This makes quality assurance (QA) a key component in the DevOps methodology. New-age applications need real-time changes and updates to meet the growing demands of the consumer. This requires continuous testing, continuous development and continuous delivery/deployment. DevOps enables teams to constantly upgrade and deliver.

    In 2016 a number of enterprises adopted DevOps. According to industry experts, 2017 will see more defining moments for DevOps. The idea of accepting DevOps is to consider a cultural shift and not just a technical shift. As digital transformation expert Jeff Sussna says about DevOps, “The key to following the continuous delivery path is to continually question your own assumptions about what’s possible.” Currently, DevOps is on the route of being questioned and defined at every level. It will result in bringing more value to the methodology.

    Every enterprise and its style of working differs. The way they would adopt DevOps will differ, too. But digital transformation and the need to be available for customers all the time with new updates is transforming the way enterprises adopt new practices and methodologies. Every enterprise has to evolve in a sustainable manner, so quality is at the core of every practice, including DevOps.

    Why is QA Integral to DevOps?

    In the DevOps scenario, QA integrates into the testing and development process and takes the collaborative approach. Quality is ensured throughout the testing and delivery cycle and both the testing and development teams are responsible for it. In the traditional pattern, quality creeps in toward the end, but with DevOps it comes in at every level.

    In terms of code development, the QA team pushes the new development code into production and DevOps facilitates to ensure a seamless deployment of infrastructure and tools. Thus, it helps in releasing a code on a consistent basis and the responsibility to ensure quality is not just limited to killing the bug; it is built in the overall application release.

    In the current scenario, hourly releases are required to meet the demands of the consumers and deal with competition. Web/mobile applications such as an ecommerce or financial services application need to be updated frequently. It is a key business requirement. DevOps helps to make this possible, but with QA involved, the quality of the release is ensured. Moreover, with DevOps, the reality check can happen in real time rather than taking the code offline and testing it.

    On the whole, the popularity of DevOps over the years has intertwined quality into every engineer’s responsibility, which is a positive sign. This helps in addressing the needs of the market and ensure that customers do not suffer. QA further enables and directs teams to check the application for performance and quality before it is delivered.

    The role of QA in the current software development process is changing, where the customer comes into focus and the needs of the customer are brought into perspective while delivering the application. Releasing new and functional builds at any point through the application’s life cycle can become a disaster if quality is not ensured constantly. Automation of tests plays a critical role to guarantee quality and bring speed.

    Building a QA strategy with the necessary best practices is essential for organizations that are embracing the DevOps movement. This will help in delivering effective software development and operations to achieve user experience. DevOps practices are adopted to improve the regularity of the releases and reduce the defects. It is a methodology that boosts rapid, short, iterative development and deployment of applications.

    So, the idea of incorporating QA is to ensure continuous improvement in the delivery process. DevOps helps digital enterprises to meet the needs of the competition and enable overall business growth. As a result, enterprises are able to streamline and pace up the delivery of applications with changes and innovation.

    Best Practices for Incorporating QA in DevOps

    The key aspect to consider while merging QA with DevOps is that the QA and testing teams must team up with the technical teams. In this way QA becomes an enabler for ensuring quality in the software development cycle. In addition, it is important to define tangible aspects of the quality that are expected from the project/application. This helps in achieving results within stipulated time frames.

    Putting down the quality metrics is absolutely critical in DevOps, which helps teams to identify bugs early in the development cycle. Defining the metrics also enables teams to integrate them within the organizational culture at various levels. In this way, quality is ensured at every level and development teams also get enabled.

    The idea of DevOps is to make the developer think in line with testers and constantly check the code for potential errors. On the other hand, a tester who holds the responsibility to test the application/software with various testing tools and methodologies also needs to fix the code. In this way, enterprises with an inadequate QA or testing strategy cannot ensure continuous testing and continuous delivery.

    In Conclusion

    The application scenario is changing and it is getting very complex. DevOps can speed up the development, but what can you expect without a robust QA strategy? QA is absolutely integral to the DevOps process. Continuous development and delivery is impossible without a comprehensive QA/testing strategy.

    This blog has been drafted in collaboration with Cigniti Technologies, an independent software testing company. Click here to read more blogs on DevOps.

    About the Author / Komal Lopez

    Komal Lopez works with Cigniti Technologies and is instrumental in helping enterprises take better decisions related to Quality Assurance products, tools and services by leveraging research and content. She specializes in writing about technology trends, testing trends and has been in the Marketing and Communications industry for over a decade. Connect with her at komal.lopez@cigniti.com

  • How To Fully Automate CI/CD – Even Secrets

    How To Fully Automate CI/CD – Even Secrets

    Imagine a world where your continuous integration/continuous deployment environment is 100 percent automated, including the passing of credentials.

    Capital One, a leading U.S. bank, has achieved this, and Andrey Utis, director of Software Engineering at Capital One, outlined how the company achieved this in his talk, “Application Secret Management with KMS,” at the All Day DevOps conference.

    iStock-683716072-1.jpg

    Andrey started by stating what we all know: Security needs to be at the forefront. Additionally, a message we heard over and over at the All Day DevOps conference is to automate where you can automate. That combination can be daunting and seemingly impossible—how can we automate security when someone needs to enter credentials and we can’t store credentials where everyone can get to them? After all, credentials provide access to databases that often contain personally identifiable information and other protected information. A breach could devastate companies and people, whether intentional or accidental.

    Andrey’s team uses Amazon Web Services (AWS), so they give instances IAM roles that allow them access to other AWS resources. What is the key (pun intended)? On an AWS EC2 instance, there is a magic IP address to which you can make an HTTP call and it will return temporary AWS keys. Those keys then make the API calls to different database services.

    Here is the solution: AWS KMS is encryption as a service. KMS Context allows you to add “salt” to the encryption. You can only decrypt with the same “salt.” KMS Key Policies restricts which IAM role can decrypt with the key/salt. Master KMS keys can be used only to decrypt their own keys.

    Below is a code sample of a KMS policy:

    Picture9.png

    This allows multiple applications to be on one AWS account while limiting access of developers to the applications they are authorized for.

    Here is the actual protocol:

    1. Create a KMS master key (can be shared by multiple apps)
    2. Create an IAM role for your server
    3. Add KMS key policy that allows decrypt to your IAM role for a specific context
    4. Encrypt your secret with the key and context and store the value in GitHub
    5. At deployment or runtime, invoke KMS API to decrypt

    There is a broader issue with IAM roles because credentials are generated by calling the “magic” metadata IP address 169.254.169.254. Developers in production, even with “read only” access to the instance, could call the KMS API to decrypt the secret. Developers should not be able to generate production IAM credentials at all, so you block the magic IP address with this code:

    Picture10.png

    To automate it and make it reusable, the team created a Chef cookbook, which they call a “briefcase,” to abstract decryption of secrets. They also have the iptables cookbook to block the metadata IP address for all except a whitelist of user groups, such as root and any application-specific group that makes AWS API calls (see above code).

    Andrey mentioned that Vault may have a viable solution soon. It uses signed EC2 Identity Document to verify the caller. The current downsides are that it only supports authorization by AMI ID (but should support more soon) and secrets are not source-controlled/versioned. As Andrey noted, that is not ideal for “configuration secrets” such as database passwords.

    In a follow-up question, Andrey was asked, “What were the lessons that were most critical to learn?” He answered, “This is a new field. Most companies either don’t fully automate or don’t fully secure the entire pipeline. So, there is little information out there.”

    Well, there is some more information in Andrey’s complete talk, which you can watch online here. If you missed any of the other 30-minute long presentations from All Day DevOps, they are easy to find and available free of charge here.

    Finally, be sure to register you and the rest of your team for the 2017 All Day DevOps conference here. This year’s event will offer 96 practitioner-led sessions (no vendor pitches allowed).  It’s all free and online on Oct. 24.

    — Derek E. Weeks