Tag: rundeck

  • PagerDuty to Acquire Rundeck to Expand DevOps Automation Capabilities Across Digital Operations

    Enterprise-scale runbook automation and auto remediation will improve productivity for teams on the digital frontline, enabling them to resolve incidents faster, reduce costs and protect customer experience

    (Graphic: Business Wire)

    (Graphic: Business Wire)

    SAN FRANCISCO–(BUSINESS WIRE)–PagerDuty, Inc. (NYSE: PD), a global leader in digital operations management, today announced that it has entered into a definitive agreement to acquire Rundeck, a leading provider of DevOps automation for enterprise. The acquisition strengthens PagerDuty’s leading incident response offering with intelligent machine automation including auto-remediation and self-healing that bring increased productivity to resolution and recovery efforts.

    “Rundeck was designed to solve difficult, time sensitive challenges for DevOps teams leveraging automation for hybrid infrastructure. With PagerDuty we can accelerate our growth in Enterprise DevOps and IT focused incident response, while continuing to innovate our offering and expand our open source community”

    Tweet this

    Rundeck gives front line incident response teams, including engineering, IT, customer service and security, a self-service way to run automated machine-centric workflows – or runbooks – that prevent, diagnose and resolve incidents at the push of a button, without needing to escalate to an expert. Customers have experienced up to 50% reduction in incident response time using Rundeck’s automated runbooks, greatly improving team productivity and reducing poor customer experience.

    Rundeck’s automation solutions have achieved strong product to market fit with large enterprises, serving over 150 enterprise and mid market companies, who are seeking to improve the availability of their digital offerings. Rundeck’s land and expand go-to-market motion starts with DevOps and is founded on an active open source community of over 60,000 users.

    “Rundeck’s impressive track record in providing scaled automation for DevOps teams in the world’s largest companies, along with their easy to use, practitioner focused offering make them a highly complementary extension for PagerDuty,” said Jennifer Tejada, CEO of PagerDuty. “Together we can now automate unpredictable, emergent work for people and machines, from detection and diagnosis to recovery, remediation and learning, providing the most robust real-time incident response platform in the market. Our 13,300 customers trust us to ensure their digital services remain always on, and now we are ensuring they can do this faster, and more efficiently.”

    “Rundeck was designed to solve difficult, time sensitive challenges for DevOps teams leveraging automation for hybrid infrastructure. With PagerDuty we can accelerate our growth in Enterprise DevOps and IT focused incident response, while continuing to innovate our offering and expand our open source community,” said Alex Honor, CEO and co-founder of Rundeck. “Like us, PagerDuty is steeped in DevOps and digital transformation, with unparalleled leadership, trusted by a large, loyal community of users in large enterprise companies and early innovators. We look forward to bringing the power of the most holistic offering in incident response and digital operations to all our customers.”

    “Bad things happen, so infrastructure and operations professionals need to prepare automated operational responses; correlate performance analytics to predict and avoid incidents; support preventative assessments, drills/game days, and chaos engineering; and use postmortems to improve systems,’ notes Forrester Principal Analyst, Charles Betz. “PagerDuty, a vendor focused on automating incident response, brings presenters to its regional summits to talk about incident command so practitioners can learn and adopt best practices.”1

    Under the terms of the definitive agreement, PagerDuty will acquire Rundeck for a total purchase price of approximately $100 million, subject to adjustment, to be paid approximately 60% in cash and 40% in PagerDuty common stock. The acquisition is subject to certain customary closing conditions and is expected to close in October 2020.

    Conference Call Information:

    PagerDuty will host a conference call and live webcast to discuss the Rundeck announcement with analysts and investors at 1:15 p.m. Pacific Time on September 21, 2020. This conference call will be accessible from the PagerDuty investor relations website at investor.pagerduty.com.

    Safe Harbor Statement

    This press release contains “forward-looking statements” that are based on PagerDuty’s beliefs and assumptions and on information currently available to PagerDuty as of the date of this press release. Forward-looking statements include, but are not limited to, statements regarding the anticipated timing of the consummation of the acquisition (if at all), the satisfaction of closing conditions and regulatory approvals, the expected benefits of the acquisition, the ability to successfully integrate Rundeck into PagerDuty’s business, the impact of the acquisition on PagerDuty’s products and services, the capabilities of Rundeck’s products and services, and the expected dilution to PagerDuty’s stockholders. There are a significant number of known and unknown risks, uncertainties, and other factors that could cause PagerDuty’s actual results, performance, or achievements to be materially different from those expressed or implied by the forward-looking statements, including: difficulties encountered in closing and integrating the merged business, technologies, personnel and operations; costs related to the acquisition; market acceptance of the acquisition and resulting products and services; PagerDuty’s inability to realize value from its significant investments in its business, including product and service innovations; and general market, political, economic and business conditions.

    Additional information on potential factors that could affect PagerDuty’s financial results is included in the company’s Quarterly Report on Form 10-Q for the quarter ended July 31, 2020, which is on file with the U.S. Securities and Exchange Commission and PagerDuty’s other filings with the SEC. Except as required by law, PagerDuty undertakes no intention or obligation to update or revise any forward-looking statements or to update the reasons actual results could differ materially from those anticipated in the forward-looking statements, whether as a result of new information, future events or otherwise.

    For more information, visit www.pagerduty.com

    Learn more about PagerDuty’s most recent platform innovation, integrations, customer use cases, partnerships and thought leadership at PagerDuty Summit20 from September 21-24, 2020.

    About PagerDuty

    PagerDuty, Inc. (NYSE:PD) is a leader in digital operations management. In an always-on world, organizations of all sizes trust PagerDuty to help them deliver a perfect digital experience to their customers, every time. Teams use PagerDuty to identify issues and opportunities in real time and bring together the right people to fix problems faster and prevent them in the future. Notable customers include GE, Cisco, Genentech, Electronic Arts, Cox Automotive, Netflix, Shopify, Zoom, DoorDash, lululemon and more. To learn more and try PagerDuty for free, visit www.pagerduty.com. Follow our blog and connect with us on Twitter, LinkedIn, YouTube and Facebook.

    1The Changing Landscape of IT Incident and Crisis Management, Forrester Research, Inc., February 20, 2018.

  • DevOps Chat: Rundeck, Putting the Ops in DevOps, with Damon Edwards

    DevOps Chat: Rundeck, Putting the Ops in DevOps, with Damon Edwards

    In this DevOps Chat we speak with Damon Edwards, co-founder and chief product officer of Rundeck. What I like to say about them is, Rundeck putting the Ops in DevOps. But Damon needs no introduction to those who have any familiarity with DevOps. He is one of the early leaders of the DevOps community and continues to evangelize DevOps around the world. In fact, he is speaking at our DevSecOps event at RSA APJ in Singapore July 25, which you can find here.

    It has always been a hope of me personally to have Damon as a guest on DevOps Chat. Besides being a great interview, Damon and Rundeck are a great story of an open-source project growing organically, spawning a great commercial business. This is harder than you think and Damon tells us a bit about it in this interview.

    As usual, the streaming audio is immediately below and the transcript is below that. Enjoy!

    Audio

    Transcript

    Alan Shimel: Hey, everyone, this is Alan Shimel, editor in chief of devopsy.kinsta.cloud, and we’re here for another DevOps Chat. We’ve got a really, really special guest for this DevOps Chat, it’s one of my heroes in the DevOps movement from when I first started researching what DevOps was and it’s about. I’m thrilled to have him appear on our DevOps Chat today, none other than Damon Edwards. Damon, welcome.

    Damon Edwards: Thank you, Alan. Thank you for having me and thank you for those kind words, I’m excited to be here.

    Shimel: Thank you. So Damon, I’m going to guess a lot of people in our audience have heard of Damon Edwards, instrumental in putting together DevOps Days, especially DevOps Days at Silicon Valley kind of drove that for a number of years. You had several different companies, both open sources and consulting, and probably maybe even more than all that, Damon, you have appeared—I don’t even know if you keep count any more—at how many different DevOps Days events or other DevOps related events where you’ve spoken at. I mean, do you know? Have you kept track?

    Edwards: I don’t know, [Laughs]. I know it’s less than John Willis, because he reminds me of that on our podcast, but DevOps Café, I’m going to give a cross plug for that. We just had you on, Alan, as a guest, and you were fantastic. Yeah, I kind of lucked into just sort of right place/right time. We were really interested in how do you move fast—we called it Dev2Ops ourselves internally, our consulting business we used to call DTO Solutions that we have. And it was really all about how do you help organizations move faster? For a long time, scale was the problem. Hey, how do you deploy, hook into a thousand servers, right? A thousand servers used to be a big deal. And then it suddenly came to be that, hey, we can deploy anything to anything, that’s not really—deployment scale is not really the problem, speed is the problem. That organizations can’t move fast enough, that the pipeline is jammed, that the organization is kind of stuck in cement. And the organizations that are able to free themselves and decouple themselves and move in this new and fast way had a strategic advantage, were really kicking the butts of the big traditional companies that were kind of locked in place.

    So we really started asking ourselves, why is that? Our customers started asking us, why is that? And that really got us into lean and all those things. And at the same time, John Allspaw gave his big talk at Velocity, I saw that, you know, fell in with John Willis and Thatcher Dubois and the rest of the crowd and the DevOps movement was born. And we kind of became known as the folks who are great at taking the design patterns that you see on stage at the big high flyers and translating them to the big enterprises out there. So that was DTO Solutions.

    Then along the way we had a side project called Rundeck that was an open source tool, I’ll get to that in a second, but it really just took off. And after awhile, enough people called us about Rundeck, we really got excited about it, and there are two reasons why we decided to jump with both feet into Rundeck. The first being that I feel like operations is a DevOps frontier that’s not truly explored yet. I think in a lot of big organizations the dev side, they’ve got dev talking better to QA, they’ve got the build pipeline going, the little dev test cycles are going a lot faster. Maybe deployment’s happening a lot faster. But these DevOps principles haven’t really entered into full operations mode yet. A lot of big operations organizations just haven’t been able to transform themselves, and we saw DevOps, especially we saw Rundeck as a tool being used just to do that. So as a way to, number one, I think we were always product guys at heart, we wanted to see where we could take it. But number two, I really feel it’s helping push the DevOps movement to a new place.

    Shimel: Yeah, I agree with you. And Damon, just again, adding a little color, you actually have a video up somewhere, it’s kind of a brief history of DevOps. For any of our listeners who really want to know kind of how DevOps got started, Damon has a great video on that that I’d highly recommend. But let’s jump into Rundeck. Damon, as you mentioned, it was a side project to what you were doing with DTO. It was open source, wasn’t it?

    Edwards: Yeah, Rundeck’s core is it’s an open source product, an open source community. I Think last time we did some sort of estimate. There’s around like 25,000 different open source users out there. So it kind of became this silent—for us, we woke up one day and said, wow, a lot of people are really using Rundeck, and they were calling us saying, “Hey, we really want to do more with Rundeck,” and that led to the formation of what’s now Rundeck Inc.

    And if you want to talk about what Rundeck actually is, fundamentally, from a technology perspective, it’s an orchestration and scheduling platform. Right now a lot of orchestration happens within different silos, so there’s container orchestration, there’s configuration management orchestration, all kinds of different things like that. There’s enterprise job schedulers. But we noticed that as folks are transforming their operations organizations there was this layer that needs to span all of those different silos of tools, all the different islands of automation that they have, all the different systems that they have. They want to create procedures, operational procedures that span all of those different silos. And they want a place where they can define those procedures using whatever scripting language that they want. They want to be able to define the work flow, they want to be able to set security around it, that’s huge, because what a lot of people are doing in those procedures is not just helping their operations teams be more efficient and effective but they’re starting to open up more of a self-service model. Self-service operations is the big design pattern that I think is really driving us forward.

    Which is, all these people that need to do operational things throughout my organization, and as I start to transform my organization, how do I safely let them do things that could be deployment but it’s often remediation activity, diagnosing things, just doing operational things that I need to do, how do I let an ops team define those and be in charge of those and say who can do what, when and where, but allow other people to participate? And by participate, I mean two things. One is give them the button or the API to hit themselves to do the operational things that they need to get done, but also give them a mechanism to which those different teams, often in different business units, can define those operational procedures for the things they’re creating themselves, then have a standard mechanism to hand it off to operations where ops and security can vet them and then that ops and security vetting process can then turn around and give access to those other teams. So it’s like how do we let the operations decouple, let more people participate in operations, and do it all in a safe, sane and secure and audited and logged way? That’s where Rundeck shines.

    Shimel: Got it. I mean, Damon, you opened a bunch of different things in here I want you to kind of flesh out a bit. So number one is this whole idea of ops, right? My personal opinion is for too long with DevOps there was a very developer-centric focus of DevOps, and really, at the end of the day, ops is where it’s at for a lot of different reasons, especially in the world we’re living in. And you’re clearly working in there. But we’re not looking to recreate silos, right? There’s a place for the development team, right? They interact within Rundeck, if you will, with the ops folks, right? This isn’t for ops only kind of thing.

    Edwards: Right, right. If you look at organizations today, again it kind of goes back to our statement before about how scale is no longer the problem but organizational speed was the problem that caused the DevOps movement. And if you look at what’s going on with operations it’s a very similar thing. It’s not scale of the infrastructure as much as it’s complexity and scale of the organization that causes the problems. So in a lot of large enterprises, most every large enterprise, you have this central place called operations. Just kind of put the teams out of the way at this point, just think about the action of running all these things. You have all these different streams of activity that come into this central place, and that has to all hang together to form this enterprise. That is operations. That is the business. That’s where all the money is made. All these streams of development activity that come into that, essentially they’re part suppliers, so the next version, the next cogs in the machine that has to run, but it all has to hang together or we aren’t making any money, unless you actually sell software, which very few companies do these days, your running the service is the business. So operations is very important. But in that same vein, that operations, because things are so tightly coupled and it all falls to this one often central team, that becomes a huge bottleneck and a big problem, and it really ends up stunting or blocking the DevOps. That decoupling transformation that these organizations want to go to, they get blocked by the sort of calcification of the classic operations domain.

    And so we see these high flying organizations, what they’re doing now, the high performers, what they’re doing is a frame of how to uncap and decouple all of that operational activity. And in doing so, they’ve kind of realized there are three key pieces to what you’re working with. There’s the ability to—who defines the procedures? Who defines the procedures for the things that they’re going to be running. The second is, who can push the button for those procedures, button meaning through a GUI or through an API. And then third, who has the secondary or the operational, the management control of that action?

    Traditionally that all kind of fell to one group or one team and they’re grossly outnumbered by the rest of the organization and that became the big ops bottleneck. Their whole mindset was all about let’s protect ourselves from being overrun. Let’s protect ourselves because people are pushing things at us that they don’t really know how it’s all going to hang together, they don’t have that responsibility for defining things. So ops was in a very, very tough position and doing their best. We see these organizations doing a bit of a Tom Sawyer paint my fence or a jujitsu move by saying, hey, let’s redefine who does what where. Let’s allow that definition. Let’s allow those teams. They created these things, let’s allow them to participate in the definition of these operational procedures. And then let’s also, once it comes back to us, let’s let them push that button as well—for some things, not everything—maybe just certain non-destructive things, or maybe certain teams can deploy but certain teams we don’t want to deploy because it’s a different type of system or different security requirement. And you can move those things around and ultimately you can keep that third piece, the management and the compliance and the audit security control, with that traditional operations organization, or traditional operations skillset or management skillset who wants to control those things.

    So in this self-service design pattern it’s about dividing those three things up—the definition of the automation, the execution of the automation, and the management security compliance control over it—and allowing those to be moved to the parts of the organization where it makes the most sense for the flow of the business, speed, taking advantage of the labor that you have. It’s that mindset that these operations organizations are using to transform their organizations. When we saw that happening we were like, wow, that’s something really cool, and we jumped in with Rundeck Inc. with both feet.

    Shimel: Absolutely. Let’s talk a little bit about Rundeck Inc. as you called it. So Damon, was it about nine months, maybe more, maybe a year ago now, you and your—I’m drawing a blank, I apologize –

    Edwards: Alex Honor.

    Shimel: Right, Alex Honor, you and Alex have been doing Rundeck for as long as I know you. But about nine months or a year ago you brought in sort of a professional CEO, someone who I’ve known for years, a tremendous person based out in the Valley, Stephanie Fohn, as your CEO. And it really went from being Rundeck’s a side project of DTO Solutions, to Rundeck Inc.

    Edwards: Yeah, we jumped in full feet. We luckily landed Stephanie, it was a big coup for us. I think she described our situation “an embarrassment of riches,” and I wasn’t sure if she was emphasizing on the embarrass or the riches part. We started what’s now Rundeck Inc. as kind of an experiment to see, hey, what’s going to happen? It was Alex, myself, Greg Schuler, as lead developer of Rundeck. Alex is the founder of the Rundeck project. We had a small team going, we had some engineers scattered about, and things were working. But next thing we looked up and we had a whole bunch of large enterprises that were paying us for the early version of Rundeck Pro, which is sort of an enterprise enhanced version. There’s all kinds of features and extra plugins that really matter if you’re doing this stuff at scale. The open source is really defined for if you’re just a team and you need to define buttons and push buttons, then the open source is fully featured and there for you. Rundeck Pro is really about kind of adding things around and on top of it that the boss cares about, that big organizations care about.

    So we looked up and things were doing well. Stephanie came in and said, hey, we’ve really got something here and I want to be a part of it. So it was a big score for us. She is very well known on the security side of the house, a number of great successes there. She’s kind of found that operations and security is a very similar mindset, very similar set of people, and she’s really taken to it. Yeah, it’s been great, she’s brought in even more professional people.

    Shimel: It’s grown up to a company.

    Edwards: Yeah, we’ve got over 50 enterprise customers so far and we’re growing fast, so it’s a pretty cool, pretty fun ride.

    Shimel: Damon, your role now in Rundeck Inc., if we can call it that, is you are CTO and obviously cofounder, but CTO and evangelist, but your hands are still getting dirty talking to customers and helping.

    Edwards: Yeah, I’d say that’s mostly what I’m doing. Actually, Alex is the CTO, I’m the, I guess, Chief Product Officer, or head of product, these names that, you know, it’s a typical start-up, right? I focus a lot on the evangelism, a lot on the product side, a lot on helping customers figure out, A, well, figure out what they’re doing with Rundeck, and then also, B, help them figure out what they want out of Rundeck and driving that forward. So I’m still intimately involved with the DevOps community and out on the road doing that because I believe this is the next frontier, this is where you have to go. Straighten out your deployment pipeline and getting automated testing in there is a great thing, but if you’re any organization of any scale, this is the next major issue. You need to transfer operations otherwise you’re not really meeting the full DevOps vision. And the companies that are, have a distinct advantage over you.

    Shimel: You know, Damon, to me, DevOps is still very much like right out of the goal, where you solve one bottleneck only to find the next bottleneck. And that’s what we do is we keep knocking them down. And as you knock one down, another one further down the road shows itself. And I think that’s what we’re seeing in DevOps, is as we knock down bottlenecks in the CICD arena and configuration management arena, ops becomes where the next series of bottlenecks are, and that’s what obviously Rundeck is hitting in there.

    I wanted to talk a little bit, Damon, about a particular subset of this, and that’s DevSecOps. I think you almost inadvertently maybe backed in to the wonderful world of as we call it now cyber security or InfoSec or whatever you want to call it. Inherent in that ops mission is the idea of deploying securely and keeping a secure infrastructure up and securing your apps. And this is something that Rundeck is spending more and more time, more and more resources, more and more of the focus that I’ve seen anyway is involved around this DevSecOps mission. Comment on that?

    Edwards: Yeah, I’d say it’s sort of the intersection of security and operations. Really security, operations and governance. That if you think about what’s going on now—especially, well, two big trends. There’s the digital transformation trend, which is really just saying all of my systems need to be interconnected now for a consistent customer experience, that’s something the business wants. And then there’s all these new server lists, containers, the new API world where everything’s an API. So now we’ve got to define these procedures that span all of these different systems that used to be stand-alone stovepipes. And, everything’s turning into an API, so I’ve got to have a way to secure that, I’ve got to have a way to figure out who’s who and who can do what. So we kind of fell into that in terms of people use Rundeck as that tool, to say, hey, I’ve got my Chef automation over there, I’ve got some Kubernetes stuff over there, I’ve got some legacy stuff over there, I’ve just got a bunch of scripts I need to run over there, I’ve got these different Amazon services I need to hit, Akamai and these kinds of things. So they’ve got all these different things, buttons they need to push, and they each have their own logins, each of their own sort of way of managed security policies.

    So they look at Rundeck and say, hey, this is a place where we can enforce policy, or enforce this authentication and enforce the security controls. By defining these Rundeck jobs, it becomes a nice abstraction for that to say they can just call all this stuff, it’s real easy to string it together, and then I can define who can do what and where, and I’ve got the full access. In a lot of ways we call it like the key card and the camera. You can define who can do what through Rundeck, sort of the key card in your office building. But then there’s also a bunch of cameras, right? Not that we don’t trust you but we have to prove that you didn’t do anything bad with your keycard. So the kind of keycard and camera metaphor, we talk about that internally, but we should probably talk about that more externally. It’s something that we see a lot of people implementing with Rundeck in this new world. It’s a way to satisfy the security and governance demands but also do the new ops stuff.

    Shimel: Damon, we’re a little overtime already but I wanted to just quickly mention that you’re going to be appearing at RSA APJ, which is the largest info sec or cyber sec related show in the Asia Pacific region. We’re putting on a full—we meaning Sonatype, DevOps Connect and a few others—are putting on, Rundeck is a sponsor, is putting on a DevSecOps day there, and you and John Willis are headlining, and for any of our listeners who are in Singapore or around Singapore, that’s on July 25. And on July 26 I think we’re going to try to do a mega meetup group, several meetups, different meetup groups within Singapore. Also at RSA APJ, and you and John will be presenting there as well.

    Edwards: Yes, we will. You guys do a very good event. We’ve done the one here in San Francisco, excited to do the one in Singapore. Singapore is actually a great DevOps community out there with all the banking and finance and whatnot. It’s good work being done there. And the event’s great. It’s not just for security, it’s really looking at the end-to-end pipeline I’d say. It’s more like how do you bring these enterprise concerns into the DevOps story. And I don’t want people to think it’s just, hey, if I’m not in security, I don’t need to go to this. I think it’s really good for anybody who works in the enterprise and wants to think about the nuts and bolts of how do we get this stuff done so we can move fast and be secure or sure in what we do.

    Shimel: Absolutely. And you mentioned RSA, Damon, actually I’ll have some news about that hopefully within the next week—RSA San Francisco next year is April in San Francisco and I think we’ll be doing a DevSecOps day there as well. I think you presented, we’ve done three or four of these, I think you’ve presented at all of them, actually.

    Edwards: I’m a supporter. I’m a supporter of the Alan Show.

    Shimel: Well, you know what, to me it’s about bringing these two tribes together. It was something that Gene Kim and Josh Corman tried to do and I’ve tried to help, and Mark Miller and the Sonatype folks. And Damon, you know what, it’s paying off because I spend a lot more time in security just because of my background, and I’m hearing from—I’m not getting the pushback I used to get from people about DevOps any more. They get it. Or they’re starting to get it. God hopes. Anyway, Damon Edwards, we’re way past our time, but that’s okay, it was worth every minute. Thank you for being this episode’s guest on DevOps Chat. We will see you in Singapore. You know what though? For people who want information about Rundeck, where can they get more information?

    Edwards: Rundeck.com. There’s a free trial of Pro. You can always use the open source. We’re on IRC, we’re available, so we’re around. Feel free to reach out to us and we love talking about this stuff.

    Shimel: Cool. Damon Edwards, co-founder, chief product officer of Rundeck Inc., and also DevOps evangelist extraordinaire. Thanks for appearing on DevOps Chat and we’ll speak to you soon.

    Edwards: Thank you, Alan.

    Shimel: All right. This is Alan Shimel for devopsy.kinsta.cloud, DevOps Chat, and we’ll see you soon on the next chat.

    — Alan Shimel

  • Embedding Ownership: A DevOps Best Practice

    Embedding Ownership: A DevOps Best Practice

    From where I sit in the DevOps community, there is often more focus on dev than on ops. Damon Edwards (@damonedwards) of SimplifyOps sought to change that with his talk, “Ops Happens: DevOps Beyond Deployment,” at the All Day DevOps conference.

    Damon dove right into the primary, systemic force behind most DevOps problems: silos. The product development process goes like this: Planning—>Dev—>Release—>Operate. The problem lies in the tendency of many enterprises to place similar functions together. Everyone ends up in a silo. Then walls build up between the silos. Eventually, people only know life in their silo, making handoffs even harder.

    We often find application knowledge and business intent are heavily emphasized on development side but light on the operations side. Likewise, operational knowledge is heavy on the ops side, but light on the development side. Furthermore, development has ownership but limited accountability, while operations has accountability but no ownership.

    While many enterprises are striving toward building cross-functional teams, the reality is that the transformations often stop short of truly integrating operations. The result?  Silos remain.

    So, one has to ask, why is this so hard?

    The reality is that enterprise operations are under tremendous pressure. One side is telling them to go faster and open it up, and the other side is telling them to be more secure and be more reliable. These are often seen as competing priorities.

    To solve this, enterprises need to “shift left” in the product development cycle operations activity as much as possible. They need to do as much as possible during development. For the deploy function, enterprises should be:

    1. Writing/running automated tests
    2. Writing/exercising deploy automation
    3. Running security scanning tools

    For the operate function, enterprises should be:

    1. Writing/exercising automated runbooks
    2. Writing/exercising monitoring/metrics
    3. Operational control (safely!)

    However, shifting operations to the left is much more difficult. How do you do it? Embed ownership.

    Let me repeat: embed ownership.

    First, those who build something define the procedures to fix it, and those who build something fix it when it breaks.

    That sounds simple, but raise questions:

    1. How do you safely and securely give out access?
    2. How do you enable the experts to contribute remediations?
    3. How do you give the experts visibility into operations?
    4. How do you do postmortems days/weeks/months later?

    Damon recommended four steps.

    Step 1: Establish a secure ops portal


    Step 2: Establish a software development life cycle (SDLC) for ops procedures

    Step 3: Connect with enterprise management systems

    Step 4: Make Compliance really happy

    Ticketmaster is a real-life of example of this working at a large, prominent scale. Ticketmaster calls its system, “Support at the Edge” and it involves:

    • Automated Ops procedures written/vetted by the delivery teams
    • Ops remained in full control of what can run and security policy
    • Empowered support teams with self-service ops tasks
    • Empowered developers with limited self-service operations
    • Combined with new incident response model

    Ticketmaster has seen transformative results. Before Support at the Edge, the average mean time to respond was 47 minutes. Now, support at the Edge has reduced that to just 3.8 minutes in addition to decreasing escalations 50 percent and overall support costs 55 percent. Ticketmaster has seen real results.

    Damon has more details in his talk, which you can watch online here. If you missed any of the other 30-minute long presentations from All Day DevOps, they are easy to find and available free of charge here.

    Finally, be sure to register you and the rest of your team for the 2017 All Day DevOps conference here. This year’s event will offer 96 practitioner-led sessions (no vendor pitches allowed).  It’s all free, online on Oct. 24.

    — Derek E. Weeks

  • Rolling Updates with Rundeck

    Rolling Updates with Rundeck

    CAUTION: If you haven’t used Rundeck yet, clear your calendar for next Valentine’s Day because you’re about to fall in love… How many times have you written a shell script that runs via cron, or a PowerShell script that lives in the Windows Task Scheduler? How many times have you struggled to remember which server you built that script on? Let’s take it one step further: when was the last time that job ran successfully? Using rundeck makes you click-your-heels happy!
    I’ve used multiple solutions to address this problem in the past: jump boxes, dedicated scripting boxes, puppet manifests, the list goes on- but each left me feeling like I could have done something better. Recently, I discovered Rundeck and have been jump-in-the-air-click-my-heels-happy with it.
    What is Rundeck?
    Rundeck is a web application server that allows you to define jobs and tasks. It’s that simple. Where it excels is in its ability to automate those jobs, hand those jobs off to other people to run (without giving them access to the servers), call those jobs via API, monitor the progress and status of those jobs, and log all of the above for auditing purposes.
    Some common use cases I’ve personally used Rundeck for include:
    • deploying EC2 instances in AWS
    • creating scheduled jobs to run on remote servers
    • implementing blue/green deployments from a centralized location
    • rolling updates to ensure only one server is being updated at a time
    I’m sure that gets your mind going with all kinds of ideas, so let’s not waste any more time and get right to
    Setting up Rundeck
    Rundeck is a java web application and as such, requires your host system to have java installed. Java 1.7 is the minimum, and either Open JDK or Sun/Oracle Java will work just fine. If you have a system that supports RPM or Debian packages, installation can’t get any easier as both package formats are available and not only handle the installation process, but shell tools and init.d scripts for startup and shutdown as well. Users of other Operating Systems can use the Launcher for running the app, but installation details are left up to you. Finally, Rundeck can also install as a war file for use in an external container like Tomcat. With the server up and running, it’s time to:
    Build our first project:
    Rundeck_create_projectWhen getting started with Rundeck, I was expecting a lot more configuration prior to building a project. I actually struggled a bit when I couldn’t find out where to plug-in my AWS details at a system level. Turns out, you can’t because it’s configured on a project-by-project basis. This actually works out well, allowing you to isolate which jobs can be run against which servers, preventing mishaps such as running a script intended for your public-facing web servers on your finance servers!
    Creating a new project is done by clicking the big, green “New Project” button on the main screen. On the “Create New Project” screen, filling out the Project Name and Description details are pretty straight forward. The Resource Model Source is where things really start happening. In Rundeck, the Resource Model Source is how you specify the list of servers for the project. In its simplest form, Rundeck receives the list of nodes in XML or YAML format and can receive this data from a variety of sources ranging from hand-crafted files, CMDB, hosted virtual machine service, or a number of available plugins. More information about the Resource Model Source can be found in the Rundeck documentation here. If you are using AWS, the Rundeck EC2 Nodes Plugin allows you to specify nodes (a.k.a. ec2 instance) by tags. Installation is as easy as installing the plugin from the github repo into the $RDECK_BASE/libext directory (/var/lib/rundeck/ if you installed from rpm on RedHat based systems).
    You will want to create a new set of access keys for Rundeck. This will allow you to control what permissions Rundeck has in AWS via IAM policies, and make it easier to rotate keys as the keys will only be used in one place. Using the “Filter Params” textbox allows you to filter the AWS nodes to match your criteria. For example, I have all of my node.js servers tagged in AWS with a tag named “app” and the value set to “nodejs”.
    EC2_tagging
    Mapping Params is another area worth noting. Use this field to adjust the mapping of the values returned by AWS to the fields used in Rundeck. For example, this is a mapping I use:
    hostname.selector=privateIpAddress,privateDnsName;environment.selector=tags/environment;username.default=ec2-user
    First I have the hostname.selector field set to “privateIpAddress,privateDnsName”. This allows me to use either the IP address or DNS name of the instance when selecting nodes. It also uses the same data for displaying which nodes a given project will run on (much more insightful than seeing a list of instance IDs!).
    Next I have an “environment.selector” set to “tags/environment”. As you can see in the screenshot above, I use a tag named “environment” and set values to “prod”, “staging”, and “dev”. This allows me to further filter jobs by selecting the environment from a selector at runtime to determine which environment the job will run on.
    Finally, the username is set to “ec2-user”. This tells Rundeck who it should log in as when opening an ssh session to the node.
    If you are using ssh as the default node executor and file copier, you’ll need to upload keys with appropriate permissions as well. With all of that done (it seems like a lot- but it packs a lot of power in those steps, and it only needs to be done once for the project), it is finally time to:
    Create A Job:
    Creating a job is much simpler than configuring the project. Most of the options on this screen are pretty straightforward. I’ve had so much success with the “Script” step, I find it is my go-to favorite. It allows you to define a script and run it directly on the host. For example, I need to restart node.js on all my servers but don’t want to take them all down at the same time. To accomplish this, I would use a Bash script similar to the following:
    (A couple of notes here:
    – I’m using pm2 to manage the node process, so that is the application the script targets
    – My node.js servers have a /ping endpoint that returns OK when the server is up and running)
    # Shutdown the local node
    echo “Shutting down node.js on $(hostname)”
    /usr/bin/pm2 kill
    sleep 15
    # Start node.js
    echo “Starting node.js on $(hostname)”
    /usr/bin/pm2 start /etc/pm2/ecosystem.json
    sleep 30
    # Wait for the server to come online
    echo “Waiting for a positive response”
    while true; do
        STATUS=$(curl -s http://localhost:3000/ping)
        echo “$(date): $STATUS”
        if [ “$STATUS” = “OK” ]; then
            break
        fi
        sleep 10
    done
    echo “Done with $(hostname)”
    Boom. This script will now sequentially roll through all of my servers tagged “nodejs” in AWS, terminate the node process (via pm2), and wait for the server to come back online before moving to the next server. If the node doesn’t come back online, the Rundeck script for that node will not continue, preventing an unidentified error from taking all of my node.js servers offline! The key to this is ensuring that the Workflow Strategy is “Node-oriented”. This tells Rundeck to perform all steps on a given node before moving on, fully allowing the job to fail if something goes wrong. It’s also noteworthy to point out that Rundeck can email the results to you as well as logging all job executions. With our job in place and ready for action, it’s time to do what all DevOps people do best:
    Sit back and troll hacker news:
    hacker_news_logoSeriously. Rundeck can be an integral part of your DevOps toolkit. Using it for rolling updates, patching, and deployments are just the start. Imagine taking it a step further so that it integrates into your monitoring system and executes jobs as a result of alert conditions, or use it as a stand-in replacement for a complete DIY provisioning system for AWS OpsWorks. With a little work upfront, you may never have to minimize hacker news again!