devopsy.kinsta.cloud

Tag: pipeline

  • Creatures of Habit, Snowflakes and Integration Tax

    Creatures of Habit, Snowflakes and Integration Tax

    Donovan Brown of Microsoft, in conversation with Matt Hilbert of Redgate Software, discusses database DevOps’ biggest challenges

    When DevOps was first talked about in Flickr’s seminal “10 deploys per day” Velocity presentation in 2009, it was regarded by some as strange and alien to corporate culture. It was the antithesis to the accepted way of doing things, a threat to established and proven processes—it was downright dangerous.

    Times have changed. Unicorns such as Uber, Airbnb, Spotify and GitHub couldn’t live without it. Microsoft has embraced it and actively promotes it. It’s become the key to gaining competitive advantage in a technology marketplace that is moving faster and changing quicker than it ever has before.

    But what about the database? Why are practices such as version control, continuous integration and automated deployment being introduced to application development but left on the shelf when it comes to the database? The 2017 “State of Database DevOps survey,” for example, found that continuous integration was in place for 39 percent of applications, but fell to 20 percent for databases. What’s going on?

    In search of some answers, we spoke to Donovan Brown, Principal DevOps Manager at Microsoft and DevOps advocate. Also known as The Man in the Black Shirt, his unofficial tagline is #RubDevOpsOnIt. He lives DevOps. In fact, you can’t stop him talking about it and enthusing about its advantages.

    He describes DevOps as “the union of people, process, and products to enable continuous delivery of value to our end users.” The most important of those three, according to Donovan, is people. It’s also, conversely, the most difficult to fix.

    ‘People are Creatures of Habit.’

    “If you’ve been doing something the same way for 30 years,” he said, “why would you fix it? We all resist change, even if change is for the better, and DevOps requires a lot of change.”

    A large part of that needs to come from the Ops side of the equation because, while processes including version control and continuous integration have become second nature to developers, Ops teams traditionally don’t do them on a daily basis. In fact, Ops are often seen as the opponents of change.

    Donovan does, however, hold his hand up as being partly responsible for the division. “Historically,” he said, “they’ve had to become the gatekeeper because developers like me 20 years ago were wreaking havoc on production servers, running on there with admin credentials. Their job in my opinion has never been to deploy software for us. Their job is to protect and manage the infrastructure, to make sure it runs as efficiently as it can. That history, though, stays with us because there is so much regulation and red tape to get code from the fingers of developers to the hands of users.”

    He sees DevOps as the solution to the problem. “What I’m hoping is that DevOps will help automate what we’re forcing them to do, which was never their job, and rebuild the trust between Dev and Ops. We really need to help Ops people realize that we’re not trying to replace them, we’re trying to empower and enable them to allow us to continuously deliver value.”

    It’s not down to Ops teams on their own, however. C-level executives also have a part to play because they often favor a traditional approach to software development, with targets and long term strategies standing in the way of DevOps.

    “What I’ve noticed in a lot of companies that are struggling to adopt DevOps is that they’re not getting the backing from the executive level,” Donovan saids. “You have this waterfall mentality from the top that still wants GANT charts and milestones, and things that just do not line up with agile thinking at all. I’ve had the best success implementing DevOps in organizations where the C-level executive got it. When you have a grassroots team who get it, but no one else does, they usually get snuffed out by tradition.”

    The database is not separate from Donovan’s goal of people, process and products coming together. Prior to joining Microsoft, he was a Process Consultant and Scrum Master, helping a range of companies introduce DevOps to their development practices. He made sure the database was part of that.

    Then, as now, including the database was a challenge, but Donovan worked around it, using the technologies available to allow customers to treat the database just like every other element in their DevOps solution.

    He does, however, admit it was hard work. “It’s actually the most difficult part of the pipeline to automate because of the risk of failure,” he said. “You can easily roll back a front end; you can’t do the same for your database because any transactions that have occurred during the deployment could potentially be lost. So, it’s a crucial, integral part of your DevOps pipeline and I challenge any company that claims to be doing DevOps that only does it for the front-end but their back-end is still done manually. To me, they’re faking it.”

    ‘Infrastructure as code should not be a special snowflake. It’s just part of your solution.’

    The key is to remember that the code written to make changes to a database is just that: code. Traditionally, it has been managed outside of and separate to application code, with database changes often managed on-the-fly, at the last minute. Consequently, deployments have become worrying at best, with teams on hand for the expected firefight against downtime.

    Include it in version control, however—and continuous integration and automated deployments—and the database can be developed in parallel with applications. Problems are then picked up earlier, deployments become predictable and reliable, downtime and rollbacks are a thing of the past.

    What has made this even more appealing is the tooling now available—software that in Donovan’s days as a consultant was only just emerging. As he admitted: “The tooling had to mature to allow us to get over a lot of the hurdles that are unique to database deployments so that we can have everyone incorporated into the DevOps pipeline, not just the unicorns. Everyone should have the power available to them to be able to take their databases and incorporate them as part of their DevOps pipeline. Until you do, you still have that being the bottleneck.”

    There is a price to pay, however, if by introducing tools to bring DevOps to the database, companies get bogged down by what Donovan calls “the vendor explosion.”

    ‘Every vendor you add to your pipeline also adds integration tax.’

    Integration tax is the cost to companies, in terms of training and introducing unfamiliar ways of working, to use a new piece of software that resolves a piece of the DevOps puzzle. If that cost is too high by, for example, moving people out of the development environment they’re comfortable with, it betrays the ideals of DevOps. One onerous way of working is simply replaced by another.

    Donovan is keen to remove that tax by encouraging companies to work with as few vendors as possible—and preferably those vendors who have already paid the integration tax in advance. By, for example, introducing tools that work inside or alongside existing software, thereby reducing the time it takes to become familiar with them, and making the most of the current infrastructure rather than adding to it.

    Otherwise, Donovan concluded, “You’re paying tax for that integration. When we at Microsoft have partners who create extensions for our products, it’s the vendor and Microsoft who pay the tax in advance.”

    Finally, I asked Donovan what the biggest challenge was for companies who adopt DevOps for the database. His answer was the most telling.

    “If you’re doing DevOps differently for your database and the rest of your system, to me there’s something wrong there. When I think of DevOps, the database is part of that. I don’t need a separate conversation about it.”

    About the Author/Matt Hilbert

    Matt Hilbert is a technology writer at Redgate Software with 20 years’ experience working for lots of the world’s biggest tech companies – and many of the smallest. He has a particular fascination for emerging technologies and is on a continuing mission to decompile, untangle, and explain techspeak to a wider audience, and excite people about the endless possibilities technology offers.

     

  • How To Fully Automate CI/CD – Even Secrets

    How To Fully Automate CI/CD – Even Secrets

    Imagine a world where your continuous integration/continuous deployment environment is 100 percent automated, including the passing of credentials.

    Capital One, a leading U.S. bank, has achieved this, and Andrey Utis, director of Software Engineering at Capital One, outlined how the company achieved this in his talk, “Application Secret Management with KMS,” at the All Day DevOps conference.

    iStock-683716072-1.jpg

    Andrey started by stating what we all know: Security needs to be at the forefront. Additionally, a message we heard over and over at the All Day DevOps conference is to automate where you can automate. That combination can be daunting and seemingly impossible—how can we automate security when someone needs to enter credentials and we can’t store credentials where everyone can get to them? After all, credentials provide access to databases that often contain personally identifiable information and other protected information. A breach could devastate companies and people, whether intentional or accidental.

    Andrey’s team uses Amazon Web Services (AWS), so they give instances IAM roles that allow them access to other AWS resources. What is the key (pun intended)? On an AWS EC2 instance, there is a magic IP address to which you can make an HTTP call and it will return temporary AWS keys. Those keys then make the API calls to different database services.

    Here is the solution: AWS KMS is encryption as a service. KMS Context allows you to add “salt” to the encryption. You can only decrypt with the same “salt.” KMS Key Policies restricts which IAM role can decrypt with the key/salt. Master KMS keys can be used only to decrypt their own keys.

    Below is a code sample of a KMS policy:

    Picture9.png

    This allows multiple applications to be on one AWS account while limiting access of developers to the applications they are authorized for.

    Here is the actual protocol:

    1. Create a KMS master key (can be shared by multiple apps)
    2. Create an IAM role for your server
    3. Add KMS key policy that allows decrypt to your IAM role for a specific context
    4. Encrypt your secret with the key and context and store the value in GitHub
    5. At deployment or runtime, invoke KMS API to decrypt

    There is a broader issue with IAM roles because credentials are generated by calling the “magic” metadata IP address 169.254.169.254. Developers in production, even with “read only” access to the instance, could call the KMS API to decrypt the secret. Developers should not be able to generate production IAM credentials at all, so you block the magic IP address with this code:

    Picture10.png

    To automate it and make it reusable, the team created a Chef cookbook, which they call a “briefcase,” to abstract decryption of secrets. They also have the iptables cookbook to block the metadata IP address for all except a whitelist of user groups, such as root and any application-specific group that makes AWS API calls (see above code).

    Andrey mentioned that Vault may have a viable solution soon. It uses signed EC2 Identity Document to verify the caller. The current downsides are that it only supports authorization by AMI ID (but should support more soon) and secrets are not source-controlled/versioned. As Andrey noted, that is not ideal for “configuration secrets” such as database passwords.

    In a follow-up question, Andrey was asked, “What were the lessons that were most critical to learn?” He answered, “This is a new field. Most companies either don’t fully automate or don’t fully secure the entire pipeline. So, there is little information out there.”

    Well, there is some more information in Andrey’s complete talk, which you can watch online here. If you missed any of the other 30-minute long presentations from All Day DevOps, they are easy to find and available free of charge here.

    Finally, be sure to register you and the rest of your team for the 2017 All Day DevOps conference here. This year’s event will offer 96 practitioner-led sessions (no vendor pitches allowed).  It’s all free and online on Oct. 24.

    — Derek E. Weeks

  • Continuous Everyone: Engaging People Across the Deployment Pipeline

    Continuous Everyone: Engaging People Across the Deployment Pipeline

    We have continuous integration and we have continuous deployment, but what’s continuous across all of what we do is people. Even when tasks are automated, someone wrote the automation. So, Jayne Groll evangelizes about “continuous everyone.”

    Jayne is the CEO of the DevOps Institute and the author of “Agile Service Management Guide.” She talked about continuous everyone at the 2016 All Day DevOps conference. She describes it as “about people, culture and collaboration mapped into your value streams.”

    Picture1.png

    We all hear about the importance of people, culture and collaboration, but how well are they integrated into our organizations and processes? Is it in everything we do? Is it inherently continuous?

    It all starts with culture. The Business Dictionary defines culture as, “The values and behaviors that contribute to the unique social and psychological environment of an organization.” As Jayne points out, culture influences how people think, feel, interact and behave at work.

    Picture2.png

    We all have a vision of a euphoric culture. For most, that doesn’t reflect the reality of the current culture, which is generally Games of Thrones. Everyone wants to sit on the Iron Throne: ITIL/ITSM, DevOps, Security, Lean, Agile, Automation, and CI/CD. Traditionally, each of these functions has operated in isolation, convinced they are the most important cog in the machine. Now, DevOps has forced us to talk for the first time in 40 years.

    Picture3.png

    Once we acknowledge a problem, there are only 11 more steps. So, next, improving culture begins with addressing the root cause of cultural fails:

    • Language barriers (industry language)
    • Tribal knowledge
    • Poor communication
    • Thick silos
    • Disparate processes
    • Incompatible automation
    • Inadequate collaboration

    These cultural fails—both the cause and a symptom of thick organizational silos—must be broken down because they cage people’s creativity, and creativity is something you can’t automate.

    Now, people typically participate in the Continuous People Pipeline like this: Develop → Commit → Build → Test → Deploy and APIs built in between. Jayne advocates that we need to focus on Application People Interfaces between engineers and testers and ops teams. We need to let computers do the boring, repetitive tasks and let people solve problems and focus on high-value activities.

    Since not everything can be automated, continuous engagement is critical. While some advocated for “NoOps” (everything is automated), we should focus on “NewOps” to capture the talent and expertise in operations we need to leverage for tasks only people can do, such as:

    • Provide input and feedback
    • Design pipelines
    • Configure automation and APIs
    • Review test and other output
    • Design metrics
    • Facilitate feedback loops
    • Interface with the customer
    • Solve problems
    • Exchange ideas
    • Be innovative

    The bottom line is that this could be the beginning of a beautiful friendship. We need to reinvigorate the lost art of the dialogue. Actually talk to each other.

    Picture4.png

    This begins with trust. Every aspect of the Continuous People Pipeline is filled with expertise, integrity and people whot want to learn. There is a trust. You have to trust each other’s strengths and take the opportunity to engage on a continuous basis.

    It is only together that we can continuously improve.

    You can watch Jayne’s entire talk online here. If you missed any of the other 30-minute long presentations from All Day DevOps, they are easy to find and available free-of-charge here.

    Finally, be sure to register you and the rest of your team for the 2017 All Day DevOps conference here.  This year’s event will offer 96 practitioner-led sessions (no vendor pitches allowed).  It’s all free and online Oct. 24.

    — Derek E. Weeks

  • Container Considerations on Your DevOps Journey

    Container Considerations on Your DevOps Journey

    Containers are things for your things. That is true in the development world, too—containers are things for your software, and, like containers for your things, software containers can help you be organized, efficient and secure.

    Chris Morgan (@cmorgan_cloud), the Technical Director, OpenShift Partner Ecosystem at Red Hat, delivered a keynote at the 2016 All Day DevOps conference. Chris lives and breathes containers and recently discussed how they are transforming DevOps pipelines.

    Chris started with an encouraging figure: DevOps has a 74 percent adoption rate (at least on a path toward full adoption) at the enterprise level. The reason is that DevOps helps solve issues around:

    • Slow service delivery
    • Poor service quality
    • Frequent down times

    However, the needs of development and the desires of IT can often hamper full DevOps adoption, which is where containers can help. Containers allow apps to be easily shared and deployed with:

    • Consistent environment and tools
    • Predictable building tools
    • Faster deployment

    In short, containers can act as a common language between development and IT operations. And when configured properly, the operations team can utilize containers to assemble environments without compromising security.

    The survey that revealed a 74 percent adoption rate for DevOps in the enterprise also shows an 18 percent adoption rate of containers in the enterprise. Why the difference? Containers sometimes can be difficult to implement across multiple hosts.

    If you are gearing up to consider the use of containers across your IT environment, Chris offered up some things to consider:

    • Where do you store the container image?
    • Which container on which host? Which has enough capacity?
    • How do you monitor the health of the containers?
    • What do you do if a container crashes?
    • How can I address scale?

    When you have large apps that span multiple containers?

    • Which containers can access others?
    • Which containers should be deployed together?
    • How do I limit certain containers?
    • How do they find each other?
    • Is there malicious software on any containers?
    • How do I handle persistent storage?

    The bottom line is that you need more than just containers. You need a host, a platform, management and persistent storage.

    Is your organization ready to at least consider containers for DevOps? Chris reminded us, “Technology should not dictate the process. Identify the ideal process first, and then ensure the technology is flexible enough.” Start by finding some low-hanging fruit, assess skills and capabilities on your team and engage in a discovery session.

    Chris’ full keynote went into more detail, notably some solutions to implementing containers at the enterprise. You can watch Chris’ full All Day DevOps conference session (just 40 minutes) or you can view the other 56 presentations from the 2016 conference online and free of charge here.  This blog series is reviewing sessions from the 2016 All Day DevOps, which hosted over 13,500 registered attendees. Next week, look for “Ops Happens: DevOps Beyond Deployment” with Damon Edwards.

    Finally, be sure to register you and the rest of your team for the 2017 All Day DevOps conference here. This year’s event will offer 96 practitioner-led sessions (no vendor pitches allowed).  It’s all free, online Oct. 24.

    — Derek E. Weeks

  • DevOps Chat: BlueOcean Puts CD Pipelines Within Reach of Every Team

    DevOps Chat: BlueOcean Puts CD Pipelines Within Reach of Every Team

    Blue Ocean brings easy-to-use continuous delivery for teams to everyone. Years in the making, Blue Ocean is a major project built from the ground up for Jenkins Pipeline. I had a chance to speak with two of the key contributors from CloudBees and the Jenkins community, James Dumay and R. Tyler Croy. Tyler is on the Governance Board for Jenkins.

    Blue Ocean provides:

    • Sophisticated visualizations of CD pipelines, allowing for fast and intuitive comprehension of pipeline’s status.
    • Pipeline editor makes creation of pipelines approachable by guiding the user through an intuitive and visual process to create a pipeline.
    • Personalization to suit the role-based needs of each member of the team.
    • Pinpoint precision when intervention is needed and/or issues arise. Blue Ocean shows where in the pipeline attention is needed, facilitating exception handling and increasing productivity.
    • Native integration for branch and pull requests enables maximum developer productivity when collaborating on code with others in GitHub and Bitbucket.

    As usual our conversation in the streaming media player is immediately below.

     Streaming Audio

    Transcript of our conversation

    Alan Shimel: Hey everyone. This is Alan Shimel, editor in chief of devopsy.kinsta.cloud. Welcome to another DevOps Chat. We have two guests today on DevOps Chat. We have James Dumay and R. Tyler Croy of the Jenkins community. James, Tyler, welcome to DevOps Chat.

    James Dumay: Nice to be here, Alan.

    R. Tyler Croy: Thanks for having us.

    Shimel: Thank you. Let’s start off though with just a little background of each of you. James, you’re coming to us all the way from Australia. In deference to the distance, we’re going to let you go first. Why don’t you share with our audience a little bit of your background?

    Dumay: Yeah, sure thing. So I’ve been probably building development tools for the last 10 years. I think I’ve found my calling here. I love making developers more productive. I also want to make developers happier. So I do have this bee in my bonnet that developer tools are difficult when they’re boring and they’re not very pleasing. It’s my life’s mission to turn that around.

    Shimel: Excellent. Tyler, how about you?

    Croy: I’ve been with the Jenkins project a little over eight years now. I started using it when I was a back-end services engineer, a few startups ago. Every single company that I’ve joined needs more continuous delivery or some non-zero amount of continuous delivery stuff. So I found myself getting more and more ingrained in the Jenkins project and bringing Jenkins to bear in companies to ship products faster. Now I work at CloudBees talking about how you can do that and help other people see the light with some of the Jenkins stuff I’ve been doing for a while.

    Shimel: Tyler, I don’t want to embarrass you. You actually are—I don’t even know the word, but you’re one of the—on the steering committee or the governing committee for Jenkins. Is that the right way to put it?

    Croy: I guess you would say I’m on the Jenkins governance board. In the beginning of the Jenkins project, there was Kohsuke Kawaguchi who was the founder of the project, and then there are a few other core folks that were floating around that helped get Jenkins off the ground. Andrew Bayer who is a major contributor to the pipeline toolset, is one of those people. So I’ve just—Jenkins became a really important part of my life. I started spending a hell of a lot more time with Jenkins, and now I’m one of the people that makes sure that the project continues to move forward.

    Shimel: Excellent. James, I don’t know if you mentioned your role other—at this point, other than you’re cuckoo for Jenkins. What exactly is your role right now?

    Dumay: I’m director of Product Management at CloudBees, but I’m also a community leader for the Blue Ocean project. I do all the outreach and talk to all the users. It’s good fun, but previously before I was at CloudBees, I was at a small Australian startup called Canva. They do an online design tool. Prior to that, I spent seven years at Atlassian, where I was product manager for actually a competing product.

    Shimel: Absolutely. Well, we’re allowed to have competing products. They may not be any good. I’m only kidding. James, you mentioned the magic word. You get the first prize today. That was Blue Ocean. Today’s a bit of a—more than a bit—today is a big day in the Jenkins community. This release has been highly anticipated. It’s been in beta and in testing, alpha before that for some time. Talk to us. What about it? Why should people care?

    Dumay: All right. That’s a big question. So I guess we’re living in a really interesting time in the industry. We have this—there’s this ride of exponential change that is happening across our industry, whether it’s tooling and technology and process, or just the raw number of people writing computer software in the—we’ll be in the year 2020 soon. I think if you look back in the previous 10–20 years, the amount of companies out there that are writing software today and were not writing software 10 or 20 years ago. So Marc Andreessen speaks about software eating the world, where all the value that companies provide slowly but surely are being made, not by the engine in a Ford motor vehicle, but it’s the software on board the Ford motor vehicle.

    So it’s an interesting time to be a software developer, or to be in this field, but it’s also a really interesting time to be a tools creator. I think with this increase of people in the industry—it’s not like we have lowered the bar to what we call a software developer, but I think there’s a lot more people today who, in their jobs, whether it be financial services, or in other industries, who are finding their job involves writing software. I think there’s a big challenge here, and Blue Ocean is founded on going out there and solving this problem, that how do we make the millions of software developers or even people who are just writing software as part of their jobs, how do we make them more productive? How do we make sure they can get what they need to get done as quick as they can get it done at the highest quality that they can get it done with the best tools possible?

    So with Blue Ocean, we’re sort of focused in on the practice of DevOps and on continuous delivery. We set ourselves a goal. What if we can make it really easy for people to adopt continuous delivery? Blue Ocean is our answer to that.

    Shimel: Tyler, James—I think James made some important points. What can you add there? Why—let’s step back a second. Let’s say our audience is vaguely familiar with Jenkins. Most of them probably use it, but there’s some maybe that don’t. What is—let’s start with what is Jenkins to them? What does Blue Ocean represent to Jenkins?

    Croy: Sort of at a high level, Jenkins is an automation server. Jenkins is probably the most well-known tool of its kind across the entire software industry. As far as helping developers build and test and deploy their software, chances are with most organization today, if someone doesn’t think they’re running Jenkins, they might just not be aware, or they might be so far removed, or that it’s hiding, that they don’t know that it’s actually there. It’s pretty omnipresent in the industry right now.

    One of the big things about I would say the shift in the industry that has made Blue Ocean important—when I first started in northern California about 10 or— [inaudible] 10 years ago, continuous integration was this really big, important thing. I kept coming to companies who either didn’t have a continuous integration server, or didn’t practice—nobody but the developers knew that there was a continuous integration server. As the industry has shifted towards this more and more rapid pace of “Let’s practice continuous delivery. Let’s get product and value to our customers as soon as possible,” it’s no longer acceptable for just the developers to get feedback to what’s going on in that process.

    So I think the important thing for Blue Ocean, and I would say broader software industry, is that continuous delivery is not a developer thing. It’s an entire team thing. It’s an organizational thing. Blue Ocean is designed from the ground up to make sure that people who might not be writing code every single day, or they might be designers, or they might be product managers, or stakeholders of some form or fashion in the software development process. They need to know what’s going on and Blue Ocean is going to give them a window into what’s going on in the delivery pipeline. That’s a pretty dramatic shift from the Jenkins that was created 10 years ago, which was just for developers.

    Shimel: Got it. Got it. Now guys, I’ve had the—I don’t know if the word is a pleasure, but I’ve had the—well, it was a pleasure in a lot of ways, but I’ve had the experience of being involved with open-source projects in the past. It’s a fickle thing being part of a community. It’s even harder when you have a corporate entity such as CloudBees in this—CloudBees isn’t alone in this. There are many companies that are mentoring, managing—whatever you want to call it—open-source projects. Talk—I loved what I saw of how that relationship was handled as it applied to Blue Ocean.

    Guys, can you give us a little color on that? No pun intended with Blue Ocean and color, but a little background on how you worked with the community to deliver, I think, what the community consensus was what was needed.

    Dumay: Just to preface this. So this is my first time working full-time on behalf of the commercial entity for _____ community. So I—when we started at Blue Ocean, I kind of put myself in Tyler’s very capable hands. Tyler had had a lot of previous experience, not with Jenkins, but with other projects, walking that fine line between community and the need for the business sponsor, I guess. So it’s been an interesting process. I think certainly for CloudBees, a healthy Jenkins community is a healthy CloudBees, and a healthy CloudBees is a healthy Jenkins community. It all feeds into each other. As long as, I think, we take things from a project perspective, like what’s good for Jenkins, we’re making good decisions in both aspects.

    Certainly, we had been running—more in terms of Blue Ocean itself, we had been running for multiple years Jenkins community surveys, and one of the top issues people would have liked to see addressed in that feedback had been the user experience with Jenkins. They really wanted it to be easier to get started. They really wanted clarity when they’re looking for particular cases, information. I think there was a real groundswell there to see some improvements, and see some sort of more modern thinking around how to solve problems. I hope that we’ve done that. Certainly the feedback that we’ve been getting from the community has been pretty great.

    Just on the community itself, I think software developers can be a pretty passionate tribe of people. So it’s—the Jenkins project is no different from the wider developer community. They’ve been passionate in a very encouraging way, in a very forgiving way as well. Particularly when you go ahead and think about any product that has—web product that has been redesigned. So the people get very upset. They’re very invested in the tools that they use, or the products that they use, and then you move their cheese, and people go “Where did that thing go? Where did this other thing go? No. I was using that. Bring it back.” I think with the Jenkins community, I think they’ve been very—I think they’ve wanted the change for so long that as things change, we—they’re getting some new goodies in their bag and they’re happy to let some of the old bits disappear.

    Croy: I think there’s also—if you look at that feedback from the Jenkins user list going back years and years and years, there’s been—most people, I would say, that contribute to the Jenkins project, note they’re new, I guess past tense, because Blue Ocean is one meadow now—knew that something needed to get fixed, and someone needed to do something to really improve and set up Jenkins for the next 10 years. The skill set and the passion and the vision really wasn’t there.

    So when James and his team, his colleague Michael Neale and some of the other people, really started to get involved, I think it was easier for other contributors in the project to rally around that. It was clear that not only did they know what they were doing, but they understood the challenge in that we needed to think about the design from the ground up, as opposed to reskinning or reseaming Jenkins. They needed to evaluate it from a user experience standpoint and start from 2016 rather than 2006, which is when the project really started. So I think that was really helpful to have some—a passionate group of people with vision to drive it forward.

    Shimel: A passionate group of people with vision can accomplish a lot. That’s for sure. Guys, let me ask you—we’re almost out of time. What—for each of you, and we’ll go with James first. James, what do you think is the single biggest aspect or feature of Blue Ocean that our listeners should take a look at?

    Dumay: That’s very unfair, Alan. I love them all.

    Shimel: How can a parent decide? But guys, so pick one. Doesn’t have to be your favorite or biggest, but one that we’d like to call out.

    Dumay: The official pipeline editor. So _____ delivery pipeline is difficult at the best of times for people to create. So we set ourselves a challenge. What if we could make it very easy for people to create a continuous delivery pipeline using Blue Ocean UI. The one thing I really like about it is it really is for everyone. We talk a lot about—we’ve got a lot of plans for the Blue Ocean in the future. One thing we talk about is the Fisher Price’ing of the editor. I don’t know if you’re familiar with Duplo blocks you give a toddler. But you can get all these different attachments, all these crazy things, and you can customize your own creation using Duplo. It’s non-checkable Lego. So really, Duplo is for everybody. So anybody of any age can—of any skillset, can construct something from Duplo.

    The thing that I like about the—I guess the vision behind the editor is anybody can construct a continuous delivery pipeline, even if they don’t have coding knowledge. Maybe they want to take—they have—we can go as far as they can have a PHP website on a Dropbox folder and they want to upload that, deploy that continuously to their webhost, their DreamHost or something like that. That’s the kind of—we want to bring continuous delivery to everyone.

    I think the communal pipeline editor is the vehicle to do that. One of the great things about the editor as well is that it is what you see is what you get. So when you build it in the editor, that’s exactly how you visualize it later on in Blue Ocean, as that pipeline is executing. It’s really powerful in that way.

    Shimel: Great. Makes sense. Tyler, how about you?

    Croy: So my interactions with Jenkins are typically when something goes wrong. I think that’s maybe a bit more common for an infrastructure administer or developer. So, for me, I would say the No. 1 killer feature with Jenkins—with Blue Ocean and Jenkins is that the clicks to get from a failure or something going wrong to seeing what actually went wrong, it’s pretty much one click. So the visualization of status is so much faster and more immediate in Blue Ocean that when I come to Blue Ocean to figure out what test failed, or what went wrong in my pipeline, I get that in a matter of seconds, as opposed to drilling around and hunting through console help. So for me, that’s probably the killer feature.

    Shimel: Make sense. Guys, we’re going to wrap up here in a moment, but just quickly, for people who want to find out more about Blue Ocean, where can they go? James, Tyler?

    Dumay: No problem. They can go over to Jenkins.io/projects/blueocean or the Jenkins blog, which is Jenkins.io/blog.

    Shimel: Okay, Tyler, I’m going to hit you with the last question. Now that Blue Ocean is 1.0, what’s next?

    Croy: [laughter]

    Shimel: [laughter] You didn’t expect to get a rest, did you?

    Croy: No, no. I think the really important thing that’s next for the Jenkins project is I would say the next three months of Jenkins pipeline and Blue Ocean adoption. Now that we’ve got the tools that we’ve brought to the broader software industry for building and constructing and running continuous delivery pipelines, we’re going to need to iterate, on them and we’re going to need to make them better as more and more people start to adopt continuous delivery. I think we’re, certainly before the end of the year, we’re certainly going to find new things that we can add into Jenkins to make that an even more enjoyable experience.

    Shimel: Absolutely. Hey guys, congratulations. I know it’s easy on release day to say something came out. There was a lot of work here, an awful lot.

    Croy: Indeed.

    Dumay: Very much so. Thanks.

    Shimel: Thank you, James, Tyler, great job. R. Tyler Croy, James Dumay of CloudBees and Jenkins talking about Blue Ocean, the big news around Jenkins and CICD Pipelines. Guys, thanks very much. Hopefully we’ll have you back soon on another DevOps chat.

    Croy: Thanks a lot. See you later.

    Dumay: Thanks.

    Shimel: This is Alan Shimel with DevOps Chat. We’ll see everyone soon on another DevOps chat, but until then, have a great day.

    — Alan Shimel

  • One Team, 5,000 Jobs: Life in the DevOps Jungle

    One Team, 5,000 Jobs: Life in the DevOps Jungle

    Damien has 5,000 jobs. While you might gasp at that workload, Damien is not stressing out. All 5,000 jobs are automated within his team’s Jenkins pipelines. How does he do it? Damien follows four key principles to keep his cool in the job jungle: self-service, security, simplicity and extensibility. But you might be surprised that one of his most important survival techniques is treating his pipeline as “not code.”

    At the recent All Day DevOps Conference, Damien Coraboeuf (@DamienCoraboeuf, nemerosa.com) discussed how his organization got out of the job jungle. Even with 5,000 jobs in Jenkins, Damien has a tiny team of six running it while supporting a large band of developers around the world.

    During his session, Damien laid out how a pipeline, the backbone of a Jenkins job, is constructed. Spoiler alert: It is pretty simple; hence, the magic of Jenkins. However, it gets complicated quickly, so Damien shared four principles his team strives toward:

    • Self-service. They don’t want the Jenkins team to be a bottleneck, and they don’t want to have to deploy an army just to run Jenkins.
    • Security. With teams of developers—many of whom they have never met—security and control is critical.
    • Simplicity. Developers can focus on developing rather than Jenkins.
    • Extensibility. As they grow and capabilities are dreamed up or implemented, their system needs to be able to adapt.

    Core to their implementation and their goals of self-service, simplicity and extensibility is the Job DSL Plugin. While Jenkins has a robust user experience, as jobs are added, it can get cumbersome and clumsy. The Job DSL Plugin brings order, automation and consistency to the jobs. Damien walks through the basics of the plugin and its benefits, noting that it treats the pipeline as code.

    Treating the pipeline as code allows the team to reuse code through versioned libraries and easily manage thousands of jobs. Requiring developers to manage more work as code does offer benefits, but giving that many people access to the code also produces security risks. So, Damien’s team actually treats the pipeline as “not code” by describing the pipeline using properties files. It also allows them to use the pipeline as metadata, so queries and reports are almost enjoyable.

     

    Damien’s team also utilizes the Seed plugin, which they maintain, to “help automate the generation and management of pipelines,” tighten security by keeping projects locked to teams and increase automation through hooks.

    The Seed plugin shares some functionality with the popular Pipeline plugin, which is maintained by a large community. However, the Pipeline plugin does not allow for treating the pipeline as “not code.” Damien’s vision is that one day Seed will be an extension of Pipeline.

     

    Damien’s team has done a tremendous job of running Jenkins across the enterprise. In fact, he just wrote me with an update that since November, his company has scaled to 7,000 jobs while still following the same principles.  If you want to learn more about how his team does it, be sure to watch his full All Day DevOps conference session (only 30 minutes). The other 56 presentations from the All Day DevOps Conference are available online, free of charge, here.

    — Derek E. Weeks

  • Defining What DevOps is Not

    Defining What DevOps is Not

    It’s not often that my emails turn into full-blown conversations on DevOps. But when it happens, it’s a great time to take advantage of the insights. John Purrier of Automic and I recently discussed DevOps sustainability. Both of us think that sustaining a modern delivery chain and adopting it are two very different things. However, most only consider the latter. And because of that, in two years (or much sooner), they will find themselves in the exact same place they started—in an environment that is unable to adapt, rooted in habit and suddenly the old guy on the block.

    (more…)

  • TASC used pipeline orchestration to onboard DevOps

    TASC used pipeline orchestration to onboard DevOps

    I often talk about how development teams ignore their pipeline. It is easy to focus on specific tools and process, but by and large the pipeline that contains them is not being treated as its own entity. Your entire process is something that should be managed, iterated over, and measured just like the code within it. While pipeline oversight is rare, there are some companies doing it, and TASC is one. At the DevOps Enterprise Summit I was able to sit down with John Gildenzoph, a DevOps Engineer at TASC, to better understand how they took a holistic strategy to adopt DevOps.

    See John’s #DOES15 talk here and his interview with devopsy.kinsta.cloud’s, Alan Shimel along with Automic’s CEO here.

    TASC is a large health benefits provider based out of Madison, Wisconsin. They have a technology team of about 100, including Developers, QA, and Ops. Their development is supporting both web and mobile applications. The mobile application is a critical tool for their customers to submit claims and manage their information, and a core web application that serves both customers and employees. After listening to John talk, I discovered that TASC found their path to DevOps by addressing the pipeline holistically.

    The problem – an inefficient pipeline

    Around 2011 TASC realized they were struggling with an outdated development process, based on the waterfall methodology. According to John, ‘deployments were manual, inconsistent, and time consuming, and infrastructure was monolithic, static, and inconsistent.’ In addition to slow processes their was no transparency into what was going on and how processes were executed. This resulted in an inability to scale, and did not support the objective they had to package and deliver apps faster, and at a higher quality.

    The solution – automate the entire pipeline

    They could have made some incremental changes to their processes to make them faster and more efficient, but this would ultimately lead them back to square one with all the same issues to fix sooner or later. They needed a new approach, one that can adapt and scale as their needs change over time. Especially with the rate that DevOps tooling is changing. John and team realized automating the entire pipeline is the only viable strategy.

    “How do we move all of this stuff forward at the same time, not just the development cycle? We wanted to move everything forward together so we did not paint ourselves into a corner” – John Gildenzoph

    It did not happen overnight, first they needed to know what they already had, and start somewhere. The first step they took was to automate parts of their development, QA, and deployment cycles. They started by automating builds with Jenkins. Jenkins being the standard in build automation, they didn’t want to reinvent the wheel and make a build automation tool themselves. These quick wins allowed them to start somewhere, and set them up for bigger wins down the road.

    The next step was to automate the entire pipeline. They realized that their needs will change over time, requiring new tools frequently. The same tools may not always be suited for the job, and new tools may rise to replace existing tools. Because single do-it-all tools tend to overpromise and underdeliver, building for the future required a ‘rip & replace’ approach to tools that makeup the pipeline. This being the case, TASC wanted to take a long term approach to managing and automating their pipeline. To achieve this they needed a pipeline orchestrator. This would make it easy to add and remove components from the pipeline as needed, and give them a holistic view.

    “we can re-evaluate … we have no real dependency on a tool. We just made a move from SVN to Git. I only had to modify a handful of SVN commands with Git and I was done.”

    To orchestrate the entire pipeline, they opted to use Automic Release Automation, an enterprise wide orchestrator. A tool they already owned but were not fully leveraging – “we basically used it like Cron to move files around” John explained. The new implementation of Automic not only abstracted their existing production deployment but also made the interfacing and execution of that deployment process simple enough for non-operations users to handle. It enabled deployment to production environments which required a sophisticated workflow to be automated.

    While Automic could in some cases be seen as a competitor to Jenkins. In their environment considerations for existing business applications was important, a place where Atomic apparently excels, which provided a greater breadth than Jenkins alone could provide.

    In addition to Automic, they are not afraid of leveraging tools. They incorporate many tools to help with automation, but also governance like Sonatype Nexus a component and artifact repository. Sonatype helped how they packaged and delivered artifacts and simplified the consistent deployment of those artifacts. And Docker to get closer to full-stack deployments and supporting their micro-services setup of around 20 services.

    The outcome – Faster delivery, higher quality

    The automation was not the only benefit. TASC found that automation is a great way of driving process change through the organization.  By taking the time to examine the current process and asking the hard questions of why they were doing certain things actually brought a lot of things to light and in the process of automating they actually improved the core processes.

    The new orchestration engine led to increased visibility into development, better QA process, and more stable infrastructure. Easy workflow monitoring meant that even business stakeholders could now gain visibility into the pipeline. All this combined to drastically reduce deployment times to less than an hour, and with zero downtime.

    The only way to break the cycle of inefficiencies in a software delivery pipeline is to automate every cycle within it – Dev, QA, and Ops. This can’t be a half-hearted, or partial effort, it needs all hands working together to overhaul every section of the pipeline. While this may sound daunting, the payoff is worth every ounce of energy spent on it.

    Adopting a pipeline orchestration tool like Automic that allows you to ‘rip & replace’ tooling whenever you see fit might not be an immediate need, but helps sustain the environment in the long term and ensures you can adapt to what comes. This takes your focus off the tools, and puts it where it belongs, on your software delivery strategy. By doing so it’s easier to adopt DevOps and enhance your pipeline in the future.

    TASC made the transition to modern development without calling it ‘Devops’. This goes to show that as long as you’re concerned about delivering higher quality software faster, a Devops-like methodology is where you’ll naturally land. Pipeline orchestration might be a new concept, but from an execution standpoint it is not new at all. It has been executed successfully within even large environments, and is the easiest way to make sure your pipeline supports current and future development.

  • The Continuous Delivery Pipeline — What it is and Why it’s so Important in Developing Software

    The Continuous Delivery Pipeline — What it is and Why it’s so Important in Developing Software

    Continuous Delivery (CD) is a software strategy that enables organizations to deliver new features to users as fast and efficiently as possible. The core idea of CD is to create a repeatable, reliable and incrementally improving process for taking software from concept to customer. The goal of Continuous Delivery is to enable a constant flow of changes into production via an automated software production line. The Continuous Delivery pipeline is what makes it all happen.

    The pipeline breaks down the software delivery process into stages. Each stage is aimed at verifying the quality of new features from a different angle to validate the new functionality and prevent errors from affecting your users. The pipeline should provide feedback to the team and visibility into the flow of changes to everyone involved in delivering the new feature/s.

    There is no such thing as The Standard Pipeline, but a typical CD pipeline will include the following stages: build automation and continuous integration; test automation; and deployment automation.

    • Build automation and Continuous Integration

    The pipeline starts by building the binaries to create the deliverables that will be passed to the subsequent stages. New features implemented by the developers are integrated into the central code base on a continuous basis, built and unit tested. This is the most direct feedback cycle that informs the development team about the health of their application code.

    • Test Automation

    Throughout this stage, the new version of an application is rigorously tested to ensure that it meets all desired system qualities. It is important that all relevant aspects — whether functionality, security, performance or compliance — are verified by the pipeline. The stage may involve different types of automated or (initially, at least) manual activities.

    • Deployment Automation

    A deployment is required every time the application is installed in an environment for testing, but the most critical moment for deployment automation is rollout time. Since the preceding stages have verified the overall quality of the system, this is a low-risk step. The deployment can be staged, with the new version being initially released to a subset of the production environment and monitored before being completely rolled out. The deployment is automated, allowing for the reliable delivery of new functionality to users within minutes, if needed.

    Your Pipeline Needs Platform Provisioning and Configuration Management

    The deployment pipeline is supported by platform provisioning and system configuration management, which allow teams to create, maintain and tear down complete environments automatically or at the push of a button.

    Automated platform provisioning ensures that your candidate applications are deployed to, and tests carried out against, correctly configured and reproducible environments. It also facilitates horizontal scalability and allows the business to try out new products in a sandbox environment at any time.

    Orchestrating it all: Release and Pipeline Orchestration

    The multiple stages in a deployment pipeline involve different groups of people collaborating and supervising the release of the new version of your application. Release and pipeline orchestration provides a top-level view of the entire pipeline, allowing you to define and control the stages and gain insight into the overall software delivery process.

    By carrying out value stream mappings on your releases, you can highlight any remaining inefficiencies and hot spots, and pinpoint opportunities to improve your pipeline.

    Don’t Add New Functionality Until You Get The Quality Right!

    Continuous Delivery is about enabling your organization to bring new features to production, one by one, quickly and reliably. That means that every individual feature needs to be tested prior to rollout, ensuring the feature meets the quality requirements of the overall system.

    In a traditional environment, development teams typically try to implement an entire new version in one go, addressing software quality properties (such as robustness, extensibility, maintainability) only when the project is close to completion. However, as deadlines loom and budget pressures grow, quality is often the first thing that is compromised.

    Poor system quality, low-user satisfaction and endless “quality band-aids” can be avoided by adopting the principle of not adding new functionality before getting the quality right. You should always first meet and maintain your quality levels and only then consider gradually adding functionality to the system.

    With CD, each new feature is required to meet the level of quality expected for the system as a whole, right from the start. Only once this quality level has been reached can the feature be moved to production.

    Getting Started With Continuous Delivery

    Obviously, organizations cannot and should not rush into adopting Continuous Delivery all at once throughout all their business units. The best approach is to focus on improving your biggest delivery bottleneck. CD will automatically show you what the next bottleneck is. This puts you on a measurable improvement path.

    The main goal of using Continuous Delivery is to roll out new features and functionalities that are better than previous iterations — gradually incorporating and refining the CD principle throughout the organization. Go slowly, go smoothly — and watch the improvements!

    Much of the material here is covered more extensively in the book, “The IT Manager’s Guide to Continuous Delivery” which  is available at go.xebialabs.com/IT-Managers-Guide-to-CD.html

    About the Author

    Andrew_Phillips_XebiaLabs_cropAndrew Phillips heads up product management at XebiaLabs. Andrew is an evangelist and thought leader in the DevOps, Cloud and Continuous Delivery space. He sits on the management team and drives product direction, positioning and planning.

  • The Delivery Pipeline is your DevOps Signature

    The Delivery Pipeline is your DevOps Signature

    Where is DevOps? Searching for it is like querying a rainbow for pots of gold. No single element in a development team’s lifecycle defines them as a DevOps-enabled organization. Were anything the rainbow and the gold, it would be the delivery pipeline. The deployment pipeline centers the team and the process, functioning as the signature of any DevOps shop.

    (more…)