Tag: continuous integration

  • The DevOps Equation: Agility + Empathy = Quality

    The DevOps Equation: Agility + Empathy = Quality

    I had the chance to catch up with Jeff Sussna (@JeffSussna), founder and principal of Ingineering.IT, ahead of his keynote address on continuous design, scheduled for DevOps Days Atlanta, April 26-27.  He discussed the importance of designing for service, responding to the unexpected, and the importance of building empathy across teams.  

    Derek Weeks (@weekstweets): Today I’m really happy that we have Jeff Sussna joining us for the latest in this series. Jeff, why don’t you introduce yourself?

    Sussna: Sure. Thanks for having me. I’m an independent consultant. I’ve been around, since rocks were young or the 80s, whichever is older. My particular background is that I have built systems and led organizations across the entire development, QA and operations spectrum. So, I’m a little unique in that respect.

    My current focus is on helping IT organizations and digital businesses really bring together their ops and design thinking so they can not only go faster and not only be more resilient, but provide more customer value at the same time.

    Weeks: Let’s talk about that a little. How is continuous design—a subject you have written a lot about—coming into play? How can it help us be more resilient.

    Sussna: Just to explain a little about what we mean by continuous design. It’s basically bringing together the entire organization and a unifying design in operations. It’s taking what we are doing in DevOps and expanding it to include product, design and support. It lets the whole organization really have an agile conversation with a market and with its customers.

    The issue that we face is that we live in a service world now. The point is no longer delivering products like cars coming off an assembly line. It’s much more about responding to our customers and having relationships with them. This is happening in a world that is more and more complex while less and less is in your control. We can’t figure everything out in advance. We have to have the ability to respond to the unexpected. We have to be able to do that at every level.


    “This is happening in a world that is more and more complex, while less and less is in your control.”


    I will give you a couple of examples of things that are perfect illustrations of this:

    I recently wished a friend of mine happy birthday. The reason I did that was that because there was a notice on my calendar that said it was her birthday. It was wrong. It wasn’t her birthday. I had no idea where that notice came from. Did it come from Skype? Did it come from LinkedIn? Did it come from email? I had no idea.

    We face that in IT, too. Systems are becoming more complex. We develop microservices. We use public and private clouds at multiple levels. We’re no longer in a situation where we can control things. We have to be able to respond and respond at all levels.


    When we make design and operations unified and continuous, we are able to say, “Where did we think we were going? Where would we need to go? and what do we do next?”


    What I talk about in my book, in my workshops, and all my work is that this needs to happen at every level. It’s not just a matter of designing for failure inside the data center or inside the infrastructure. We also have to do it at other levels.

    Here is another example:

    I was trying to get paid recently by a large enterprise client in Europe. They were trying to wire money internationally and my bank wasn’t really set up for it and they were having problems. They just kind of gave up. I kept getting these messages saying, “Sorry, we tried paying you and it didn’t work.” That was the end of it.

    I started to get frustrated and my client within the enterprise started to get frustrated to the point where they sent a scathing email to our AP department which they CC’d me on, saying, “Why can’t we take the lead to get his guy paid?”

    That’s an approach that is based on resilience in the sense of assuming that something can go wrong and figuring out what he can do to fix it when it can go wrong. As opposed to the traditional approach where we’ve been trained to push certain buttons and take certain actions. If those actions don’t work the way they are supposed to, well, I guess that is someone else’s problem.

     

    https://www.youtube.com/watch?v=ISSMHmwIMtA

     

    Weeks: This brings to light some recent conversations I was having with Paula Thrasher at CSRA. She was talking about developing systems and how to make them more resilient. She is using pictures as part of her conversations with people to help visual their work. What work steps are there? What work is flowing? What is communicating with what?

    Are you taking similar approaches with continuous design in how you work with clients to pursue it?

    Sussna: Well, I try to introduce them to the principles and I also try to introduce them to a set of practices which actually comes out of Mark Burgess‘s work on promise theory. It is really a conceptual model for how we think about success and failure and how do we think about service. For example, Amazon helps you get everything you need in your life from shaving cream to batteries to snow tires to books as quickly and easily as possible. In order to keep that promise they make a whole variety of other promises. Some of which filter all the way down to a promise to be able to go over and make a change to production as quickly as possible.

    Once you take that mindset. The interesting thing about a promise and the reason we use that word is because it is very intuitive. We all know what a promise is. We all make promises to each other every day. On the other hand, when we think about it in terms of service, IT, or whatever it may seem a little odd. The thing about promises is that on the one hand we are making commitments to do things and on the other hand promises sometimes get broken. That is actually good because it forces us to think. If I promised to pay my vendor. What do i do if I’m not succeeding in paying my vendor?


    The thing about promises is that on the one hand we are making commitments to do things and on the other hand promises sometimes get broken. That is actually good because it forces us to think.


    It’s sort of flipping our mind. Again, we are doing this in DevOps world where you start thinking about things like mean time to repair (MTTR) instead of mean time between failure (MTBF). We shift from how do we create a situation to how do we create a situation from where nothing will ever break—which becomes less and less possible. When then need to shift to a situation where we can respond more and more quickly.

    We go to executives and say, “We don’t want to worry about how often stuff breaks anymore,” and they freak out. We have to teach them and we have to teach ourselves that we can’t prevent things from breaking. Instead, let’s be able to fix them so quickly that maybe no one notices. My focus is on really helping people shift their mindset from thinking about ourselves to thinking about who we are helping. And from thinking about what we are doing to thinking about how well it succeeds and how well it overcomes problems.


    We have to teach them and we have to teach ourselves that we can’t prevent things from breaking. Instead let’s be able to fix them so quickly that maybe no one notices.


    Weeks: You talked about delivering on promises as quickly as possible. How does continuous design account for doing things as quickly as possible but making sure the quality is there at the same time?

    Sussna: In two ways—one way is that when you do things in small increments you are able to find problems more quickly. This is classic; it comes out of lean. It comes out of integration. It’s pretty basic stuff for our community now that it’s much easier to find and fix a bug five minutes after you change a code than five weeks after you change the code.

    When you put things like Linux, agile, DevOps, and design all together, they all focus on incremental feedback loops where you discover the gap between what you think you’re accomplishing and what you’re actually accomplishing very quickly. Secondly, when you take this all to the way to the marketplace and you make it part of the relationship with your customer, you can reduce a tremendous amount of waste.

    The way to think about continuous design is: it’s all about steering. Think about a boat and you’re trying to sail across a bay. You can’t just point the boat at the other side and say go because the winds are changing and you’re going to be able to respond to things that you can’t predict. If you sail for 20 minutes then discover you had pointed in the wrong direction, you’re going to be way off-course. You’re then going to generate a tremendous amount of energy getting back on course.

    What continuous design says is just an extension of continuous integration and continuous delivery taken all the way though the business. When we steer continuously we steer efficiently.


    What continuous design is really just an extension of continuous integration and continuous delivery taken all the way though the business. When we steer continuously we steer efficiently.


    Weeks: I know you’re going to be keynoting DevOps Days Atlanta coming up in April. Chris Corriere, one of the guys on the organizing committee, tells me that you’re leading a full day workshop on continuous design before that. Maybe you could tell some of the people that are attending DevOps Days Atlanta why they should be there a day earlier for your workshop and what things you’re going to address during that time.

    Sussna: To some degree what I am going to address is everything we have been talking about for the last 10 minutes or so—relative to DevOps. I think the key thing is, and we need to think about it. We need to think about more than just going fast. One of the big misunderstandings that I see with DevOps is this notion that it is primarily about delivering application changes to production faster. That’s only half of it. We really need to think about from the perspective of providing service.


     One of the big misunderstandings that I see with DevOps is this notion that it is primarily about delivering application changes to production faster. That’s only half of it. We really need to think about from the perspective of providing service.


    You know what we are talking about is essentially developers and operations people providing better service to each other. As a result, they can deliver better service to the business.

    Where we talk about microservices. You know that word service is in there for a reason. If all you do is you kind of shatter your organization and shatter your architecture into a bunch of little pieces, you still have to figure out how those pieces come back together into something coherent. The answer to that is taking a really deep service-oriented approach. That is very much what my workshop is about. What does that word even mean and how do we do it?

    Weeks: You mentioned thinking about interactions as delivering better service to one another within the organization. How does empathy come into your thinking around DevOps and Continuous Design?

    Sussna: I’d like to say agility plus empathy equals quality. What I mean by that, in particularly empathy, I think it’s important to take a minute and clarify. When we talk about empathy we are not talking about wallowing in someone else’s pain. We’re not even necessarily talking about things on an emotion or feeling level. It’s very simply the ability to see things as if from an other’s perspective. I will give you a prime example of where that comes into play and how it’s important and how it helps.


    Agility + Empathy = Quality


    When I work with clients, I tend to look at their process using their organization from the product beginning to the support end. Often times I see support getting the short end of the stick. I see places where they get phone calls from customers with features that they don’t even know exist yet. They sometimes get talked down to by development or product or even operations. They say things like, “Well you’re just supposed to support the stuff. You’re the young people. You’re the less experienced people.”

    When we have the ability to see things from their perspective, we can think about how support is actually experiencing our delivery process. If we don’t think about them when we go through the process of delivering new features, they will be lost. Them being lost will show when customers call for help. That will significantly affect customer satisfaction and then even beyond.

    The reason we need to empathize with support is so we can empathize with the customer. Often times when we build things and we don’t think about the notion that our customers might have trouble with it. On the one hand we have the illusion that if we design it and we build it really well, they won’t have trouble. We can’t perfectly predict who they are, what they need or what they will encounter. There’s an inevitability to the idea that they are going to need some amount of help.

    Weeks: I think that is probably the best description of empathy within a DevOps environment that I have heard anyone talk about. Jeff it has been a great conversation. Thank you very much. I really appreciate it and we look forward to seeing you at DevOps Days Atlanta coming up.

    Sussna: Thanks. It has been great talking to you and I am really looking forward to the conference. It will be fun.

    If you want to learn more about continuous design from Jeff, be sure to join us at DevOps Days Atlanta. I will also be at the event giving an Ignite Talk, titled, “Real Heroes Draw Pictures,” focusing on lessons learned from the 31 DevOps and continuous delivery reference architectures I assembled.

  • 5 Things DevOps is Not

    5 Things DevOps is Not

    For those who have been embracing and championing DevOps for years, the growing popularity of DevOps is as exciting as landing your first job out of college! Okay, maybe not that exciting, but still really cool. More and more companies of all sizes are embracing DevOps by including the adoption of DevOps practices directly into their fundamental strategies.

    On one side, this is giving teams the leverage they have needed all along to either begin their DevOps journey or to increase the maturity of existing DevOps adoptions. On the other side, when DevOps becomes a mandate, some of that grassroots passion can become watered down, as the use of the word becomes increasingly overused or incorrectly used.

    DevOps incorporates a new collaborative culture that embraces numerous practices combined together for a continuous software development methodology that places significant emphasis on feedback loops, collaboration and continuous improvement. DevOps requires fundamental cultural changes and includes so many practices that can be overwhelming to those just starting their journey.

    Instead of highlighting everything that DevOps is, I thought it would be good to take a step back and highlight what DevOps is not instead.

    1. DevOps is not simply combining Development & Operations teams

    We’ve all seen this one when the term DevOps comes up. “Let’s combine the Development & Operations team and, voilà!, we now have DevOps.” DevOps combines a set of processes and practices to be adopted throughout the entire delivery pipeline and spans multiple stakeholders. A couple of the key practices within DevOps adoptions include continuous integration (CI) and continuous delivery (CD). Simply combining two teams and calling it DevOps cannot accomplish those practices.

    2. DevOps is not a separate team

    Setting up a separate DevOps team is another trap many organizations fall into when beginning their DevOps journey. In general, I’m not a fan of the DevOps team as I believe it leads to more silos in the end. I’ve also found the creation of these teams can lead to further confusion when the mission is not clearly defined.

    There are some cases where a temporary DevOps team may make sense to help enable the processes and potential tooling that may be needed for adoption, but the key is making it temporary—which is often better in theory than in practice. There are several great blogs out there discussing DevOps teams such as Matthew Skelton’s blog, “What Team Structure is Right for DevOps to Flourish?”

    3. DevOps is not a tool

    Let me be the first to say I actually love the growing number of tools that enable us to continue maturing our DevOps adoptions, but I’ve noticed the use of one or two tools starts to lead to the perception that DevOps is a tool. How many times have you heard?

    “We’re already doing DevOps. We have Chef.”

    “We do DevOps. We automatically deploy through Jenkins.”

    Keep in mind, I’m a big fan of both Chef and Jenkins, but I think the power of DevOps is drastically underutilized if you’re equating the utilization of a single automation tool to DevOps success. Adopting automation and tooling is, of course, a part of DevOps, but only when combined with end-to-end practices of increased collaboration with continuous integration/continuous delivery, amplified feedback loops and continuous improvement (to name a few)! DevOps is a journey and that journey may start with a tool, but the goal should be to first identify your DevOps strategy and then find a tool or tool chain that meets those goals.

    4. DevOps is not a one-size-fits-all strategy

    Because there are so many different business drivers and technologies to consider when setting up an overall DevOps adoption strategy as well as identifying your DevOps tool chain, it’s important to apply the same tenets of DevOps to your DevOps strategy. Embrace change, gather metrics, understand feedback, fail fast and correct your course quickly. As an example, if you initially identify a tool that no longer works for your technology or environment, abandon it and move on. Just because you used it on the last project doesn’t make it a silver-bullet fix for the next one. We need to first understand our current strategy and environment, then react accordingly.

    5. DevOps is not automation

    That one may have caught the eye of a few people, so I’ll clarify: DevOps is not solely automation. Automation is absolutely a huge part of DevOps; however, it is not the only part, and deploying some degree of automation is often used interchangeably with DevOps. I think understanding the key DevOps practices is a great precursor to ensuring DevOps is viewed as more than just automation. Understanding the core DevOps principles is key to understanding the true benefits of DevOps adoption.

    There are a lot of things that DevOps is and I think I share a widely spread passion for DevOps. There are also a few things that DevOps isn’t—or isn’t solely. If you’re just starting your DevOps journey or still maturing your existing model, making sure everyone on your team has some basic DevOps education and is aware of the DevOps strategy for your project is key. It will go a long way in helping everyone understand that DevOps is about a lot of things. It’s also not about a lot of things.

    The list above isn’t exhaustive and I’m sure you all have a few you’d like to see on the list. I’d love to hear you thoughts and suggestions at @Shelbee_SE.

    You also can download and browse through the DevOps for dummies e-book.

    About the author/Shelbee Smith-Eigenbrode

    Shelbee

    Shelbee Smith-Eigenbrode is a Senior Software Engineer/IT Architect at IBM Cloud Infrastructure Center of Competency. She has worked across varying roles spanning the software development life cycle. Her experience across functional roles as well as performing that work deeply inside traditional silos has contributed to her passion for embracing the DevOps culture and transforming the way teams are delivering technologies by adopting DevOps practices. She is a member of the IBM Academy of Technology committed to driving innovation and technical advancement.

  • Flash Mob Inflection: Rugged DevOps Revolution

    Flash Mob Inflection: Rugged DevOps Revolution

    Truthfully, I was never a huge fan of the HBO series “The Sopranos.” It’s not that it wasn’t entertaining; I just didn’t agree with the “best ever” label that so many espoused during the show’s halcyon days.

    This had something to do with living in Hoboken, N.J., at the time and feeling the show was a heavily glamorized caricature of the real-life Mafioso I still perceived to be in my midst. More specifically, it’s hard for anything mob-related to stand up next to “Goodfellas” and “The Godfather”—among my favorite films, ever.

    Either way, I currently find myself identifying with the series’ defining sentiment (copped directly from The Godfather, but hammered home as The Sopranos main theme): “Just when I thought I was out … they pull me back in.”

    It wasn’t that I assumed that I’d never delve back into the domain of IT security after transitioning into the DevOps market last year; leveraging DevOps methodologies to improve security, policy compliance and related workflows stands as a core element of the overall movement.

    But, in recent months and weeks it has become increasingly obvious that the security facet of the larger DevOps revolution appears poised for explosion as the buzz around emerging best practices, business models and solutions providers grows from a low hum into a steady roar.

    After spending a few days talking DevOps security with Derek Weeks of Sonatype at CA World ’15 (we were booth neighbors and dinner mates) and giving more consideration to his company’s angle on securing the systems development life cycle (SDLC) supply chain, the internal gears really got spinning.

    Regardless of whether you buy into Sonatype’s security-heavy approach, the concept of aggressively and proactively addressing security across the DevOps spectrum makes infinite sense. The ultimate downfall of applications security has always been the challenge of securing all the underlying layers of code (and today, microservices), especially as applications are assembled using heavier doses of shared, open source componentry.

    As further evidence, nearly every major solutions vendor focused on the DevOps space—whether oriented toward the vibrant application performance management (APM), configuration management and continuous testing segments, among many others—seems to have launched some initiative aimed at integrating security into their products.

    The rise of security capabilities adjacent to—and supporting inclusion into—the DevOps world, such as container giant Docker’s Content Trust initiative, along with the advancement of related startups including Twistlock, Scalock and StackRoxs, offers more proof of this growing momentum. And that’s only the container security piece of the puzzle.

    Of course, there are also a number of emerging startups aimed at addressing the DevOps security opportunity, backed by well-known venture capitalists. A shortlist might include providers such as Immunio, Prevoty and UpGuard (formerly known as ScriptRock, which recently rebranded itself under the guise of a “DevOps security” provider), and we’re sure to see many more. And don’t forget the growing emphasis on DevOps security among established security industry stalwarts such as CyberArk, Snort and Tripwire, among others.

    For a comprehensive list of the various types of DevOps security best practices and tools that we should expect to see gain wider adoption, this October 2015 blog by noted security analyst Adrian Lane of Securosis serves as a basic framework. Each of the areas of focus that he cites clearly offers significant opportunity for both new and existing methodologies.

    Finally, one need look no further than this year’s RSA Security Conference, to be held in San Francisco at this end of this month, as tacit proof of the continued growth of what people now varyingly refer to as DevSecOps or Rugged DevOps (I prefer the latter as it just sounds cooler).

    In addition to multiple conference sessions, notably a “DevOps Throwdown” between security industry veterans Caleb Sima, Chris Wysopal and Gary McGraw, there’s also the 2nd Annual DevOps Connect: Rugged DevOps Edition. If last year’s event is any indication (standing room only, venue literally at max capacity), this daylong DevOps security track at RSA will be an even more high-profile element of the entire week.

    Further, if DevOps security—or, at the very least, the growing focus on adjacent segments such as container security—isn’t the unofficial theme of this year’s entire RSA show, I’d be extremely surprised.

    Meanwhile (self-serving), my employer CA Technologies absolutely has a lot going on that complements and contributes to this growing DevOps Security market momentum.

    For all of those reasons, here I find myself again, drawn closer back into the security space that I’ve called home for roughly the last decade. I guess that’s just how it goes, as DevOps and security are inarguably two of the most significant trends in IT these days and inextricably linked at their respective cores.

    There are definitely worse fates out there. But, just when you think you’re out …

  • Never Mind the Bollocks, Here’s the DevOps

    Never Mind the Bollocks, Here’s the DevOps

    What is DevOps?

    “DevOps is a cultural and professional movement” –Adam Jacob

    [movement (n): a group of people working together to advance their ideas.]

    So, who are these people and what’s the big idea?

    The people are system administrators, developers, product managers, QA technicians, security analysts, network engineers, and anyone else involved in the business of developing software.

    The big idea is to increase the value your product or service delivers, more quickly with less effort.

    (Having a product or service that is designed and built well isn’t inherently valuable. The value comes when your customer is able to use your product or service to solve a problem. DevOps is all about delivering value quickly and effortlessly.)

    DevOps practices and technologies are evolving, but here are some of the core concepts paraphrased–be sure to read the links in the footnotes for the full explanations:

    The Three Ways

    • Systems Thinking
      • Learning how everything is interconnected, from concept to customer
    • Amplifying Feedback Loops
      • Cutting the time between action and reaction, failure and success
    • Culture of Continuous Experimentation and Learning
      • Recognizing failure is important data about how and where to improve

    CAMS

    • Culture
      • It’s about how people relate and work, not something you can buy
    • Automation
      • Faster, less error-prone processes help amplify feedback loops
    • Measurement
      • You can’t get where you want to be until you know where you are
    • Sharing
      • Effective teamwork ignores organizational boundaries

    ICE

    • Inclusivity
      • Everyone is responsible for (and benefits from) these practices
    • Complex Systems
      • Because simple systems are simple
    • Empathy
      • Real collaboration requires real understanding of another’s point of view

    That’s the theory from a very high level; the following Q&A provides some broad explanations of related concepts. These are actual questions I was asked in a recent discussion of DevOps.

    Q: What specific skills or experience do I need before I can put “DevOps” on my resume?

    A: It depends. At this point in time when recruiters and/or hiring managers list DevOps in a job posting they are almost always looking for a “DevOps Engineer” (i.e. a sysadmin who knows just enough coding to be dangerous), or a “full-stack developer” (i.e. a developer who knows just enough about system management to be dangerous). The reality is that (as of today) there is no DevOps certification, organization or even manifesto, so anyone who pretends to authority on what is and isn’t DevOps is overstating their case at best, a charlatan at worst.  

    Q: What is source control?

    A: Source control is a repository for files that make up a modern software application during development. It maintains a historical record of changes so it’s easy to review (or rollback to) changes made to an application’s source files. The most popular source control technology today is Git, and GitHub is one of many online services used to manage Git repositories.

    Q: What is Jenkins?

    A: Jenkins is to task engines or “build servers” what Kleenex™ is to facial tissue. It’s currently the most widely used software product for automating various functions of software build and deployment. For example, a system might be configured such that any code change pushed to source control triggers a job in Jenkins to build, test and deploy that code.

    Q: What is configuration management (e.g. Salt, Ansible, Puppet, Chef)?

    A: Configuration management is a means of scripting server configurations so it’s easier to track, deploy or make configuration changes to large groups of systems.

    Q: What are containers?

    A: In the same way that server virtualization allows you to have many isolated “virtual machines” on a single physical machine, container technology allows you to have many isolated application environments on a single operating system. Containers have been around in Unix/Linux for a long time, they just boomed in popularity when Docker came along and made them easy to use. The next version of Microsoft Windows Server (2016) will introduce Windows containers.

    Q: What are microservices?

    A: Microservices are small, independent components of a software application that are built, tested and operated in isolation from the core product. Using microservices enables development to be faster and more nimble than possible when working with a large, monolithic application.

    Q: What is Continuous Integration/Delivery/Deployment?

    A: Continuous Integration is the practice of continuously integrating (and testing) new code with your existing source code. Continuous Delivery is the practice of automating as much of the build/test/release process as possible while still requiring a manual push to production. Continuous Deployment is the practice of fully automating build/test/release all the way into production.

    Q: Are there any conferences or meetups I can attend to learn more?

    A: Yes! The original DevOps conference is DevOpsDays and there are many of these events around the world annually. A popular conference with large enterprises is the DevOps Enterpise Summit.  Various DevOps-related meetups (#CoffeeOps on Twitter) are springing up around the country. Check the CoffeeOps.org website for details, or search for DevOps at Meetup.com.

    Q: [Do I need to understand everything above | Is this all I need] to be or do DevOps?

    A: Absolutely not, on both counts. You can successfully implement a DevOps way of thinking/working by incorporating all or just some of the above, and there are many other tools and practices around delivering value more efficiently. Naturally there are extensive resources available (a few linked in the footnotes) to do a much deeper dive on all these subjects.  

  • Does Agile need continuous integration?

    Does Agile need continuous integration?

    Michael Oubre – Agile evangelist, practitioner and trainer, and Marc Hornbeek – DevOps continuous testing evangelist and thought leader, share their views on popular questions regarding Agile and DevOps from their respective viewpoints.

    If you are doing Agile do you need to do Continuous Integration?

    Michael: From an Agile perspective, let’s consider your company decided to take the plunge and switch its software development process to Agile, formed Agile teams, got them some training and set them up with a coach. The team have been scrumming for a couple of iterations but there’s not a lot of work getting finished, and many of the stories are having to be carried over from sprint to sprint to get the defects resolved.

    You might be tempted to start worrying about the quality of the code being written, or whether you have the right ratio of test engineers, but it might be more useful to look at your build/test cycle. One of the big benefits of Agile is the reduction of the time between the code being written and the identification of defects – the shorter that time is, the lower the cost is to get it fixed. Agile removes the process barriers, but if you only rely on unit testing and infrequent integration testing, you won’t achieve the results that you hope for.

    Agile teams will have a very difficult time of ever reaching high performance without the benefits of Continuous Integration providing the ability to deliver new code in an integrated, testable build to the Agile teams as close to “instantly” as you can manage.

    Marc: From a DevOps perspective development teams that are “doing Agile” are the primary customers of Continuous Integration together with Continuous Testing.

    Both Agile and DevOps emphasize the importance of incremental development processes rather than large, serial waterfall processes because incremental processes facilitate fast feedback and reduce process waste compared to long waiting periods between software creation and validation. In fact the DevOps “movement” was born out of a system test consultant’s frustration with the disconnect between some Agile development teams and separate operations teams. (Reference: http://itrevolution.com/the-history-of-devops/) . Today the words Continuous Integration and DevOps, together with Continuous Testing and Continuous Delivery are practically synonymous.
    The mere fact that there are plenty of successful Agile examples that existed prior to DevOps or Continuous Integration became a word tells you that Agile can indeed be done without CI, but clearly CI is a great complement to Agile and supports the goals of Agile while helping to tie it to operations. So the strict answer the question “If you are doing Agile do you need to do Continuous Integration?” is, well, “no” but to leave it at that would be incomplete, so a better answer is “ no, but CI is definitely recommended to complement Agile”.

    Bottom line is, from both Agile and DevOps perspectives continuous integration is really important to achieving performance goals. At Spirent we think Agile and DevOps work together to help teams develop and deliver quality software efficiently. More information about Spirent’s views can be found at the following link: Spirent.com/solutions/devops

    What do you think of these views and do you have other views?

    About Michael Oubre

    Michael Oubre is an expert in Agile practice, transformation and management and has spent more than 10 years as a practitioner, coach, trainer and evangelist. His long and eventful tech career has given him practical experience and insight into every aspect of building, deploying, supporting and managing software, systems and networks.

  • Continuous Deployment is king of the DevOps hill

    Continuous Deployment is king of the DevOps hill

    One of the promises of DevOps is continuous deployment. DevOps is, first and foremost, a matter of shifting the traditional business culture more than the tools and technologies that enable it. Once that culture exists, though, one of the defining aspects of DevOps is automation and continuity.

    Earlier this year I wrote about the seeming ubiquity of “continuous” solutions throughout DevOps:

    If DevOps is the buzzword du jour for tech, then continuous is the buzzword du jour of DevOps. Everything is continuous. Continuous development and continuous testing lead to continuous deployment and continuous delivery, which requires continuous support. Continuous monitoring produces continuous integration and continuous change. Continuous security results in continuous incident response…or vice versa. To top it off, all of the continuous activities continuously feed each other to drive more continuousness in some sort of DevOps Mobius strip of continuity.

    Taking a closer look at all of that continuous continuousness, however, continuous deployment stands out as one of the prevailing goals of effective DevOps. Continuous integration, continuous delivery and the other continuous tools and processes are all in some way related to facilitating the flow of continuous deployment or monitoring and managing it after the fact—which ultimately feeds back to the beginning of the loop and back to continuous deployment.

    Chris Riley tried to muddle through the confusion of continuous integration, continuous delivery and continuous deployment. In his explanation of continuous deployment, he stressed the impact of this symbiotic loop that surrounds it: “Continuous deployment has additional costs, as it relies on instrumentation, to ensure that new functionality does not result in bugs and also on infrastructure that allows effortlessly backing out new features when a defect has not been caught by automated tests.”

    I defer to Riley on most things, but I don’t agree with his summation that everyone should do continuous integration, some should do continuous delivery, but only the rare few should strive for continuous deployment.

    I guess it depends on what perspective you’re considering it from. From the standpoint of ease or simplicity and the likelihood that an organization can execute it effectively I can see where Riley is coming from. From the point of view of the goal of embracing DevOps, though, I would reverse his order and say that everyone should do continuous deployment, most should implement continuous delivery to facilitate it, and some (or maybe most as well) should adopt continuous integration to drive continuous delivery.

    The primary objective of DevOps, though, is to deploy apps. Everything else plays a supporting role.

    That said, there is a difference between embracing DevOps with the goal of developing and deploying apps more efficiently and effectively, and adopting continuous deployment tools that automatically pass all changes that pass automated testing and are verified as deployable by continuous delivery tools into production.

    A blog post from Puppet Labs explains, “There are business cases in which IT must wait for a feature to go live, making continuous deployment impractical. While application feature toggles solve many of those cases, they don’t work in every case. The point is to decide whether continuous deployment is right for your company based on business needs — not on IT limitations.”

    There are many acronyms and various interpretations out there for DevOps, so my opinion certainly isn’t gospel on the subject. What do you think? Should continuous deployment be the primary goal and centerpiece of DevOps automation? If not, which of the many “continuous” solutions would you suggest as the central focus?

  • Librato Seeks to Unlock DevOps Application Monitoring

    Librato Seeks to Unlock DevOps Application Monitoring

    As wider adoption of DevOps methodologies continues to foster demand for new tooling to help streamline evolving workflows, at least one vendor is attempting to address the growing need for related monitoring capabilities.

    Librato, a San Francisco-based provider of Amazon AWS applications performance monitoring (APM) services, claims that it is finding favor among the rapidly expanding DevOps community driven by its ability to inject critical visibility into the engineering lifecycle.

    Acquired by Austin, Texas-based SolarWinds in January for a lofty $40 million and renamed Librato SolarWinds Cloud, the company is touting its recent adoption by well-known providers such as Slack for monitoring of DevOps-centric applications performance and health metrics.

    “At its core, DevOps is destroying conventional wisdom around ritualized processes and the need to perform them, mostly in the domain of scale,” said Dave Josephsen, Developer Evangelist at Librato. “By attempting to include monitoring as a key element of the development lifecycle, this has created a whole new range of challenges, and opportunities.”

    By providing a centralized Web interface and open API to help create visibility into critical applications engineering requirements, Librato maintains that it is creating the tooling necessary to empower integrated development and deployment processes, such as continuous change control.

    “There are a lot of disparate tools for monitoring individual engineering tasks, but there hasn’t been any practical way to bubble up that data for efficient monitoring,” Josephsen said. “Within the context of DevOps, change control is a ritual that we’ve abandoned as we’ve embraced test driven development and automated away the human element, so there’s a distinct need for monitoring of feedback and metrics.”

    For example, cloud-based productivity messaging specialists Slack are using the Librato “StatsD” lightweight daemon process to stream business and app-health metrics regarding PHP applications into the monitoring platform. Slack also leverages Librato to chart data regarding connect, query and fetch latencies for its server clusters, and to monitor internal metrics about overall system health.

    “Having those graphs side-by-side lets us clearly see and compare the effects of network latency on one of our new services, and that’s very cool,” Richard Crowley, director of operations at Slack, wrote in a recent testimonial posted to the Librato Solarwinds blog.

    Industry experts agree that broader adoption of DevOps strategies will continue to drive demand for relevant monitoring tools, with solutions such as Librato seeking to deliver “modern stack architecture for looking at the modern stack.”

    “I think that APM is always interesting because it’s always new,” said James Governor, industry analyst with RedMonk. “The pace with which we’re building new architectures to deliver services to end users is increasing all the time, and with every new wave of technology, we tend to get new and associated models of APM tooling.”

    Librato is taking advantage of the need for a system that brings together the elements necessary for cloud native monitoring, and in an agentless manner different from other competing tools such as New Relic, the analyst noted.

    However, whether or not the platform is able to break down traditional engineering siloes to enable more efficient DevOps management, as Librato officials claim, remains to be seen, he said.

    “Each wave of technology has promised to do this, to remove the inherent finger pointing, and each has always made the same claim to break down organizational firewalls,” Governor said. “DevOps hasn’t boiled the ocean in this manner yet, but there are some unique opportunities; unlike previous waves, where automation was intended to replace humans, this is now about augmenting people, and we are seeing that teams are choosing new tools from which developers can truly derive value.”

  • Essential Ingredients for Continuous Deployment

    Essential Ingredients for Continuous Deployment

    To build the system to automate the build and deployment (continuous deployment) process there are few main areas which need to be covered. Following are our prerequisites.

    – Source code
    – Builds System
    – Deployments
    – QA Automation

    To describe above here are the main ingredients
    – Configuration Management
    – continuous integration/ Automated Build
    – Automated Deployment
    –  Automated testing

    Configuration management
    Very fist step to align your development practice is to select the right source control tool. Once you have the tool then you need to think of branching policy to make sure the starting point of the workflow is well defined and accepted by developers during their development phase.
    There could be lot of process around merging, branching code review etc which needs to be address and policy driven. Some key points could be
    – Developments and release branch should be separate.
    – Each change goes qualifies for next stage only if reviewed.
    – Code quality should not go down.

    Following are a few options
     Source control tool
    – SVN (Free)
    – CVS (Free)
    – Git (Free)
    -Perforce (Paid)
    -Clearcase (Paid)

    Build system
    Team needs to have robust and automated build system which can take care of all our needs (build, analysis, deployment etc).
    – Need to have automated build system which can keep building the source code from various branches to make sure we catch the broken piece fast.
    – Build can run the unit test and provide report
    – Build can check the health of the code by running findbugs and checkstyle (if any).

    Above checks can verify the heath and quality of the code (static analysis).
    Nothing should go beyond this point if any of the above is not matching with our criteria.

    Till this point we have reached to qualify the minimum criteria for continuous Integration.
    Here are few tools which comes under Build system
    Build Tools
    – Ant
    – Maven
    – Gradle
    – Make
    CI tool
    – Jenkins
    – Cruise control
    – Luntbuild
    -Bamboo
    Repository management
    – Nexus
    – Artifactory
    (Above are not open source but they are free to use with limited features).
    Code quality check
    – Findbugs
    – jacoco
    – cobertura

    **Deployments **

    Deployments are the next big bottleneck for continuous deployment.
    We need to have a deployment tool to move ahead,
    – configuration management tool like puppet or chef
    – Private cloud management (Openstack)
    – Third party tool like xl deploy, Udeploy, Opsworks etc..

    Deployment strategy can be chosen on the basis of our need and the tool we are using, most of the plugins are available in jenkins (or bamboo) to make tight integration of build and deployment.

    (At this stage I am not considering the dynamic provisioning of hardware in this section).

    Automated testing

    Once deployment phase is over QA regression suite can be triggered via jenkins (or any other tool) and entire reports can be published to central QA metrics board.
    This this point we can even use jacoco/cobertura to provide functional code coverage(for Java code) to demonstrate how much % of code QA regression/automation is covering and what is the level of automation.
    Similar way job can be setup to raise jira/bug ticket to file issues found during any of the above step.

    Finally we can have last step which can perform the final release build and push to staging /production in similar fashion.

     Resource Provisioning

    Developers/QA can just get the hardware either from infra team or if auto provisioning is enabled they can setup the environment and use the jenkins to perform build+deployment+QA
    If we talk about automatic scaling of environment, we can use AWS for same.
    Jenkins has AWS plugin to launch instance, This can be setup to launch instance dynamically and run the application and perform the QA Automation. Once need is over aws instance can be terminated automatically.

    Reference-1

    Reference-2

  • Apply DevOps automation to APIs with Akana

    Apply DevOps automation to APIs with Akana

    One of the core values of DevOps is automation. The ability to automate routine tasks and have triggers that automatically initiate other tasks takes care of all of the routine functions so developers and IT personnel can focus on bigger issues. One thing that has also emerged as driving force for DevOps—and IT in general—is APIs. Organizations have different platforms and tools from different vendors and they need everything to integrate smoothly together. The APIs enable that synchronicity but add another layer of complexity to develop and manage the APIs as well. That’s where Akana comes in.

    Akana enables organizatiions to quickly deliver scalable applications, connect and integrate applications, and share data as APIs. The Akana API Management Platform was created to give developers and IT personnel the tools necessary to effectively develop and manage APIs. Akana recently introduced new capabilities in its API Management Platform to streamline API management and reduce the time required to update APIs.

    One problem organizations face is that DevOps disrupts the traditional workflow and creates new bottlenecks. DevOps is sort of a natural evolution of Agile development—taking the principles of Agile beyond developers to the IT infrastructure and the company as a whole so that everyone can keep up with the faster pace of development. The rapid deployment of apps, however, can have a cascade effect that impacts the API lifecycle.

    As new apps are deployed the API definitions can change. Those changes have to be propagated to the API portal. The API owner has to update associated documentation. In most cases this process is a tedious manual exercise. The new Akana API Management Platform seeks to change that by bringing DevOps-style automation to the API lifecycle management process.

    The new API Management Platform integrates with continuous integration platforms like Jenkins and TeamCity. It has provisions for tracking necessary approvals to push updates to the API portal or API gateway. It also includes support for standard API descriptor document types like Swagger, RAML and WADL.

    “APIs have fast become the de-facto standard for digital enterprises to connect applications and services with digital end-points like mobile apps and internet-of-things,” said Brent Carlson, Senior Vice President, Technology, at Akana. “With this new capability, enterprises can deliver continuous innovation with speed and agility, ensuring that new updates and capabilities are automatically, efficiently and securely delivered to their developers and partners, in a timely fashion and without manual intervention.”

    No matter how streamlined you get you’ll eventually hit a bottleneck. As more of the development and deployment process gets automated that bottleneck gets moved farther down the chain, but then new platforms and tools emerge to address evolving issues and fix the new bottlenecks. For some organizations that have already embraced DevOps APIs may be the new bottleneck, but a tool like the Akana API Management Platform may alleviate that problem.

    It will be interesting to see where the next bottleneck strikes. At some point as DevOps evolves and every element is streamlined and automated the only bottlenecks will be organic—meaning the human beings involved in the process.

  • 5 Ways Continuous Integration Reduces Risk in Development

    5 Ways Continuous Integration Reduces Risk in Development

    ***This post is based on themes and ideas from the great e-book, “Continuous Testing,” by our friends at Parasoft.

    There’s a lot of talk about DevOps and Continuous Integration in the tech space right now, but sometimes the implementation of those methods can be tricky.

    An effective Continuous Integration strategy requires DevOps teams to manage risks in software development with a range of practices.

    Here are five processes you can use to make Continuous Integration work for you.

    Automated and Self-Testable Builds

    DevOps professionals can use separate machines for software integration to eliminate the tight coupling between the build process and Integrated Development Environment (IDE) that leads to last-minute integration concerns.

    Everything needed to build the software should be contained in a separate version control repository.

    Dedicated Continuous Integration servers and open source build automation tools can run these builds through continuous testing processes and deploy the product in the required environments.

    Automated deployment further eliminates the bottleneck of waiting for other team members to deploy testable versions of the latest software.

    Synchronize Version Control Repository

    Continuous integration is only possible as long as Dev, Ops, QA and the wider IT team is tightly integrated, collaborative and supportive of each other’s efforts.

    From a tooling perspective, effective collaboration requires a separate-but-unified master database updating changes in real-time.

    All database artifacts — including scripts, procedures and other assets should be placed in this version control repository.

    DevOps teams can enable Continuous Database Integration by using automated database integration, local database sandbox and version control repository to share database assets, as well as empowering developers with capabilities to modify the database and making DBAs a part of the Dev team.

    Build a Stronger Foundation and Expand Test Coverage

    Limiting test cases to individual assets executed at every check-in to the version control repository ensures the continuously integrated builds are defect-free in combined form.

    Tests should be repeatable and performed continuously in virtualized systems replicating unavailable testing environments.

    Continuous Integration methodologies allow you to determine how much of the code was actually tested without much manual intervention.

    Although some code coverage tools can assess the percentage of test coverage by package or class, Continuous Integration systems ensure this coverage is updated in real-time.

    For instance, DevOps teams can run these tools as part of the Continuous Integration server build script every time the version control repository is updated with a code change.

    With this information, untested code would never get through to the next stage and the latest test will be conducted across all areas as required.

    Enhance Project Visibility

    With automated build testing and deployment using a centralized code repository, build feedback mechanisms update Devs and Ops with the latest test results and the necessary progress information.

    In cases when the same level of automation for complex builds seems impractical, DevOps teams using manual procedures must establish a feedback loop to bring everyone on the same page with real-time communication and notification of accurate performance and results.

    These capabilities ensure the same high standard of reporting is maintained for manual testing practices as available for automated build testing systems.

    Follow Consistent Dev Standards

    Quality code forms from the ground up, with adherence to common code and architectural standards established for an organization’s specific requirements.

    Automated inspection tools enable Devs working on short and quick builds to maintain their focus on quality from the very foundation of the software itself.

    DevOps teams should assess every integration build for potential defects that surface after integrating several iterative builds but should be taken care of far earlier during the development stages.

    In Conclusion

    The right toolset is key to Continuous Integration, as is the cultural change that facilitates this DevOps practice.

    Continuous Testing presents several areas of improvement for DevOps teams to develop iterative builds in individual, isolated branches and then merge the latest builds in a unified, single and well-synchronized Master branch.

    The idea is to ensure all risks are mitigated beforehand so DevOps professionals are not bogged down with endless needs to rework the product — and that gives them time to focus on true Continuous Integration.