devopsy.kinsta.cloud

Category: Doin’ DevOps

  • DevOps-in-Name-Only? Don’t do DOINO

    DevOps-in-Name-Only? Don’t do DOINO

    Let’s not be ‘that’ guy. And let’s not be ‘that’ organization.

    You know, the one where “DevOps” is some magic pixie dust that we can sprinkle around, and then pat ourselves on the back for being forward-thinking, modern IT managers. It’s actually harder than that. Harder but simpler. Harder because DevOps involves humans. Simpler because pixie dust doesn’t actually exist. (I prefer software instead.)

    There is thankfully a lot of good “devops” software out there. We are pretty proud of ours too but we fully admit that paying closer attention to human factors of DevOps helps ensure that you get the most from your software whether you buy it or build it.

    Respect

    Any “DevOps” organization worthy of that title needs respect and understanding between Dev and Ops. Each brings their own talents, mindsets and constraints. Those other guys generally aren’t trying to make your life difficult just because they can, or because they don’t care. They are making your life difficult because they may not fully understand your environment, your pressures, your management and even the metrics that define ‘your side’. Or maybe they have pressures and constraints of their own. This isn’t a matter of touchy-feely kumbaya. Getting to know and respect each other is actually just good business.

    Communication

    A DevOps organization should also have good communication. Note that I didn’t say lengthy communication or countless tedious meetings. Just good communication. I also didn’t specify whether that should be verbal or written. It just needs to be good and it needs to fit your organization and your environment. Whether you are a multi-national organization with datacenters across the globe or if your entire IT department can fit in one room,

    1. Focus on sharing what is important for your colleagues to get their jobs done.
    2. Make clear what you need or what you need to know from them so that you can do your job well.
    3. Don’t hide that key nugget or even that bit of bad news under mountains of data. Over-sharing is as destructive as withholding key info.
    4. Say ‘thanks’, ‘good job’ and ‘sorry’ every now and then to your colleagues in Dev/Ops.

    Process

    A) People learn by doing, more than being talked at. B) What gets measured, gets done. So find something that works for your organization and your culture to nudge the DevOps mindset that you are seeking. Maybe it’s a job rotation program (even for a couple of hours per week). Maybe it’s paid company lunches. Or working in the same physical area. Perhaps it’s having Dev report on traditional Ops metrics and vice versa. Or maybe it’s sharing the risk and rewards of holding Dev and Ops accountable for combined DevOps metrics. The point is to foster team performance in a we’re-in-this-together mindset.

    DOINO

    The bad news is that #DOINO (DevOps-in-name-only) is worse than useless. DOINO can cheapen and make a joke of a pretty good concept that should be helping IT teams accomplish more and enjoy their jobs while doing so.

    The good news is that you don’t have to be perfect at all of the human factors of DevOps to make real progress. Your improvements may be sparked by formal cross-functional meetings, informal sharing of more beers or even (my favorite) by clever new software that automates the hell out of tasks you had previously taken for granted. The important thing is to just get started.

  • Nine Common Ops Mistakes (and How to Prevent Them)

    Nine Common Ops Mistakes (and How to Prevent Them)

    Constant change is a reality for any growing, dynamic organization. Being receptive to change helps us quickly innovate and learn, yet at the same time, poorly-managed change can create instability and downtime. Ask any engineer what the number one cause of downtime in their systems is, and they’ll say change – software changes, network changes, configuration changes. While it would be nice to avoid this instability by working on completely static systems, to cope with the needs of modern technology businesses,  operations teams must learn to better manage change by preventing common ops mistakes before they compromise systems.

    Arup Chakrabarti, operations engineering manager and my colleague at PagerDuty, stopped by Heavybit Industries, a community workspace for developer-focused entrepreneurs, to discuss the biggest mistakes an operations team can make and how to head them off:

    1. Getting It Wrong in Infrastructure Setup

    • Creating Accounts
      A lot of people use personal accounts when setting up enterprise infrastructure deployments. Instead, create new accounts using corporate addresses to enforce consistency.
      Be careful storing passwords. Keeping them in your Git repository could require you to wipe out your entire Git history at a later date. It’s better to save passwords within configuration management so they can be plugged in as needed.
    • Selecting Tools
      Another important consideration with new deployments is selecting your tools wisely. Leverage PaaS tools as long as possible so you can focus on acquiring customers instead of building infrastructure. And don’t be afraid to employ “boring” products like Java. Well-established, tried-and-true tech can let you do some really cool stuff.

    2. Poorly Designed Test Environments

    • Keep Test and Production Separate
      Don’t mingle your test and production environments. Be sure to set up test environments with different hosting and provider accounts than what you use in production. At the same time, make sure your test environments resemble production infrastructure as best as possible.
    • Virtual Machines
      Are you performing local development? There’s no way around it: applications will run differently on local machines and in production. To simulate a production environment as closely as possible, create VMs with a tool like Vagrant.

    3. Incorrect Configuration Management

    • Infrastructure-as-code
      Essentially, infrastructure-as-code is the process of building infrastructure in such a way that it can be spun up or down quickly and consistently. Server configurations are going to pose problems regardless of where your infrastructure is running, so you have to be prepared to restore your servers in as little time as possible. Ansible and Chef are two tools that makes infrastructure-as-code deployment super-simple for ops teams.Whatever tool you use, as a rule of thumb, it’s best to limit the number of automation software tools you’re using. Each one is a source of truth in your infrastructure, which means it’s also a point of failure.

    4. Deploying the Wrong Way

    • Consistency matters
      Every piece of code must be deployed in as similar a fashion as possible. Standardizing deployment practices takes time and effort, but eliminating variability and the potential for human error is essential for success. And that applies to rollbacks, too— anyone with deploy rights should also be able to easily roll back any code they ship.
    • Orchestrate your efforts
      Powerful automation software can certainly help enforce consistency, but automation tools are only appropriate for big deployments. When you’re getting started, Arup suggests running development using Git and employing an orchestration tool, such as Capistrano for Rails, Celery for Python or Ansible and Salt for both orchestration and configuration management.

    5. Not Handling Incidents Correctly

    • Have a process in place
      Creating and documenting an incident management process is absolutely necessary, even if the process isn’t perfect. Be sure to review the incident-management document on an ongoing basis, too. Incident response should be defined relative to the severity of the incident— a minor latency blip and a total downtime event have different response workflows and expectations.
    • Put everyone on-call
      It’s becoming less and less common for companies to have dedicated on-call teams – instead, everyone who touches production code is expected to be reachable in the event of downtime. This requires a platform that can notify different people in different ways. What really matters is getting a hold of the right people at the right time.

    6. Neglecting Monitoring and Alerting

    • Start anywhere
      The specific tool you use for monitoring is less important than just putting something in place. At PagerDuty, we use StatsD in concert with Datadog; open-source tools like Nagios can be just as effective. For application performance management, you might also look at a tool like New Relic or AppDynamics. There are hundreds of great monitoring tools out there, but the important thing is to get something in place and make sure you’re alerting on the exceptions that matter to you.
    • Check externally
      If uptime is important to you (and let’s face it, it always is), make sure you’re also running some sort of external check on your service or site, with a tool like NodePing or Ghost Inspector.

    7. Failing to Maintain Backups

    • Systematizing backups and restores
      Just like monitoring and alerting, backing up your data is non-negotiable. Scheduling regular backups to S3 is an industry-standard practice today, and always have one more backup method than you think you need.And backups are useless if you can’t restore from them! At least once a month, try restoring your production dataset into a test environment to confirm that your backups are working as designed.

    8. Ignoring High Availability Principles

    • “Multiple” is the key.
      Having multiple servers at every layer, multiple stateless app servers and multiple load balancers is a no-brainer. Only with multiple failover options can you truly say you’ve optimized for HA.
    • Datastore design matters, too.
      With multimaster data clusters like Cassandra, individual nodes can be taken out with absolutely no customer-facing impact. Clustered datastores are ideal in fast-moving deployment environments for this reason.

    9. Falling Into Common Security Traps

    • Relying solely on SSH
      Use gateway boxes instead of SSH on your database servers and load balancers. You can run proxies through these gateways and lock traffic down if you suspect an incursion.
    • Not configuring individual user accounts
      When an employee leaves your organization, it’s nice to be able to revoke his or her access expediently. But there are other reasons to set people up with user accounts to your various tools. Someone’s laptop may get lost. An individual might need his password reset. It’s a lot easier to revoke or reset one user password than a master account password.
    • Failing to activate encryption in dev
      Making encryption a part of the development cycle helps you catch security-related bugs early in development. Plus, forcing devs to think constantly about security is simply a good practice.

    These are just a few of the mistakes that operations teams can potentially make when managing change. There are millions more out there that you’re likely to run across, but these nine mistakes tend to be the most commonly seen, even at larger companies like Amazon and Netflix. By putting these best practices in place, you’ll help set your team up to effectively manage change.

  • Is DevOps a Competitive Advantage? A Debate

    Is DevOps a Competitive Advantage? A Debate

    The idea of DevOps being a competitive advantage is not a new one. Service providers, software sellers, and internal IT frequently use this argument to justify all kinds of investments. But is there any real merit to this argument? Is DevOps truly a source of sustainable competitive advantage? Or is it simply just hype and a new way to sell a product / service?

    Here are some of the more popular points of view from both sides of the aisle.

    “Yes Absolutely” “No, Go Take A Hike”
    Speed is sexyIn the mobile, social, and web world customers expect speed and responsiveness. Bringing an idea to the market faster than your competitors is definitely an advantage and DevOps allows us to do just that. DevOps is not the only fish in the seaAgreed that speed is sexy. But at this given moment there are multiple technologies / philosophies out there that claim to do the same – i.e. cloud, big data, analytics etc. What makes DevOps so different?
    DevOps is hard to replicateWhile purchasing tools is easy, the real benefit lies in getting the culture and process right. Any organization that can get that right will be able to form a sustainable competitive advantage. Same has been said about other strategiesRemember when outsourcing started. It too was touted as a sustainable competitive advantage. Now it is the de facto strategy for IT erasing any advantages that the early adopters had. DevOps, if it does take off, too will follow the same path.
    New IT operating modelDevOps represents a new way for enterprise IT to operate. It breaks down the silo structures and unlocks the value trapped within the existing organization. Enterprises are not start-upsJust because it worked in a start-up environment does not mean that it can work in a large enterprise.
    Listening to customersSpeed is only one equation of DevOps advantage, the other is faster feedback loops so that you can determine that you are going in the right direction. Combine the two and you have a powerful competitive advantage. Not an IT responsibility / competencyIsn’t that what most organizations already do? I have my entire business / product / marketing org. focused on just that. The last thing I need now is IT to jump in and muck it all up.
    The new Software Delivery LifeCycle (SDLC)DevOps represents a new way for enterprise IT to operate. It breaks down the silo structures and unlocks the value trapped within the existing organization. Just good marketingThis is just repackaging of an old wine into a new bottle…DevOps is just a new thing. Like all other fads this too will pass.

     

    My Take

    Allow me to be the quintessential consultant and speak from both sides of my mouth. DevOps is and is not a competitive advantage. Now before you close the browser and move on, let me explain.

    DevOps in the short term is most definitely an advantage. Moving an organization from Traditional to a DevOps approach is not an easy task. Any corporation that can successfully achieve this transformation will be the first to reap its benefits – from faster speed to market to being more responsive to their customers. And the benefits are real! Ask Netflix and Etsy, who have been able to out innovate their competitors and gain a sizable lead in the market.

    However, in the long term, as more and more firms invest in DevOps the advantage will continue to shrink. DevOps, if you strip all the marketing jargon and go down to the most basic definition, is about reorganizing the internal plumbing of IT (i.e. how IT performs the various tasks within the software delivery lifecycle). And while in the short term plumbing can allow you to do some neat stuff, at the end it’s only internal plumbing and rarely a source of lasting competitive advantage.

    To see this is action, look at the automotive, financial, and then our very own Information Technology industries. In all three examples, the company that was able to transform was able to establish an early sizeable lead, reap the rewards, and the rest of the industry forced to catch up eventually.

    In automotive industry, early investment in lean and TQM earned Toyota customer loyalty and a significant lead over its traditional competitors. And while the lead lasted for a decade (short time in an industry where product lifecycles are 4 – 5 years), competitors quickly started to invest and catch up. You can now argue that U.S. manufacturers are at par with Toyota in both quality and speed. Where the competitors have still not caught up is in perception.

    In the 1970s it was not unusual for a home buyer to wait for up to 90 days to get a credit decision and a loan, as per “industry standards” in those days. Citibank rearranged its internal processes to give buyers a decision in 30 days. It took the industry by storm. Competitors eventually caught up and now 30 days is the norm.

    And as for our very own, Information Technology industry…outsourcing was considered to be a competitive advantage for many of the first movers. Like other strategies, it too required a structural change to make it happen. Many of the initial pioneers were able to cut IT costs by over 30% and establish a cost advantage over their competitors. But within a decade competitors copied, banked the savings, and eroded the cost advantages.

    In my opinion, DevOps too will follow a similar path. A few leaders will emerge (already happening), they will set the pace of change for the industry, and the rest will have to follow….whether they like it or not.

     

    About the Author

    MustMustafa Kapadiaafa Kapadia is the Service Line Leader for IBM’s DevOps practice, a business advisory practice focused on helping large enterprises transform their software & application delivery.  He has over 17+ years of experience in the tech. space, both as a service provider and a management consultant.

    Prior to joining IBM, Mustafa was a management consultant with Deloitte’s Strategy & Operations practice. He lives in San Francisco Bay Area, is an avid blogger (when time permitting), a speaker, and adviser to start ups.

    Connect with Mustafa LinkedIn| Twitter

  • 9½ Simple Steps On How To Start With DevOps Today

    9½ Simple Steps On How To Start With DevOps Today

    Isn’t it funny that a lot of people, me included, don’t really know the difference between DevOps and Continuous Delivery? Wikipedia tells that both are pretty much overlapping. In order to continuously deliver one must be familiar with DevOps. Anyway, I will tell you what you can do now to start with DevOps and Continuous Delivery. Here are 9½ simple approaches that tell how you can start immediately.

    (more…)

  • The Hypocrisy of Vendor Neutrality

    The Hypocrisy of Vendor Neutrality

    Disclaimer: I’m going to name the product and company I work for in this article.  I assure you this does not mean this post is a product pitch.  Really, I promise.

    In my capacity as the open source community manager/dev relations/community evangelist/[Enter Alternate Title Here] at Cloudify, I’m a huge advocate of knowledge sharing. My personal credo is that the single most important element in a product’s success is spreading the wealth of knowledge — empowering your users to be autonomous, and even hopefully achieve a level of understanding that surpasses the original development team. Intelligence of the masses. Better code. Better product.

    To this end, I’ve been trying to cultivate a culture among our R&D to document document document everything you do.  I’m not talking about a product’s traditional documentation. I’m talking about those less formal processes of trial and error and learning that are never formally written down anywhere – and are lost when the person moves on to new endeavors, or even forgets after a time.

    I’m also a big fan of internal wikis…but that’s an even tougher culture to cultivate.  On top of toiling endlessly to write a great product, and then adding the necessary product tools to make sure people can actually use it – from docs to tutorials – the last thing devs want is the added overhead of actually having to document the technical aspects of the process.  So I relent.  But I have a dream, nonetheless.

    Backtracking to the actual topic at hand.  So in terms of interesting stuff my team is always doing – and working with excellent, awesome, interesting and cutting-edge open source tools – we write a lot of blog posts.  Really good ones (if I may say so myself) – technical, hands-on, useful…actionable posts.  Kind of like meta-documentation.  On top of these posts, and the existing tools we leverage in our daily work – we’ve actually discovered a lot of gaps with known tools, and have written additional open source tools that help us get the job we need done – not even as our primary work at hand, just as an added bonus.  And I’ve found a scathing irony/hypocrisy when it comes to propagating this knowledge to the masses.

    True story.  One of our devs did an excellent job of porting Vagrant .box files to AWS.  He wrote a really great post about it.  There was one caveat though, in the first line he had mentioned why he had needed to do so in the context of his work. He made the major faux pas of mentioning the product he did this for.

    I submitted the article to a prominent tech publication – and the response I received was:

    “Sorry your product is mentioned in the first paragraph of the post…we’ll need to pass”.  

    That was the only mention of the product in the entire post (btw, in this new post, we were actually asked to add a bit more info on the technologies mentioned in the article, hence the augmented intro. The initial post started with the pain point).  The entire post focused on Vagrant and AWS  – these products they had no issue with.  But the mention of our product in the first paragraph.  Tsk tsk tsk.

    Now, I truly and deeply understand that there are plenty of blog spammers out there.  Doing all kinds of SEO shenanigans to build traffic, and writing fluffernutter posts.

    But seriously just reading the article, and personally I felt this was one of the best written ones I’ve had, you can tell this one does not even remotely fall under the same category.

    So then I started to have a discussion with a team member who really likes to write posts.  And I love him for that.  And we asked the crazy question of… when do you reach that critical mass that you can freely mention your product and its values in the context of other technologies and technological needs without being perceived as pitchy?

    Why are certain technologies from the Vagrants to the Dockers (open source) and the AWS to VMwares and even Apples (not even remotely open source) given a stage to discuss the benefits of their products without being perceived as pitchy – while others are completely passed over or even scorned for their mention?

    Is it a critical mass thing? A tipping point? Is it what delivers traffic?  How can you know a product won’t deliver traffic if it’s not allowed to be mentioned? How can a product garner adoption if it isn’t known about?

    And let’s be honest, nearly all open source tooling these days has a monetization angle – people can’t work for free, it just isn’t a sustainable model.  So every single tool out there has what to gain.

    So who are those powers that be that decide which product is worthy of being mentioned, and those that are only possibly allowed to be mentioned (best case scenario) in the author’s bio?

    The even bigger irony is that these sites that are offered these technical posts, and are monetizing themselves off of these contributions, and building their own reputation and traffic and traction and SEO…

    I’m not intending to sound bitter. I know these sites receive A LOT of content, and have to be very strict with moderation.  I get it.  I also know that while pastures may seem green to us on this side of the keyboard – tech sites from the prominent GigaOms to the more niche Dr. Dobbs are shutting down left and right.

    And this is where I have to give a shout out to the DZones, devopsy.kinsta.cloud and Developer.coms of the world – who do understand the technical worthiness of posts that albeit mention a now esoteric product, but provide information of technical value to their readers, and publish them.

    Who knows – maybe the sites that have the foresight to give respect to new tooling will have gotten in on the ground floor of the next Docker.  (Because you just can’t have a technical post without mentioning Docker.)

  • 7 Principles for Building Amazing Wearables

    7 Principles for Building Amazing Wearables

    We are seeing a wave of wearable products coming to market. According to IDC research 19.2 million shipments this year will build to a global market of 111.9 million units by 2018. The long predicted rise of wearables is upon us.

    Fitness trackers were in the vanguard then came smartwatches. Pioneers like Pebble are now being joined by Google’s Android Wear platform, the Apple Watch, and Microsoft Band. New form factors are emerging, from the Moto Hint earpiece to Google Glass. It feels a lot like the early days of feature phones and MP3 players. But, the wearables industry is lacking some guiding principles.

    To create a truly compelling experience, manufacturers and designers in the world of wearables need to consider the following:

    1. Superhuman extension of you

    Wearables should be an extension of you that gives you super powers. Like any good technology, they need to make life easier. Wearables should allow you to see better, hear better, think better, and optimize your performance. Wearable technology should be your personal force multiplier, a mentor, an assistant, and a confidant. And it needs to be smart enough to know when to switch roles.

    1. The world is the interface

    When wearables and the digital world intersect the world becomes the interface, not a black piece of one inch, four inch, or 10 inch glass. We need to break away from the tyranny of the touchscreen. Consider the Apple Watch giving map directions using haptic feedback. You should be able to control things at a distance. We need new ways of interacting with the world by simply looking, saying, pointing and gesturing.

    1. Act cool and be polite

    Devices that are present with us all day hearing and seeing what is going on, need to be smarter about how they join the conversation. They shouldn’t talk when you’re talking. They should only interject when they have something relevant to say or show you. They need to get to know you, to learn, and to customize their behavior for you. You don’t want a notification when you’re in the middle of an important conversation. Wearable technology must be able to act cool and be polite, just like a person.

    1. Respect the human condition

    Wearables need to improve the human condition, not make it worse. You must be able to maintain privacy and trust, especially for devices that witness your whole day.  We need technology that doesn’t enslave, but rather that enables us. The loveable, intelligent operating system from the movie Her, not the Big Brother state of George Orwell’s 1984.

    1. Keep me present

    Wearables shouldn’t focus on glanceability, they should focus on keeping us present. Anticipate things for us so we don’t have to drift out of the present to worry about the future. The power of now is that now is the only moment we have. Too often our technology takes us away from the here and now, making us less present and less human.

    1. The right fit

    Technology that you wear all the time has to be incredibly comfortable. It has to fit right, or you won’t want to wear it. Most hearing aids are customized for the person wearing them. We’ll see more custom fitting of wearable technology in the future. The only technology that’s more personal than wearables will be embedded, but that’s another story.

    1. Invisible or fashionable

    If you can’t make your wearable technology invisible at least make it fashionable.  People don’t want to feel socially awkward. When Nielsen surveyed consumers 62% said they wish wearables came in forms besides wrist bands and watches, and 53% wanted wearable devices that look more like jewelry. Most of the time technology should not be seen or heard.

    The perfect nexus of hardware and software remains elusive right now. The iPhone moment for wearables has not yet arrived, but if designers can keep these seven principles in mind it surely won’t be far away. 

    About the Author/ Guy Bieber

    GuyBieber_Citrix-001Guy Bieber is the Director of Strategy and Innovation for Citrix. He formerly worked as the Chief Architect for Warrior Systems at General Dynamics on the first deployed wearable computing platform for soldiers called the Land Warrior System. Citrix is a global leader in mobile workspaces, providing virtualization, mobility management, networking and cloud services. 

     

  • What DevOps Needs to Know About Microservices

    What DevOps Needs to Know About Microservices

    In this fast-moving, immature space, expect to spend cycles assessing where the greatest value lies for each team and the business as a whole.

    The term “microservices” is popping up everywhere in software development circles. Indeed, microservices are frequently touted as The Next Big Thing in terms of how to write applications to avoid the problems of complex, monolithic architectures.

    From a Devops perspective, the microservices discussion is usually related to Docker or other containers. Both share an emphasis on lightweight individual units that can be managed independently, so containers are often regarded as a natural implementation choice for a microservice architecture. But what are microservices actually?

    Microservices constitute a software architectural style that approaches a single application as a suite of small services, each running its own process and communicating with lightweight mechanisms, such as an HTTP-based API. The services are small, highly decoupled and focus on providing a single “useful” business capability. Typically, microservices involve very little centralized management, may be written in different programming languages, and use different data storage technologies.

    A good way to understand the microservice style is to compare it to the monolithic style, in which an application is built as a single unit. A classic example of this approach is the “standard” three-tier enterprise application, consisting of a client-side user interface, a database, and a server-side application. Changes made to monolithic applications can be both painful and costly as any change made to a small part of the application requires the entire monolith to be rebuilt and deployed.

    Microservices clearly offer many advantages — notably the ability to make small changes to code quickly and efficiently, and with minimum risk. The services are organized around business capabilities, and they lend themselves to Continuous Delivery.

    Some of these advantages include the following:

    Small, Easy-to-understand Code

    As a microservice app is responsible for only one thing it requires little code, is easy to understand, and involves minimal risks when changes are made.      

    Easy-to-scale, Easy-to-deploy and Easy-to-discard

    Size is important. Microservices technology make it easy for developers to scale, deploy and discard apps and features. In contrast, a single change to a monolithic app requires changing the whole application.

    Smaller Teams

    It’s much easier and faster to work with a small team than a large one. Each small team can own a microservice and access other services via a high-level API.   

    Facilitates System Resilience

    If a monolith application stops working a lot of functionality will stop working. In contrast, if a microservices app stops working only a small, specific functionality will be lost.

    However they are not without their serious challenges, especially relevant from a DevOps perspective — such as coordinating interdependent deployments, fusing developer skills with technologies, and confronting disagreements between developers. 

    As developers break down monolithic applications and move complexity from within to between applications, obvious challenges come to the fore. Some of these include the following:

    Coordinating Interdependent Deployments

    Individual microservices are supposed to be independently deployable, but in order to provide any kind of useful functionality from the perspective of the user you typically need to get a whole set of microservices running.

    Deploying microservices requires a new generation of tools that can manage a multitude of simple deployments, while tracking and managing their dependencies. These tools need to be able to bring everything together on time to deliver an app without being too rigid, for instance by supporting version ranges for services rather than fixing on individual versions. These tools should help developers to deploy different bundles of services easily, without having to ask an administrator to reconfigure the deployment tool.

    Planning and configuration should be done through a single interface with a real-time view of app deployment. Essentially, developers should seek to attain the same balance of speed and efficiency that Continuous Delivery offers.

    Microservices Need Skilled Developers and Operators

    While there are many potential productivity boosts to separating services, tracking all of the relationships between them is vital. The team needs to know the entire development landscape and be able to test live components together.

    Testing isolated services is a simple enough proposition, but testing the entire user-facing business service can be problematical.

    Developer Disagreements

    There will be a lot of time-consuming debate and friction when developers butt heads about the pros and cons of keeping a monolithic stack versus embracing microservices: which applications to refactor, which to rewrite, which to leave alone, etc.

    The worst-case scenario: developers lose sight of the overarching system goals, focus on the service they are working on, refuse to be flexible, and resist being part of an overall team.

    Finding the right set of technologies for a microservices implementation will not be a quick or simple task. As this is a fast-moving, immature space, expect to spend a lot of time assessing where the greatest value lies for each team and the business as a whole. Delivering the greatest benefits may mean adopting a hybrid system that draws on and integrates different technologies.

    About the Author/Andrew Phillips

    Andrew_Phillips_XebiaLabs_cropAndrew Phillips heads up product management at XebiaLabs. Andrew is an evangelist and thought leader in the DevOps, Cloud and Continuous Delivery space. He sits on the management team and drives product direction, positioning and planning.

  • Why do DevOps? Winning hearts and minds with a Lean boxscore

    Why do DevOps? Winning hearts and minds with a Lean boxscore

    Change is difficult. It can be painful, disruptive, and costly. Making a case for change requires us to win over hearts and minds. In my experience, we need to win the hearts of the ‘do-ers’, the people whose daily work is going to be disrupted. And the minds of the executives, whose financial support and personal sponsorship we need to make change happen.

    A Lean boxscore allows you to address both the hearts, and the minds. Boxscore is a simple tool which we use to compare the current state with the future state, and turn that comparison into a compelling case for change.

    Step One: Current State Baseline
    Let’s start with where we are today, by capturing performance metrics that tell an accurate story of today’s development lifecycle. This may mean looking beyond the project plan to the reality, by the way. By interviewing or workshopping with development and operations teams, we get a true picture of the way the development process currently performs.

    We’re always interested in end-to-end leadtime, of course – from the moment the business provides a prioritized set of requirements, to the moment that functionality is in production and working as intended. We’re also interested in how much wait time there is within the process – how much time between steps, waiting for an approval, a developer, or a tester. We do this by creating a value stream map, capturing all those metrics and validated by the teams who work within this process every day.

    We want to look beyond leadtime and wait time too. We capture the number of resources involved in each release, the number of handoffs, and the number of bugs caught at each stage of development and deployment. If we can get to it, we might capture the percentage of time a developer spends writing code for their current release, as opposed to firefighting issues from the last release, arguing over schedules, trying to secure an appropriate environment, or other ‘non value adding’ activities. We might also capture some usage statistics, or other measure of customer satisfaction, and any relevant performance indicators, like server response time.

    leanboxscore

    Before moving on, it’s crucial to ask stakeholders: ‘is this a fair reflection of the way things currently work?’

    Step Two: Future State Target
    This next step requires some imagination, and some bold thinking. Imagine a world where you’ve adopted DevOps: you’ve removed the waste within the development lifecycle and instilled the ‘shift left’ principle in all your working practices, your ‘4 in a box’ teams are rapidly developing, testing and deploying code, Cloud-based environments give you immediate provisioning, and automation and smart tooling have taken the bellyache out of testing. What does that future look like? How quickly can you develop – if provisioning is immediate? How many bugs do you expect to put live – if testing is automated and your production environment matches your test environment? How many changes to scope – if you’re working with smaller batches and faster delivery?

    We create a new value stream map, which details the way this brave new process will work in practice. This allows us to set sensible but challenging targets for our future state.

    leanbox2

    Before moving on, be sure to ask stakeholders: ‘are these targets achievable, if we make the changes we have outlined?

    Step Three: So What?
    Finally, we get to the money question. What’s it all worth? For each improvement there will be an impact. Many of these will be financial. Ask your stakeholder ‘if we can get this new functionality out to market 2 months faster, what does that mean for revenue?’ Or ‘if we reduce the number of defects in production by 90%, can we reduce our operating costs?

    leanbox3

     

    Putting a dollar value on the financial impact will win over those minds with a logical, fact-based argument. Now, investing in a DevOps pilot seems like something they can’t afford NOT to do.

    And the case for change for hardworking developers, testers and production resources? Metrics like number of handoffs, bugs and rework, and batch sizes can easily be translated into ‘how much pain you have to live with each day at work’ to win those hearts over.

  • Think you’ve established Kanban? Why you might only be halfway there

    Think you’ve established Kanban? Why you might only be halfway there

    As a Lean Coach working in IT I often hear development teams refer proudly to their scrumboards as an example of Kanban. To which I say, ‘that’s a great start… but it’s really only just the start’.

    Let’s start with what a scrumboard is, and does. A scrumboard is a visual display of work, showing the progression of user stories in development. As you see some stories moving through the phases of development, and others getting stuck, you huddle to figure out how to get them unstuck. Now you’re using your visual display for visual management, which is a great Lean solution.

    Change management boards, like the one used in The Pheonix Project, are another good example of visual management. The physical display of changes, shown by release date and system, is a great way to get a handle on the amount of change and risk you’re throwing into live production. When you start moving them around to limit the risk, again, you’re using visual management.

    The word ‘kanban’ roughly translates as ‘signboard’, and a scrumboard really is a good start. But it’s only half the story. Until you limit the number of stickies on your change management plan, or the number of story cards on your scrumboard, you’re really only half way to kanban nirvana.

    True kanban requires you to limit the amount of work in progress and to establish the pull of work. In a manufacturing plant, a kanban card is attached to each unit in production. When a completed unit is, say, taken off the shelf by a customer, the card is sent back into the production line to signal that there is now an empty slot on that shelf which needs replenishing. Limiting the kanban cards sets a constraint around the amount of work in progress – and this limit should reflect the capacity of the process. In the same way, you can limit the number of story cards, boxes on your scrumboard or spaces in your change management plan. When a scrumteam complete a story that frees up a slot for them to pull the next story into their workload: the visual display should reflect this by showing each team’s capacity. In addition to visual management, you can create rules around work in progress, requiring each scrumteam to limit their intake so they have no more than, say, two workpackages in progress at a time.

    This is all well and good, but what do you do when demand from the business requires a greater capacity? Historically you’d have thrown that extra demand into the mix, asking your teams to do more with the the same amount of resource, maybe pushing back a release date to accommodate the extra work.

    Kanban requires you to create additional capacity in the system to allow for the additional demand either by:

    • rebalancing the load, for example switching some resources from a work package that is ahead of schedule
    • removing or deprioritizing other work and putting work on hold
    • increasing resources by bringing in new team members

    All of these actions will generate some churn, so it’s going to be worth a tough conversation with your customer first. ‘Are you sure you want me to disrupt the current sprint, or the next release, to squeeze this in? How about we pick it up next week, when we’ll have some new capacity?’ By limiting your work in progress, you’ll be speeding up the flow of development into production, so you can make that ‘next week’ promise with increasing confidence.

    So the next time you walk over to that scrumboard, why not take the next step, and limit the work in progress to truly establish kanban.

  • Reducing Incident Resolution Time

    Reducing Incident Resolution Time

    From raw incident count to response time, there are several key metrics that top Operations teams track to measure and improve their performance. One of the most popular metrics teams track for is mean time to resolution (MTTR). It’s the time between failure and recovery from failure, and is directly linked to your uptime. While MTTR may be the gold standard when it comes to operational readiness, it’s important for teams to look at the bigger picture to effectively decrease incident resolution time.

    Putting MTTR into perspective
    Your overall downtime is a function of the number of outages as well as the length of each. Dan Slimmon does a great job discussing these two factors and how you may want to think about prioritizing them. Depending on your situation, it may be more important to minimize noisy alerts that resolve quickly (meaning your MTTR may actually increase when you do this). But if you’ve identified MTTR as an area for improvement, here are some strategies that may help.

    Working faster won’t solve the problem
    It would be nice if we could fix outages faster simply by working faster, but we all know this isn’t true. To make sustainable, measurable improvements to your MTTR, you need to do a deep investigation into what happens during an outage. True – there will always be variability in your resolution time due to the complexity of incidents. But taking a look at your processes is a good place to start – often the key to shaving minutes lies in how your people and systems work together.

    Check out your RESPONSE time
    Some call it MTTA (mean time to acknowledge) or MTTR (same acronym, but meaning mean time to respond), but the clock starts ticking as soon as an incident is triggered, and with adjustments to your notification processes, you may be able to achieve some quick wins.

    If your response time is on the longer side, you may want to look at how the team is being alerted. Do alerts reliably reach the right person? If the first person notified does not respond, can the alerts automatically be escalated, and how much time do you really need to wait before moving on? Setting the right expectations and goals around response time can help ensure that all team members are responding to their alerts as quickly as possible.

    Establish a process for outages
    An outage is a stressful time, and it’s not when you want to be figuring out how you respond to incidents. Establish a process (even if it’s not perfect at first) so everyone knows what to do. When you’re designing your process for responding to incidents, make sure you have the following elements in place:

    1. Establish a communication protocol – if the incident is something more than one person needs to work on, make sure everyone understands where they need to be. Conference calls or Google Hangouts are a good idea, or a room in HipChat or Slack.
    2. Establish a leader – this is the person who will be directing the work of the team in resolving the outage. They will be taking notes and giving orders. If the rest of the team disagrees, the leader can be voted out, but another leader should be established immediately.
    3. Take great notes – about everything that’s happening during the outage. These notes will be a helpful reference when you look back during the post mortem. At PagerDuty, some of our call leaders like using a paper notebook beside their laptop as a visual reminder that they should be recording everything.
    4. Practice makes perfect – if you’re not having frequent outages practice your incident response plan monthly to make sure the team is well-versed. Also, don’t forget to train new-hires on the process.

    To learn more, check out this talk about incident management from Blake Gentry, a former lead software engineer at Heroku.

    Find and fix the problem
    Finding out what is  actually going wrong is often the lion’s share of your resolution time. It’s critical to have instrumentation and analytics for each of your services, and make sure that information helps you identify what’s going wrong. For problems that are somewhat common and well understood, you may be able to implement automated fixes.