devopsy.kinsta.cloud

Category: Doin’ DevOps

  • Continuous Documentation

    Continuous Documentation

    Is documentation dead in DevOps? Yes and no. It is transformed, it is automatic, and it is actionable.

    (more…)

  • A Puzzle – Gradle or Maven

    A Puzzle – Gradle or Maven

    Build Tool

    Gradle vs Maven (or Both)

    The world of build tools evolved from simple compilation (gcc) to make files, then evolution of ant(+ivy) and Maven, now the latest build tool is Gradle.

    It’s always a puzzle which build tool to choose – Maven, Gradle or both. Sometime it’s first time project setup or it could be migration from existing tool, but it’s always a debate to choose one or the other.

    There have been lot of talks and discussions on Gradle, Maven (and/or Gradle vs maven). There is buzz around that Android/Google adopted Gradle as its standard build tool.

    Here, I would like to explore the possibilities of adopting Gradle or Maven (Or both), Intentionally i am keeping ant+ivy out of this talk.

    Lets take a look at the details on why Android may have adopted Gradle

    Key points for Google to go ahead with Gradle for their android development are:

    • Create multiple APKs for your app with different features using the same project.(create several variants of an application)
    • Reuse code and resources.
    • Customize, configure, and extend the build process.
    • Customized integration with their own IDE Android Studio.
    • Single build tools to support multiple languages.

    They had some special requirements for their diverse eco system, they worked with gradleware development team to include customized feature for the android application. They even have customized IDE for android developers build around Gradle.

    What we may be looking from build tools

    • How easy is the learning curve?
    • How fast are different builds with each tool?
    • How complex is it to create and maintain the build script?
    • How many plugins exist and how simple is it to customize your own plugins?
    • How good is the community and documentation for each tool?
    • How well does each tool integrate with developer tools? (IDE, App Server, CI server).

    Lets take a look at Maven and Gradle build tools on basis of above points and give them score as per our use case.

    Here is the Tabular comparison for each point between Maven and Gradle.

    Points Maven Gradle Score
    How easy is the initial learning curve 1) Maven is XML based tool, XML is very commonly used/known.2) If existing project are using maven then developers are comfortable with the system. 1) Gradle is DSL based system and need to learn explicitly.2) Developers need to learn new system from beginning if they are not familier Maven – 8/10Gradle – 5/10
    How fast are different builds with each tool Taken a report from zeroturnaround, they have done detail analysis of the speed of builds with both built tools and found maven and gradle are close enough on build timings. Maven – 8/10Gradle – 9/10
    How complex is it to create and maintain the build script? Maven build scripts are xml based, which has predefined structure and only one way to write.So it makes it more standard and less flexible. Gradle has its own DSL which is introduced by Gradleitself and tightly connected to Gradle internals But its flexible and simple and short Maven – 5/10Gradle – 7/10
    How many plugins exist and how simple is it to customize your own plugins? Maven is called “plugin execution framework” Hundreds of plugins exist for Maven and you can create your own plugin is simple. Gradle’s architecture is also plugin-based, It’s easy to write plugins But availability of plugin in community may not be ease. Maven – 8/10Gradle – 6/10
    How good is the community and documentation for each tool? Maven is in the market for very long time, Documentation is good, Lots of resources and help available in open community and forum. Gradle is very recent. It is open source but still under control of gradleware. They have option for commercial support. Maven – 9/10Gradle – 8/10
    How well does each tool integrate with developer tools? (IDE, App Server, CI server). With many years of background Maven has full support to almost each tool and every category (IDE, App Server, CI). Lacking in the App Server and CI ServerCategories, mainly due to newness. Maven – 9/10Gradle – 5/10

     

    (Note – I haven’t given any tool 10/10 because there is always room for improvement).

    My scoring is based on following fact.

    • One is already using maven for all their projects.

    Total Score

    Maven – 47/60

    Gradle – 40/60

    One can decide the score on the basis of their user case and requirement.

    Choices

    • Gradle

    You may not be using other features which are used by Google for android development as those features may or may not match with your needs, however Gradle is essentially emerging as a popular tool and you may definitely want to have a look at it.


    • Maven

    Maven is widely used and has robust defined workflow around it. Developers are very familiar with the system. However this may not be a viable option as android development needs to use Gradle.


    • Maven and Gradle (both)

    This would be the one option for a company with a wide range of development options. It may give developers flexibility to adopt tools as per their choice with a given workflow.

    Reference #1

    References #2

  • Chef best practices

    Chef best practices

    Chef Best practices 

    This article is the II part covering Chef. Please refer to Automation, Provisioning & Configuration Management (CHEF) for an introduction.

    As a matter of fact, Chef is truly an astounding tool. It keeps the cooking metaphor actively going.

    Following is the list of best practices that I would like to cover in the article:

    • Project creation and “readme driven development”
    • A sane build system with Rake
    • Continuous integration with Travis
    • TDD with Chefspec
    • Workflow with Spork
    • Verification with Minitest

    Getting Started:

    1). First and prime important step is to create a new Project. In this section everything should be source controlled. Here in the example a Github repo has been created for all the codes and for the verification or to see step to step process of the development as well as mistakes, they can be easily viewed from the commit history. Follow all the steps and do same for your project:

    git init

    Second objective is to perform a Source Code Management. By running the following command you can make use of git flow to manage your features by selecting all the default settings and if you aren’t using it, then its time you should start working on it.

    git flow init

    Using Readme Driven Development is top best practice that can be started by README.md file. This practice is really important as it helps you to focus your mind and cause you think like a user.

    Here Rake is being used for an already build system. Following is the list of tasks that are to be considered:

    • Rake check: This is responsible for checking all other dependencies.
    • Rake build: The main aim of rake build is to manage and run the test suites.
    • Rake start: It acts as an envelope for vagrant up.

    2). Another best practice is to Set up a Ruby version for the project. This is done by creating two version files: .rvmrc and .rbenv.

    After Ruby files have been created, next step is to create a Gemfile. The main aim of creating this file is to set up all the project dependencies.

    Below is the example with the commands that will help you in creating a Gemfile:

    source "http://rubygems.org"

    ruby ‘1.9.3’

    gem ‘rake’, ‘10.0.4’

    After creation, installation of these dependencies is the essential key step which is possible with below mentioned instruction:

    bundle install

    Creating a new Rakefile is also necessary. It contains a placeholder task per those that have already been defined in our Readme file.

    By making use of Rake for setting up a build system helps in automating some common tasks.

    3). Continuous Integration with Travis is another elite practice.

    But before we proceed further, it’s important to create a link between the repository and the Travis. It is an essential step in the cases where users have already created a repository for the code as part of the series.

    After link is made, you can configure Travis by creating a new Travis file i.e; .travis.yml in the root of the project directory.

    Configuring Travis helps in eliminating development gem dependencies so as to pace things up and tests on the CI server. Furthermore, it also tests the master as well as the development branches.

    Updating your Gemfile is also important in order to ensure that any future development related Gems are correctly excluded. Add this to your Gemfile:

    group :development do

    end

    After Travis is configured, you can update your README.md by adding the Travis notifications to indicate the status of the build.

    4). Another best practice is to perform TDD using Chefspec to “spec” out the cookbook.

    First step is to add chefspec to your Gemfile by updating the file and by all means also do bundle update.

    After updating is done you can Automate Chefspec Cookbook Creation. As per the Chefspec documentation, you are capable of creating a cookbook and on the other hand use another command to create the spec directory.

    According to user or your varying needs you may even wish to create a new_cookbook.

    For example: To build up a new rake task: new_cookbook below are the instructions to be followed

    desc "Creates a new cookbook."
     task :new_cookbook, :name do |t, args|
 sh "bundle exec knife cookbook create #{args.name}"
 sh "bundle exec knife cookbook create_specs #{args.name}"
 end

    As soon as you generate new cookbook you can also create your sample Cookbook

    rake new_cookbook motd

    At this point you have a new blank cookbook and can run the basic tests with rspec cookbooks. You can even add a rake task and upgrade the build tasks for the usage. At last you can add the spec for the motd cookbook in the motd/spec/default_spec.rb file by creating a Failing Spec. By executing the rake, build should be able to show “this now fails”. Thereafter you are required to implement the cookbook.

    Make your Spec Pass and the templates/default/motd.tail.erb file:

    Finally test driven cookbook is made and as Travis file is already referencing the build tasks, when we push this code up, CI will run this test and verify your work.

    5).Working with Spork is another practice in Chef. This tool helps in collaborative cookbook development. It also helps single developer to control and place things in order when it comes to bumping versions.

    So to test this out, initially we will install knife-spork and will do a small edit to your motd cookbook to use its features. First step is to always update your Gemfile by adding the spork gem to the Gemfile. This is not required by the CI server and is put under the development block:

    Following example exemplify the instruction:

    group :development do
     gem "knife-spork", "~> 1.0.17"
 end

    After updating, next step is to configure your Spork by creating a config/spork-config.yml file to put in the default configuration. Though there is a little issue of having lots of configurations in various places and is consolidated over a period of time.

    Configure your Spork using following instructions:

    mkdir config
     touch spork-config.yml

    Running the rake build task shows that our test has failed and can be set right just by updating the template. The moment you complete your updates and the test pass, we should bump the version number of the cookbook by bundle exec knife spork bump motd patch. And finally you will make use of the upload feature so as to upload it to the chef server “freezing” it at the same time.

    Another great feature of spork is the spork promote command that allows you to constrain a specific version of a cookbook to a specific environment. A use case could be:

    • Cookbook 2.0 is being verified on your development environment.
    • Cookbook 1.9 is in production environment.
    • When things looks good can be rolled out to production.

    6). Last but not the least is to set up the mini test suite to verify your cookbooks on a real server. This is done by placing your files in the right place and adding default to your existing cookbook:

    mkdir -p cookbooks/motd/files/default/tests/minitest
     touch cookbooks/motd/files/default/tests/minitest/default_test.rb

    There is also another option available with you in order to avoid the above mentioned action. You can simply modify the new_cookbook rake task to make this part of the cookbook buildout. Last stage comes with deploying the Cookbook and testing it. This helps you in building up a cookbooks that actually works when used on the servers. They also act as a safety net as they run whenever the server is provisioned.

    Conclusion:

    Above mentioned are the best practices of Chef that will help you in building cookbooks in a true, test-driven way. Implementing all these will help in saving you from all key headaches by keeping your custom cookbook changes in new cookbooks and without problems be able to create targeted pull requests to share back with the community.

    I expect you’ll be smart enough to take away some good practices here.

    Happy Chefing!

  • Balancing Between Impostor, Blowhard, and Burnout

    Balancing Between Impostor, Blowhard, and Burnout

    I’m glad John Willis wrote his post on karojisatsu, or suicide from overwork, and that it has been getting the attention it deserves in the community. If it weren’t for the push behind the article that has kept it surfacing in my feeds the past few days I definitely wouldn’t have realized I needed to stop and write this right now.

    It’s great that this issue has been brought to our attention and the community response has been awesome. I’ve seen a lot of people offer their availability for anyone that needs someone to talk with, but there’s definitely some frustration surrounding what can be done to help without offering useless platitudes, as Gene Kim so eloquently put it.

    I have noticed a cycle I can get caught up in that can leave me in a bad place, but now that I’m aware of it I’ve been better at avoiding it. I wanted to share my experience so that others dealing with any of these problems have something they can try that may help.

    Be Confident, but not a Blowhard
    Christina Xu recently posted her thoughts on blowhard and impostor syndrome. I agree we all have areas we’re comfortable in and shouldn’t take on any work we don’t feel comfortable doing. It doesn’t matter if it’s because we don’t feel we’re qualified to do it yet or if we just aren’t interested in a particular space. Anyone that doesn’t immediately jump head first into a technology isn’t suffering from impostor syndrome. Anyone that does or says they can is probably being a blowhard.

    I can and do move in and out of a lot of different spaces in tech but I definitely know my limits. I never really learned much C++ and have no business writing it. However, if the need came up and it made dollars and sense I would try to take some time to transition into C++. This would take months, not weeks, and it would be a lot of hard work. I would even argue it would be painful, but I’m not scared of hard work if it’s worth it.

    I Allow a Reasonable Amount of Time to Transition
    I have no plans to touch any C++ any time soon. I am presently moving back into JavaScript though which shouldn’t be too bad. I’ve been working in JavaScript since I was in middle school but I haven’t done any deep diving in this space in years. I remember wanting to learn backbone and marionette but the need and the time to transition never lined up. I’ve been working with mostly back end code like services, DAO layers, and build tools the past few years. I’ve used JavaScript sparingly for things it does well, but I haven’t had to think very hard about JavaScript in a while. I really need to gather my self confidence as I dive back into JS, and that means I am at risk of being a bit of a blowhard. Allowing time to transition can keep me from getting too far ahead of myself.

    Asking someone if they “ know” JS is a very vague question that could lead to a lot of misunderstandings these days. I need things like angular, ember, node, and D3 but they’re all pretty new to me. It’s really awesome that there’s all this new stuff to play with and better tools to use. I feel I can do more with less thanks to the changes that have happened in the JavaScript ecosystem since the last time I was here.

    True Impostor Syndrome
    I didn’t always have such reasonable expectations about how fresh a particular knowledge base was in my mind. Years ago I would have dove back into JS head first expecting things to be where I left them and then panicked when I realized the landscape had changed and my skills were rusty. If this realization came late at night after a particularly hard day (never mind week or month) it could leave me in a very bad place. In that moment it’s easy to declare yourself a failure of a human being and vow to never code again. This is true impostor syndrome. At it’s worst it can lead to thoughts of ending one’s own life.

    I Limit My WIP to Account for Adversity
    Impostor syndrome can attack when you’re working in a domain your comfortable in as well. Because there’s so much work available it’s easy to take on too much. When you’re spread too thin, teetering towards burn out and then slip on a banana peel of an issue with no slack in your WIP to accommodate it you can end up in the same pit of despair.

    How I Deal with Burnout
    I’ve come to the conclusion it’s better to be proactive in avoiding these scenarios than reactive in responding to them but we need to do both. Let’s assume you’re burned out from trying to transition too quickly or taking on more work than you should have. These things often lead into each other I’ve come to find. Here’s what I do when I know I’ve bagged my limit.

    Step Away from the Keyboard
    Taking time off seems obvious but we can’t always drop everything and take a week or even a whole day off. There are things I do to mandate several daily breaks during different parts of the week. The key here isn’t to decide if I have time to do them, I just drop what I’m doing and go do them on principle. Whatever I’m putting down should be able wait. This is about establishing a lifestyle change more than just taking a break.

    Exercise
    If you don’t do anything start with daily walks outside or yoga and meditation. I’m better off doing these twice a day for 15 minutes than going to a class once a week. It wasn’t until I started doing these things this regularly that I started to really feel the benefits. Even a stroll around the office building for some fresh air when the heat gets hot can do you a world of good.

    Eat Right
    GIGO in it’s truest form. Eat healthy stuff and splurge in moderation. My family eats fish and vegetables almost every night because it’s healthy, quick, and relatively inexpensive (if we buy the vacuum sealed frozen fillets). I’ve been working on drinking more water and less coffee. We do have a larger meal filled with red meat and chocolate desserts every couple of weeks at least. I believe any food that’s good for your soul is soul food but it’s usually not very healthy from a nutritional perspective. You should definitely eat it in moderation anyway.

    Go to Sleep!
    I used to get home and poke at work in front of the TV until late at night with the DVR spewing garbage at me. I got rid of the DVR and only watch shows as it’s convenient. I get to bed by 10pm more frequently and feel like I’m making better use of my evenings. I have trouble sleeping more than 6 hours at a time, but I try to leave room for 8 in case I need it and I definitely take it when I do.

    Find a Computer-Free Hobby
    I happen to be very passionate about BBQ. I try to smoke some brisket or pork at least once a month. I also play a handful of instruments very casually as I don’t want it to turn into more work. Band practice and playing gigs takes a lot of energy and would not result in the desired effect in my particular scenario.

    Volunteer
    Nothing will make you feel like you matter in this world than giving back and being of service. I coach my kids’ soccer team but the sincerity of the “thanks Coach!” I get from some of the kids and their parents feels really good. I completely wasn’t expecting it but I now understand why some coaches stay in the program long after their kids have grown up and moved away.

    There are a lot of people eager to learn to program that would love your help. Look in your community and see who you can help with HTML and CSS. It doesn’t have to be children either. A lot of adults are look to refresh these skills or are trying to move into technology from another field. They would love a mentor they can learn from and you’ll end up being better at what you’re teaching too.

    They also need help at food banks, animal shelters, and many other places. Find something you care about and try to get involved. Trust me, you’ll feel better about life.

    No Burn Out BBQ and Free Cookies
    I am also here for you if you need me. Specifically, if you’re in the Atlanta area and need someone to talk to hit me up. You could live here or be through on business but I will absolutely break out and come meet you if it’s at all possible. I feel the face to face interaction is needed for me to really connect with someone and give them my undivided attention.

    On that note, I’m interested in starting a No Burn Out BBQ to provide an opportunity to do all of these things in Atlanta. Having a BBQ one weekend a month would get us away from the keyboards and outside into the fresh air. It takes a lot of volunteers to get the meal together, and it would provide a safe place to come talk about burn out on top of serving as a monthly reminder to unplug and decompress a bit.

    I have a side project with my kids’ that involves sending people cookies and a drawing for filling out a survey. I’ve opened a survey to gauge the level of interest in a No Burn Out BBQ. Feel free to answer this even if you don’t live in Atlanta. If I get a large enough response from a community we’ll figure out how to hold a BBQ there too. There’s also a fair chance that some cookies and a picture in the mail will do you more good than you realize.

    Please don’t be shy.  You’re in a community that really does care and wants to help. If you prefer something anonymous the Suicide Prevention Life Line is available. Even if you’re not at risk and just want to talk to somebody you can reach out to them 24 hours a day, 7 days a week: 1-800-273-TALK

  • “I want to do Continuous Deployment”

    “I want to do Continuous Deployment”

    “We want to do continuous deployment” – i’ve heard this many times, but have yet to find a scenario where it actually fit. Why? Because you are confused. And rightly so. How many two letter acronyms can be thrown at you in one movement. DevOps takes the confusion further and has two of the acronyms identical, but meaning very different things. The CDs, which are not the round shiny discs in this case. This straight forward post hopefully helps clear up some of the confusion.

    (more…)

  • Repex – Regex Management Simplified

    Repex – Regex Management Simplified

    Our Versioning Complexity

    Cloudify is a Python shop.

    Our REST service is Python.

    Our Workflow Engine is Python.

    Our Plugins are Python.

    We have different version formats and different dependencies across different types of files which need to be changed when a version is updated.

    This is complex, to say the least. We didn’t want to manage this manually having more than 60 repos to take care of when handling version updates.

    Why not sed?

    • sed is a bitch, isn’t it? It’s unmaintainable and unmanageable. While this is solely my opinion, handling hundreds of files, each change in a different context without any safety net, makes sed something you should walk away from.
    • sed isn’t configurable.

    Why not Jinja?

    Using Jinja templates, while safe, was out the windows pretty much instantly as we needed the repos to be usable out of the box or we’d be forcing users to change templates so that they can use our product.

    The Solution

    We turned to Regex. Yes, there is the regex problem. You’re probably familiar with the famous quote:

    –Some people, when confronted with a problem, think “I know, I’ll use regular expressions.” Now they have two problems.

    While this can be true, it also depends on how you use regex.

    I believe we’ve done this well.

    Repex

    We developed repex.

    repex allows you to specify the structure of changes you need to perform in YAML format:

    variables:
  replace: 3.1.0-m2
  version: 3.1.0-m4

paths:
  - description: demo thing
    path: repex/tests/resources/mock_VERSION
    match: '"version": "3.1.0-m2"'
    replace: "{{ .replace }}"
    with: "{{ .version }}"
    to_file: repex/tests/resources/mock_VERSION.test
    validate_before: true
    must_include:
      - date
      - commit
      - version
  - description: handles core modules version in all core repositories
    type: setup.py
    path: cloudify-.*
    excluded:
      - my/excluded/path
    match: version=('|")(\d+)(\.\d+){1,2}(dev|(\w+\d+)?)('|")
    replace: (\d+)(\.\d+){1,2}(dev|(\w+\d+)?)
    with: "{{ .python_core_version }}"

     

    You define a list of paths you’d like to iterate over, provide it with what you’d like to search for, what you’d like to replace, what you’d like to replace with, and some more parameters and off you go.

    For example, the first object in the above configuration will:

    • Look for a file in the path repex/tests/resources/mock_VERSION (regex)
    • In the file, look for `”version”: “3.1.0-m2″` (regex)
    • Attempt to replace 3.1.0-m2 with 3.1.0-m3 only if the string was found during the matching stage and if the strings `date`, `commit` and `version` were found in the file.
    • Write to the output file repex/tests/resources/mock_VERSION.test

    The second object will:

    • Look for `setup.py`(regex) files under `cloudify-.*` (regex) and not under `my/excluded/path`.
    • Look for `version=(‘|”)(\d+)(\.\d+){1,2}(dev|(\w+\d+)?)(‘|”)`(regex) in the files found.
    • Replace `(‘|”)(\d+)(\.\d+){1,2}(dev|(\w+\d+)?)(‘|”)`(regex) with a variable which is not harcoded and therefore must be provided using the API.

    Layers of protection

    repex attempts to provide layers of protection and comfort:

    Safe!

    • You can first `match` whatever you want to match so that you’re sure you’re only replacing something in the exact context you wish to address. If `validate_before` is true, replacement will only occur if one or more matches were found.
    • You can verify that only files containing very specific strings will be addressed by listing them under `must_include`.
    • You can exclude files or directories you don’t want to search in by listing them under `excluded`.

    Don’t take this the wrong way though – regex is regex. You make a mistake, you pay for it. All repex grants you is the ability to have everything organized in one place, with some layers of protection to keep you from making some trivial mistakes.

    Comfortable

    You can declare every single directory or file you’d like to address in one manageable YAML.

    Of course, we needed to create some kind of naming convention so that we don’t have to configure each and every file separately.

    Accepts Variables, both hardcoded and via an API

    One of repex’s strongest features is that it supports using variables for string replacement. The above example contains a {{ .version }} variable.

    The API allows you to provide a dict of variables which will be placed in placeholders. You can either hardcode a variable in the variables section or use the API to send a dict of variables to be used. If a variable is hardcoded in the YAML and you use the API to send a variable with the same name, it will overwrite the hardcoded one. In addition, validation will occur upon variable replacement to make sure it was replaced.

    Functional API

    The API provides three basic functions. I’d rather not rewrite repex’s documentation.. so just refer to this.

    Basically, the three functions allow you to decide the level of granularity in which you’d like to handle the changes. It will allow you to perform actions in different stages of the replacement process.

    For example, we don’t really use the highest level of the API which just iterates over all objects in the YAML. We wrote a Cloudify specific wrapper which can be used as a reference implementation. It enables validation after every replacement. So, for instance, after every blueprint specific replacement, we run a validation that the blueprint is valid vs. our DSL-parser.

    Repex is available on PyPI. You can install it by running:

    pip install repex

    We might, at some point, provide a CLI for repex. Also a logging feature might be implemented to allow for JSON formatted log messages to be generated so that they can be transported for analysis.

    After using repex throughout our entire build process for the last five months or so, it has proven to be extremely useful. We can now follow our entire version update process and easily identify problems by looking at one YAML file and one execution log file. We’re planning on using repex to replace additional types of data (not only versions).

    We would appreciate feedback and, even better, pull requests.

    Hope you find this useful.

  • Technology is All About the Business

    Technology is All About the Business

    In 2011 I joined a company as their fifth employee. They had been in business a little over a year. They had a product. They had customers. Oh, and the other four people? They were rockstars.

    Customers, however, were unhappy. The system was frequently down. Rebooting a machine was often necessary and lead to poor performance for hours.

    I was hired to fix “the database problem” in a multi-tenant SaaS system. On my first day I was given a tour of the technology by the engineer. The code was immaculate, but the database had some core issues. We had the database in good working order in a couple of weeks. We lost our first customer about two weeks after that.

    Why did this customer leave? Ultimately because we were not working on the problem that mattered. We created a local optima in the database and another in code quality. It turned out that there were more fundamental problems.

    Achieving local optima is a false economy. (1) We needed to think outside our core skill sets. Systems thinking pointed the way.

    The Reaction that Mattered

    We had three months cash left. We had two months until our make-or-break event. The business closed a sale for two months operating capital! The customer was a very big brand whose needs would stress the existing software beyond its limits.

    At this point the problem that mattered was immediate and clear. How do we guarantee this customer’s success? As with many things, the answer was obvious once we thought of it: We stood up a duplicate of the production software just for them.

    The Patterns we Recognized

    Standing up a second production environment was no easy task. Over 200 bash commands lovingly executed across five hand-crafted cloud servers. It took days to get our bespoke second system stitched together.

    While it was ideal that we landed such a big contract, the truth of the matter was the business was not confident in our product’s technology. For each potential sale after landing the big one, I would get a call, “Are you sure we can handle this next customer?” The answer was already apparent. We, the tech team, were not confident either.

    Changing the Software

    We needed to improve the software so one customer’s activity had no possibility of impacting performance for another. This would localize problems to a single customer and increase the business’ confidence. We needed to do it quick. We needed to do it with a single engineer and DBA.

    Each customer would get their own full copy (share nothing) of the software. Deployment of the software would be reduced to the simplicity of stamping out parts in a factory. With this model we rewrote the system and closed three new customers on it for our big event.

    Moving to this “share nothing” paradigm is likely the single biggest contributor to the company staying in business, thriving and eventually being acquired. The confidence of the sales force increased dramatically. Sales increased dramatically. Our runway moved out past the six month mark.

    Why Share Nothing

    We recognized our system was a B2B offering, not B2C. This a type of pivot know as a business architecture pivot. (2)

    Trying to build all of the infrastructure for a self service system that scales to unknown demand, while really cool, was not the problem we needed to be solving. Yes, there are inefficiencies in such a share nothing system. No, none of them impacted our bottom line.

    For example, we didn’t care that we were wasting compute. The price of the compute was built into the product. The customer was not paying for compute. They were paying for the product and they were willing to pay so much that this cost didn’t matter.

    Powerful Consequences

    We effectively rewrote the original system in two months. This still seems miraculous, and to some extent it is a testament to the amazing quality of the business logic code. There were, however, other consequences to the “share nothing” model that gave us a significant, if temporary, market advantage.

    • We had no need to spend engineering cycles on plumbing such as user management.
    • We had no need for a billing system. Human-time invoicing in a B2B world was plenty effective.
    • We had no massive scale of data or load. We used familiar technologies.
    • When an instance went down, only one customer called. We learned most of these things were training issues.

    For each item above, the need for engineering was eliminated or dramatically decreased. This means the engineer used that time to create new features. More features means more revenue if you are building the right things. Our CEO and founder was a genius; we were building the right things.

    And it Broke

    Sales trusted the system. New customers were coming in. Existing customers were happy. Infrastructure costs decreased.

    It was short lived. The tools and technology used to implement the repeatable, “share nothing” software were not up to the task. The success of sales was creating a new problem, though a much nicer one to have! We needed more systems thinking.

    Did DevOps Fail

    We did not initially choose the right tool. Is this story a DevOps failure? If you subscribe to the idea that process informs tool choice, then the answer is a surprising, “No.” The DevOps thinking was sound, we simply need to choose better tools to support the model we had chosen to implement. Technology is all about the business at the end of the day.

    Part Two of this case study will highlight some of the evolution of the tools and processes we put in place that lead us to further success.

    References

    (1) The Goal – Eliyahu Goldratt, Jeff Cox
    (2) The Lean Start up Eric Ries

  • Yes, I said “Culture” again

    Yes, I said “Culture” again

    “Culture”. Any time that two or more people get together to do anything  —  whether it’s DevOps, or marketing, or just eating lunch  —  they form a culture. Even if they pointedly ignore each other, it’s a kind of culture (just not a very good one). When we the preachers of DevOps say the word “culture” we are simply stating the fact that people are a big factor in DevOps success or failure.

    There’s no point in trying to get away from it, even if you want to.

    Cultures form naturally.The character of a culture is determined just as naturally by the people who compose it, and the conditions under which they operate. If a development team, for example, has the right mix of people, and they’re working in an environment that allows easy, ongoing communication, they’re likely to develop a culture with some very positive characteristics. If some of the key people on the team have personalities that clash, on the other hand, or if there are major barriers to communication, the culture that develops may be seriously weakened or dysfunctional.

    If you are a manager with responsibility for a DevOps team, it will develop a culture, whether you actively participate in it or not. Either way, you’ll have to deal with the results. 

    At this point, you may be thinking, “Wait a minute!” We’re using agile, and we’re fully committed to the process.  We’re also using the best tools available for DevOps implementation and management.  Shouldn’t a strong, all-encompassing process and a set of first-rate tools be able to override or even turn around a weak team culture? The long and the short answer to that question are both the same: You cannot separate process from culture, and you cannot even separate tools from culture.

    Suppose, for example, that your development team is broken up into cliques with a minimum of communication between them, and that your operations team has long since become resigned to having almost no communication with development. You can have as many meetings as you want, but until you break down the cultural walls, they’re probably going to be very frosty and uncommunicative affairs.

    The success of any process depends on the willingness of the people involved to participate; otherwise a process that requires cooperation is going to be blocked, if any of the key participants are unwilling to engage 

    And it’s the same with tools. Tools can create processes for you, but when they do, they will own you, and not the other way around. Even the best tools are useless in the hands of a culture that has no interest in using them. You might, for instance, implement a well-thought-out, integrated set of DevOps tools, but if your individual team members go around singing “I did it my waaaaay!” and refuse to let go of their existing tools and practices, the new tools won’t amount to much. 

    So what can you do? As it turns out, there’s a lot.

    Culture grows from the top down as much as it grows from the bottom up. Managers can and do have a major influence (whether it’s by commission or omission) on the development of a team’s culture.

    One of the best ways to foster communication, for example, is to be the communicator.  Speak honestly to your team members, listen to what they have to say, draw them out (in a friendly and supportive way) when they appear to be holding back, and when necessary, work with individuals to get them to communicate with each other.  If your team members know that’s what they’ll experience every time they go into a meeting, they will become more communicative  —  and it will show up in their level of day-to-day cooperation.

    It’s the same way with tools. The truth is that people frequently become attached to their old tools. So much so, they disappear into their daily activities. They are often unwilling to give them up, even if those tools make their jobs more difficult. Giving up your existing tools can feel a lot like you’re giving up your independence, particularly if you’re being asked to shift over to a much more integrated and collaborative environment.

    This is also part of a team’s culture, and as a manager you can, and often must, do a considerable amount to shift that culture from one that values autonomy above all other things to one that places a premium on cooperative use of tools. Another way to do this is to create shared objectives. Tangible motivators that cross the entire team, not part of a single one.

    Tools won’t make a culture.  A culture makes use of tools.

    So maybe the place to start is with your own understanding of the culture that is already in place. Are you clear on the nature and the current state of your team’s culture?  Do you have a clear understanding of what your team would be like if it were fully committed to DevOps culture?  How near or how far apart are those two visions?

    Start from there, and take your team where they need to go.

    DevOps tools without a DevOps culture does not create a DevOps environment. They are tools with DevOps capabilities waiting for a committed DevOps team to put them to good use.

    If team members understand the value of having a DevOps culture  —  to them as individuals, and to your organization  —  they will be willing to commit to that culture.  But they need to see that value, and in order for them to see it, you as their team manager must have a clear vision.

  • Expert, like Master, can be a Dangerous Word

    Expert, like Master, can be a Dangerous Word

    William Sealy Gosset was a mathematician that published while working at Guinness. The company was concerned about trade secrets being leaked so all brewers had to write under a pseudonym. Given he was publishing strictly for academic purposes, he decided to write under the name “Student”. If you have ever taken a statistics course you’ve probably used his Student’s t-test. I couldn’t find a completely consistent account of why he selected the name, but Student seems to have been chosen in the spirit of continuous learning.

    I often confuse the L in the CALMS model for Learning, but it’s actually Lean. At first glance it may appear that continuous learning and lean practices are in opposition of each other, but in fact continuous learning is an operational expense of implementing and maintaining lean practices along with devOps as a whole.

    The Old Way of Learning

    Up until the 1900s brewing was a trade that operated off of an apprenticeship system. I’ve known a few people in apprenticeship positions and they were treated horribly for the most part. If they swept enough floors and cleaned enough toilets they may graciously be given the privilege of getting tenuously involved with some side work. Often the hazing got pushed to the point that the apprentice quit before they were given any work that wasn’t custodial in nature. Only a couple of times did the expert realize the apprentice left because they weren’t being respected. Most of the time it was chalked up to an inadequacy of some sort.

    There was some of this mentality at Guinness. Newly hired positions were instructed that if they were lucky enough to meet a brewer in the hallways they should not identify them but lower their eyes until they had passed.

    If you’re in an toxic environment you should quit and go work somewhere else if at all possible. I’ve heard it’s easier than you think from many walks of life, but Katherine Daniels (conveniently known as @beerOps) was a recent guest for a podcast on developing careerOps with @SoberBuildEng, @eciramella, @sacha_d, and @SonOfGarr on the The Ship Show. You should check it out for additional information on finding the best fit position and employer for yourself.

    Transitioning to a New Way

    I’m guessing this brewer centric culture at Guinness had caused enough problems for Cecil Guinness to get sick of it. With the assistance of the managing director at the time, a Mr. Christopher Digges La Touche, Guinness started hiring STEM graduates from Cambridge and Oxford and making them brewers after a two year apprenticeship. They had decided to make the brewing of beer scientific.

    The culture and environment sound like they belong to a start up. These newly hired brewers lived together in a house at St. James’ Gate and spent a lot of free time on team outings. A group of really passionate people with similar interests will start to brainstorm and collaborate if they spend enough time together. They started to figure out ways to measure properties of the raw materials used in brewing that had always been qualitative in the apprentice program, such as subtleties in the grains that had to be learned from experience under the tutelage of a master brewer.

    I’ve never brewed any beer and I have no idea what a brewer would mean by the rub of the hops or the texture of the barley, especially when described as milky or steely. The real problem with subtle nuances that are driven from intuition instead of data is the difficulty in comparing results. I certainly couldn’t explain how one barley was milkier than another barley. By identifying a method of measuring these properties consistently and accurately the brewers were then able to compare different sourcing options empirically instead of off their gut. It also helped the brewery identify how to store their materials until needed while maintaining the quality of the ingredients.

    Op-ex Well Spent

    The discovery process wasn’t cheap or easy. Facilities and laboratories were built. Tours of fields and networking events where brewers could meet and converse with the farmers and maltsters where arranged. Outside of finding the right hops to grow they studied the cost associated with producing them. I suspect it was worth the cost of growing their hops if they could control the quality consistently. Guinness eventually bought enough farms for these young ‘brewers’ to experiment with that Guinness became one of the largest hop growers in Ireland and England.

    Not only was this expensive, but it was culturally difficult. Getting brewers and farmers to change traditional practices that had been handed down over generations is not easy. Especially when your reasoning depends on science and mathematics that aren’t always easily understood.

    Good Experts Collaborate

    Each of these young ‘brewers’ did become a resident expert that could help the other ‘brewers’ when they encountered a problem in a specific area. Not only was Gosset good at listening to their concerns but he made sure he helped them find a best fit solution they understood. He even built tools to help solve their problems when needed, the t-test being the most famous among them.

    A maltster named E.S. Beaven was a bit of a Ron Swanson and had no interest in statistics or statisticians. Gosset was so fascinated with his skill in cross breeding barley though that he eventually overcame the social hurdle of being a mathematician and became close friends with Beaven. Gosset was then able to help him set up several nursery experiments, including a group of experiments where farmers in different areas swapped seed to see how the yield varied with agricultural methods and environment.

    These young ‘brewers’ Gosset was working with were good experts. They were always learning more about brewing by taking measurements and sharing their findings, even if they had to do it anonymously. This was all done in an effort to eliminate waste and improve quality. I’m not sure how focused they were on automating everything, but this was just before the time Henry Ford’s assembly line started rolling. Automation probably came earlier at Guinness. A Brewery, unlike many other factories, can power automation using only gravity instead of steam or electricity. Scaling one up is easy to do compared to something like the Ford factory.

    The First DevOps Engineers

    I see the CALMS model being followed here. I see a systems perspective being embraced, feedback loops being completed, and a culture that understands the connection between experimentation, failure, and eventual improvement. This is consistent with the Three Ways. Was Guinness an early (albeit unaware) adopter of devOps? In my opinion yes. Not only was Guinness transcending industrial engineering by running one of the first devOps incubators in the 1900s, but Gosset and his group of young ‘brewers’ would be more aptly titled DevOps Engineers.

    DevOps requires production. It is a lot like industrial engineering, but the current revival of these practices under the banner of devOps is a result of the rapid changes in technology we’ve seen. Viable applications can be provisioned at scale using these techniques, but this isn’t a job just for software companies any more. If you produce or sell something and aren’t optimizing from a systems perspective, leveraging technology to automate and measure where ever you can, and keeping your employees and customers satisfied in the process then your business could probably be making more money.

    There Are Bad Experts Too

    I have walked through the debate on hiring devOps engineers with a few folks. Being a devOps engineer myself, I obviously think there’s a place for it and it can have merit. Growing a devOps advocate internally can be really difficult because of history, baggage, and biases that are already in place. A devOps engineer, much like Gosset and his buddies, really needs to have a fresh set of eyes on the work flow and where the bottlenecks, waste, and quality concerns are located. Guinness was smart to hire more than one of them though. If you’re going to have experts or masters in your organization you should have more than one expert in any single subject.

    I’ve encountered a lot of resident experts in enterprise sized companies that have become the single source of truth on a particular subject. When brought a problem that needs a solution it’s very easy for them to declare it “as designed” or a job for someone else because it’s not quite specific enough for their level of expertise. If there’s no one to give a second opinion on the matter how can they ever know when they’re wrong?

    I had a boss that used to tell people, “A Master electrician can still change a light bulb” when they complained someone was doing work outside their area. If people are showing up with problems and someone can easily help them they should help them. If your expert isn’t getting their job done then their customers are going to take the demand somewhere it can be met. A good expert should have the awareness to collaborate with someone who can help them manage their workload instead of expecting people to wait or just deal with support issues.

    Which is why Gosset and Beaven could become friends. Beaven was truly a master maltster and an expert in cross breeding barley. Gosset respected this and wanted to learn from him. If Beaven had been mediocre at his craft I doubt Gosset would have cared what he thought of statistics, but when Beaven saw the mathematics was a tool Gosset used to understand and confirm why he was good at cross breeding barley he then became interested in it. A demonstration of utility is the best sales pitch you can make, as seeing is believing.

    Collaboration Makes the Difference

    A good devOps Engineer must become an expert in many things. It helps us help each other, it fosters empathy, it keeps our career options open, and it keeps us honest. If we get bored or frustrated with a job where there’s no traction then we can take our diverse set of skills and do something else. Sometimes this can even be done in the same company. Most companies would rather re-purpose a talented individual if they can move them into a role they can perform with some passion again than see them leave. There’s strength in diversity and it ensures continued survival, even when talking about what you do day to day in the line of your work.

    If you’ve landed in a position of absolute authority you run the risk of degenerating into a bad expert that’s mostly sitting around, punching a clock, and telling people to get lost instead of helping them. It’s not that you’re a bad person, but absolute authority corrupts absolutely and you’re still human. If there’s any question whether you might run this risk yourself then find a colleague you can collaborate with that can help you learn as much as you teach in complementary subjects. Find a Gosset to your Beaven (or a Beaven to your Gosset). Cross training will help manage your work flow but it’s a balancing act between optimizing hand-offs and individual task transition. Still, if you diversify your skill set it will definitely be good for your own self-preservation in your current role and in your next one.

     

    SOURCES/ADDITIONAL READING

    Guinness, Gosset, Fisher, and Small Samples
    Author(s):
    Joan Fisher Box
    Source:
    Statistical Science,
    Vol. 2, No. 1 (Feb., 1987), pp. 45-52
    Published by:
    Institute of Mathematical Statistics
    Stable URL:

     

     

    Guinness’s Brewery in the Irish Economy 1759-1876
    By Patrick Lynch, John Vaizey

  • 6 Common Causes of Conflict DevOps Sets Out to Solve

    6 Common Causes of Conflict DevOps Sets Out to Solve

    It’s a very exciting time to work in Information Technology – as John Willis* put it:

    “DevOps finally proves how IT can be a strategic advantage that allows a business to beat the pants off the competition. This is the moment we’ve all been waiting for.”

    I work with organizations that are palpably excited that the software and applications they are creating deliver groundbreaking, mission-critical capabilities to their end users and plan to outsource all their ‘generic’ IT – the services that don’t offer anything unique to their customers’ experience. The challenge though, is that these organizations have a lot of habits and history to deal with; while the business puts pressure on IT to develop and deliver more innovation faster, the increasing criticality of these applications and the evolved complexity and fragility of the systems they run on results in lots of risk and lots of pain when things don’t go according to plan. Here’s how we see this often playing out in today’s enterprise IT marketplace.

    Herewith, 6 of the most common causes of conflict DevOps sets out to solve that we hear when we’re talking to our customers:

    1. I need a server… now

    This is without a doubt the most common complaint we hear: that operations teams, or those responsible for making infrastructure available to development are blockers, too slow and don’t understand the needs of the business. What we see actually happening is demands for immediate drops of physical and virtual servers from teams who not only have literally just heard of this requirement coming down the line but have a pretty full on schedule of planned work too. Oh, and more than likely unexpected system failures and defects are generating a fair amount of unplanned work too having a direct knock on effect to their ability to respond to these demands and their planned work.

    Recommendations:

    • Culturally something has to happen to ensure everyone knows that they are on the same team and it feels like it. Try involving operations in the development cycles earlier so they have visibility of what’s coming down the line and are bought into the overall goals of the project
    • Make it easier to deliver new infrastructure by automating it, creating or using cloud and virtualization infrastructure
    • Aim for an on demand deployment capability
    • Analyze your teams’ planned/unplanned work proportions and try and figure out ways to eliminate the problems that distract from the goals trying to be achieved – this might be better testing or better monitoring of outages

    2. I want access to the production systems

    This one usually comes about after the developers have reached a tipping point in their frustration in not getting their needs/demands met in what they would consider a timely fashion. Sometimes it indicates that an organisation doesn’t appreciate or hasn’t implemented workable controls in the operations teams. DevOps doesn’t mean throwing all process out of the window and putting live environments at risk – it’s by no means mutually exclusive with methodologies like ITIL but it is about individuals collaborating better to achieve a shared desired outcome.

    Recommendations:

    • Read this article: “DevOps Does Not Equal Developers Managing Production”(link)
    • Involve development in the definition of your change control processes and set up of any systems
    • Set some Service Level Agreements as a team

    3. Something is wrong…whose fault is it?

    One thing DevOps is very clear it’s not about is blame. The “S” in CAMS* is for “Sharing” – that’s sharing ideas, goals, workloads and responsibility – it’s about being innovative, taking calculated risks, protecting against failure and BEING NICE. That means no finger-pointing and no swearing at each other.

    Recommendations:

    • Watch this video from AppDynamics: “The Real DevOps of Silicon Valley”
    • Read more about running a good, healthy retrospective – and practice it. Here’s an example from the UK Government Service Design Manual
    • Consider implementing tooling that will enable you to pre-empt failure or fail smartly. And that will help you identify the root cause fast

    4. So many alerts they are meaningless

    We spend a good proportion of our time working with companies to implement systems for smart failure or that can pre-empt failure as mentioned above. This is one third of our devops utopia triumvirate. The kind of systems that can deliver this capability are the new breed of Application Performance Management solutions like appdynamics. One of the challenges we often find though is that enterprises already have monitoring systems. Lots of them. Monitoring lots of different things. All producing a multitude of alerts. This is when we see alert fatigue. This is not something I believe is, or have indeed witnessed to be, relevant only to the healthcare industry. When someone receives a lot of alerts, of which a very large proportion are meaningless, it becomes highly likely that the critical ones are missed.

    Recommendations:

    • Review and consolidate your existing systems
    • Take some time to really understand what alerts matter and adjust your policies according – and consider this an ongoing task to tweak your systems as your situations change
    • Some DevOps proponents recommend alerting development on system outage immediately – I think this could work, if development are bought into the process and not thinking they are simply being blamed (see 4 above) – sharing the load is definitely a good idea, as long as the process makes overall responsibility clear and forces resolution

    5. I don’t have time to save time

    We hear this one a lot, particularly when we start to talk about business cases for automation tools. The two things driving DevOps (1: the increased rate of innovation and 2: the fragility of today’s evolution of IT infrastructure) together result in very high workloads and a LOT of unplanned work. Throw a recession and budgets that have been squeezed hard year on year on year and what you get is a lot of late nights and a lot of missed targets. DevOps is designed to streamline the throughput of work at a cultural, interaction and application based level – diffusing the pressure building up in many IT departments today.

    Recommendations:

    • Consider ringfencing some budget so invite in an external consultancy to perform an audit or assessment of your current DevOps state – you may feel they won’t tell you anything you don’t already know, but you will have someone documenting your position for you and giving you justification to take the next step in your transformation
    •  Make time – find a senior decision maker who will listen and mandate a project where you can prove something – for example the savings made by automating a piece of a process

    6. Our hero is a bottleneck…

    This one is particularly common when we are talking about release and deployment automation. And if you’ve read “The Phoenix Project” * you’ll recognize Brent as being one of these. What tends to happen is this: one day an engineer decides that building servers is a tedious, manual process and that they could write a quick script to make that bit a bit quicker. The next day, they write another one. And another one. Until, over a period of months or even years a whole bunch of scripts have been written automating lots of parts of the process – I like to call this a ‘script monster’. The problem is that even though the script monster works pretty well most of the time, when it fails it’s virtually impossible for anyone other than its creator to troubleshoot. So when there’s an outage at 3am, you have to call ‘Brent/the creator/Frankenstein’ and he breezes in and saves the day. But what if he isn’t there one day?

    Recommendations:

    • Create an environment embracing autonomy, mastery and purpose – where ideas are heard and acted upon. Often these ‘script monsters’ wouldn’t have been born if Frankenstein had felt he had alternatives to explore
    • Understand motivations: yours and your colleagues’. Some people are particularly motivated by certainty at work and if they have made themselves indispensable by creating something like a script monster, they may feel threatened if someone proposes to take it away from them
    • These heroes are often genuinely some of the most talented members of the team, so if they are resistant to change, talk to them about exciting new projects they can be involved with – make them the master of a shiny new system that will multiply the frequency of releases ten – one-hundred fold by allowing others to push them through an automated process – deployment on demand

    In Summary

    DevOps isn’t voodoo, a silver bullet or automatic. But it is a drive to encourage a different, positive and innovative way of thinking to address a common set of issues affecting the experience of living in, working with and using IT in the world we live in today. Be DevOpstastic.

    What other causes of conflict have you experienced that DevOps can resolve?