devopsy.kinsta.cloud

Author: Will Button

  • Creating Dev Environments With Docker Compose

    Creating Dev Environments With Docker Compose

    down-the-rabbit-holeThe Slack notification popped up on my phone at 2pm Friday afternoon. “Argh. This is never good” I thought to myself. The message said “do you know anything about Vagrant?”. It was from a client of mine and I knew their environment well enough to know that this question was going to go down a rabbit hole the likes of which would leave Lewis Carroll awestruck. To compound matters, I was in the middle of nowhere, halfway between Las Vegas and Phoenix. I found a place to pull off, overlooking a vast, expansive maze of canyons and riverbeds leading nowhere and everywhere at once. I thought to myself “how fitting” as I started to reply on my phone.
    When launching a startup, there is no rule other than “find out what the
    customer is willing to pay for, then build it”. My client, Roger, had nailed it. His next message confirmed this as I read “I’ve been beating my head against a wall for two days trying to get this Vagrant image running. I’ve got new developers starting next week and need to get them a dev environment setup”. Roger had just discovered startup rule #2: “nothing can prepare you for the panic of success”. craft beerHis current environment was very solid: running in AWS, everything was deployed via OpsWorks to both staging and production environments. It works great… if you’re the only one in staging. It wasn’t going to support multiple developers changing code simultaneously, so he was looking at Vagrant for a solution. Each developer needed mongodb, elasticsearch, redis, and two node.js servers: one for the api and one for the web ui. Vagrant images are great, but I’ve often found them to be most valuable when hand-curated, like craft beer. They are also difficult to share as it requires shuttling large vm images around between developers.
    After reviewing the error logs, I could tell he was facing issues with security permissions between the vagrant image and the host OS. I decided I wanted to try something different. I wanted to try building it out using Docker containers. I had a feeling I could accomplish the same results with Docker, plus the added benefits of the entire environment being stored as config files in a git repo. Small, concise, version controlled, portable… I was liking this idea more and more. I sold him on the idea as well as buying enough time to get back to civilization where I could work on the idea (though the overlooking view was pretty inspirational).
    Scenic Overlook
    Back at the office, I broke open my laptop. I was familiar enough with Docker, but had a suspicion that Compose would help me tie the disparate pieces together. I started with the node.js servers. It was easy enough to kickstart them using a Dockerfile. (NOTE: I could have done this in compose as well, but had a feeling that I would want some extra flexibility later so I created separate Dockerfiles for them.)
    FROM: node:latest
EXPOSE 3000
    Next, I created the docker-compose.yml file to define the environment. I was able to define all of the containers (minus my previously created node.js servers) using images from Docker Hub.
    web:
  build: ./web
  ports:
    - “3000:3000"
  volumes:
    - ./code/web:/srv/www
  command: node /srv/www/server.js

api:
  build: ./api
  ports:
    - "3000:3000"
  volumes:
    - ./code/api:/srv/www
  command: node /srv/www/server.js

redis:
  image: redis

mongodb:
  image: tutum/mongodb
  ports:
    - "27017:27017"
  environment:
    MONGODB_PASS: password

elasticsearch:
image: elasticsearch
ports:
- "9200:9200"
- "9300:9300"
    Once this was done, I created links in the web container. This creates environment variables and hostfile entries in the web container, allowing the web container to access the elasticsearch container via the hostname “elasticsearch”, and the mongodb container via the hostname “mongodb”, etc…
      links:
    - api
    - redis
    - mongodb
    - elasticsearch
    Docker Compose links
    The environment was complete, and I brought it up with the “docker-compose up” command. The added benefit of this environment is that the docker-compose terminal window maintains a steady stream of logs from all of the containers in a central window. This allows the developer to view the logs realtime, without having to hunt for them.
    Docker container logs
    VirtualBox portsThe only real stumbling block was in the use of VirtualBox. I created the environment on OS X. Since OS X can’t run Docker natively, it all runs in a VirtualBox machine on my workstation. This meant that I needed to go into VirtualBox and setup port forwarding for the required ports (3000, 3030, 27017, 9200, 9300,6379) so that I can access the services locally.
    Including time spent learning the basics of Compose, the entire environment was up in 2 hours. By using Docker, Roger’s new developers will be able to clone the config from a repo and be up and running in minutes regardless of whether they are developing on OS X, Windows, or Linux. Future additions to the repo can be shared by all by continuing to update the repo.
    Interested in trying it out for yourself? All the code described can be found in my git repo at https://github.com/rekibnikufesin/docker-compose. Be sure to let me know your thoughts in the comments or via twitter @wfbutton!
  • Rolling Updates with Rundeck

    Rolling Updates with Rundeck

    CAUTION: If you haven’t used Rundeck yet, clear your calendar for next Valentine’s Day because you’re about to fall in love… How many times have you written a shell script that runs via cron, or a PowerShell script that lives in the Windows Task Scheduler? How many times have you struggled to remember which server you built that script on? Let’s take it one step further: when was the last time that job ran successfully? Using rundeck makes you click-your-heels happy!
    I’ve used multiple solutions to address this problem in the past: jump boxes, dedicated scripting boxes, puppet manifests, the list goes on- but each left me feeling like I could have done something better. Recently, I discovered Rundeck and have been jump-in-the-air-click-my-heels-happy with it.
    What is Rundeck?
    Rundeck is a web application server that allows you to define jobs and tasks. It’s that simple. Where it excels is in its ability to automate those jobs, hand those jobs off to other people to run (without giving them access to the servers), call those jobs via API, monitor the progress and status of those jobs, and log all of the above for auditing purposes.
    Some common use cases I’ve personally used Rundeck for include:
    • deploying EC2 instances in AWS
    • creating scheduled jobs to run on remote servers
    • implementing blue/green deployments from a centralized location
    • rolling updates to ensure only one server is being updated at a time
    I’m sure that gets your mind going with all kinds of ideas, so let’s not waste any more time and get right to
    Setting up Rundeck
    Rundeck is a java web application and as such, requires your host system to have java installed. Java 1.7 is the minimum, and either Open JDK or Sun/Oracle Java will work just fine. If you have a system that supports RPM or Debian packages, installation can’t get any easier as both package formats are available and not only handle the installation process, but shell tools and init.d scripts for startup and shutdown as well. Users of other Operating Systems can use the Launcher for running the app, but installation details are left up to you. Finally, Rundeck can also install as a war file for use in an external container like Tomcat. With the server up and running, it’s time to:
    Build our first project:
    Rundeck_create_projectWhen getting started with Rundeck, I was expecting a lot more configuration prior to building a project. I actually struggled a bit when I couldn’t find out where to plug-in my AWS details at a system level. Turns out, you can’t because it’s configured on a project-by-project basis. This actually works out well, allowing you to isolate which jobs can be run against which servers, preventing mishaps such as running a script intended for your public-facing web servers on your finance servers!
    Creating a new project is done by clicking the big, green “New Project” button on the main screen. On the “Create New Project” screen, filling out the Project Name and Description details are pretty straight forward. The Resource Model Source is where things really start happening. In Rundeck, the Resource Model Source is how you specify the list of servers for the project. In its simplest form, Rundeck receives the list of nodes in XML or YAML format and can receive this data from a variety of sources ranging from hand-crafted files, CMDB, hosted virtual machine service, or a number of available plugins. More information about the Resource Model Source can be found in the Rundeck documentation here. If you are using AWS, the Rundeck EC2 Nodes Plugin allows you to specify nodes (a.k.a. ec2 instance) by tags. Installation is as easy as installing the plugin from the github repo into the $RDECK_BASE/libext directory (/var/lib/rundeck/ if you installed from rpm on RedHat based systems).
    You will want to create a new set of access keys for Rundeck. This will allow you to control what permissions Rundeck has in AWS via IAM policies, and make it easier to rotate keys as the keys will only be used in one place. Using the “Filter Params” textbox allows you to filter the AWS nodes to match your criteria. For example, I have all of my node.js servers tagged in AWS with a tag named “app” and the value set to “nodejs”.
    EC2_tagging
    Mapping Params is another area worth noting. Use this field to adjust the mapping of the values returned by AWS to the fields used in Rundeck. For example, this is a mapping I use:
    hostname.selector=privateIpAddress,privateDnsName;environment.selector=tags/environment;username.default=ec2-user
    First I have the hostname.selector field set to “privateIpAddress,privateDnsName”. This allows me to use either the IP address or DNS name of the instance when selecting nodes. It also uses the same data for displaying which nodes a given project will run on (much more insightful than seeing a list of instance IDs!).
    Next I have an “environment.selector” set to “tags/environment”. As you can see in the screenshot above, I use a tag named “environment” and set values to “prod”, “staging”, and “dev”. This allows me to further filter jobs by selecting the environment from a selector at runtime to determine which environment the job will run on.
    Finally, the username is set to “ec2-user”. This tells Rundeck who it should log in as when opening an ssh session to the node.
    If you are using ssh as the default node executor and file copier, you’ll need to upload keys with appropriate permissions as well. With all of that done (it seems like a lot- but it packs a lot of power in those steps, and it only needs to be done once for the project), it is finally time to:
    Create A Job:
    Creating a job is much simpler than configuring the project. Most of the options on this screen are pretty straightforward. I’ve had so much success with the “Script” step, I find it is my go-to favorite. It allows you to define a script and run it directly on the host. For example, I need to restart node.js on all my servers but don’t want to take them all down at the same time. To accomplish this, I would use a Bash script similar to the following:
    (A couple of notes here:
    – I’m using pm2 to manage the node process, so that is the application the script targets
    – My node.js servers have a /ping endpoint that returns OK when the server is up and running)
    # Shutdown the local node
    echo “Shutting down node.js on $(hostname)”
    /usr/bin/pm2 kill
    sleep 15
    # Start node.js
    echo “Starting node.js on $(hostname)”
    /usr/bin/pm2 start /etc/pm2/ecosystem.json
    sleep 30
    # Wait for the server to come online
    echo “Waiting for a positive response”
    while true; do
        STATUS=$(curl -s http://localhost:3000/ping)
        echo “$(date): $STATUS”
        if [ “$STATUS” = “OK” ]; then
            break
        fi
        sleep 10
    done
    echo “Done with $(hostname)”
    Boom. This script will now sequentially roll through all of my servers tagged “nodejs” in AWS, terminate the node process (via pm2), and wait for the server to come back online before moving to the next server. If the node doesn’t come back online, the Rundeck script for that node will not continue, preventing an unidentified error from taking all of my node.js servers offline! The key to this is ensuring that the Workflow Strategy is “Node-oriented”. This tells Rundeck to perform all steps on a given node before moving on, fully allowing the job to fail if something goes wrong. It’s also noteworthy to point out that Rundeck can email the results to you as well as logging all job executions. With our job in place and ready for action, it’s time to do what all DevOps people do best:
    Sit back and troll hacker news:
    hacker_news_logoSeriously. Rundeck can be an integral part of your DevOps toolkit. Using it for rolling updates, patching, and deployments are just the start. Imagine taking it a step further so that it integrates into your monitoring system and executes jobs as a result of alert conditions, or use it as a stand-in replacement for a complete DIY provisioning system for AWS OpsWorks. With a little work upfront, you may never have to minimize hacker news again!