devopsy.kinsta.cloud

Tag: code

The Dark Side of Infrastructure as Code

We spend a lot of time extolling the virtues of infrastructure as code. All of it is true. Treating infrastructure as code can add great value in terms of its ability to promote consistent, predictable and repeatable results during the application deployment process. One of the ways in which infrastructure is treated as code is through the use of programmability.

Programmability is an integral component in software defined architectures and a part of the DevOps approach is automation through integration with traditionally development lifecycle focused tools and technologies. APIs, app templates and the ability to programmatically interact with the data path are enabling rapid provisioning of services in the network. That’s important to the business. Improving service velocity in the network is critical to achieving faster time to market and enabling the operational scale necessary to meet growing pressure on networks arising from the growth of applications driven by cloud, mobile and the Internet of Things.

Speeds and feeds of processes are now as important to gain a competitive advantage as speeds and feeds of packets.

Those who see DevOps and SDN as having a strategic impact on their organization view programmability as having even greater importance.

But that does not mean that treating infrastructure as code doesn’t have its dark side. Because adopting a programmatic approach to anything means you get both the benefits and the risks.

I happened to be cogitating on this topic when I stumbled across a post reminding us of the importance of code reviews. The content was so-so, but the premise was spot on: code reviews are a must for those embracing the notion of treating their infrastructure as code.

Why is that (or should it be) an imperative?

Consider some stats from InitialState, “a technology startup helping engineers capture and understand product data. Our cloud-based log visualization-as-a-service platform helps engineers make sense of the exploding number of logs being produced by “smart”, complex products and applications.”

According to their data, developers have to find and fix 100 bugs per 1000 lines of code. And it takes 30 times longer to fix one of those bugs than to write one line of code. Overall, developers end up spending 75% of their time (about 1500 hours per year) debugging code.

Now spend a moment thinking about what that means when you start treating infrastructure as code, writing scripts to automate provisioning and integrating with other systems to orchestrate deployment processes.

I won’t lie, code reviews can be tedious. But the first time someone points out an error, particularly one that could have resulted in a security breach or an undesirable service outage, the pain is more than made up for. In the long run, code reviews are beneficial to eliminating disruptive errors. A SmartBear survey in 2012 found that “70% of organizations indicated that code reviews are very important” and “Those that do code reviews clearly have a greater level of satisfaction in the overall quality of their software – they were twice as likely to indicate that they were very satisfied with their software quality.”

In other words, code reviews are an important part of the software development lifecycle (SDLC). Which means it should be a part of the software deployment lifecycle when it leverages programmability to treat infrastructure as code.

Therefore it is important in the early stages of DevOps (or SDN if you’re going that way) adoption that you put into place processes to check (and perhaps double check) the scripts, templates and integrations you’re using to automate application and service deployment along the entire operational chain – from the app to the network.

October 7, 2014
Keep Calm and Refactor Operations

Developers have been refactoring forever to gain efficiencies and avoid mistakes. Let’s apply more than just theory to ops to do the same.

There’s about a million articles, blogs, tweets and infographics available today that bespeak of DevOps as this or that or the other thing. But one core truth remains: DevOps in part attempts to apply methodologies like agile to operations in order to gain efficiencies and avoid mistakes. We need the efficiencies because the number of apps that need to be deployed, updated, migrated, and maintained is increasing, not decreasing, and there simply isn’t enough time in the day (or during the monthly change window) to get it all done.

We need to eliminate mistakes because they are, according to data from initialstate, very costly. It takes 30x longer to fix one bug (an error) than it does to write one line of code. Anyone who has ever spent time chasing a wild pointer will immediately say that’s too low an estimate, but anecdotally I’m willing to go with that figure. It is an average, after all. To put that in perspective, the same data indicates that developers must find and fix 100 product bugs per 1000 lines of code. That’s a 10% error rate.

I’ll let you ponder for a moment what that means to DevOps and the desire to treat infrastructure as code.

Exactly. You probably want to avoid that kind of error rate as you go about automating and orchestrating provisioning, configuration and general management of all that infrastructure. Just a few errors can effectively negate time gained by automation. So how do you avoid that?

One of the ways to avoid some of that is refactoring.

Refactoring is most commonly used to extract pieces of code (script) that repeat over and over into composable functions (in a procedural language) or methods (in an object-oriented language) or services (in a micro-service or SOA-based system). It’s a primitive form of abstraction that serves to produce more manageable, extensible code bases.

Basically, you’re taking a set of actions that you repeatedly perform and turning them into a reusable, repeatable “thing”. In the case of operations that “thing” is probably a script.

Say, for example, that every time you deploy an application you’re also deploying a load balancing service because, of course, you’ll want it scalable and/or reliable. You always perform the same set of steps to complete this task with only a few changes – usually IP addresses, host names, or other network and app-specific settings.

But you’re doing it manually, or writing a new script every time. Manually making changes are, in most cases, where errors are introduced. Fat fingers are a common cause, especially when you’re not paying close attention because, well, boring! You’ve done this fifty million times and quite frankly, you’re sick to death of the tediousness of it all.

We’ve all done it, don’t pretend you haven’t both thought it was boring and made a mistake. You’re in good company, trust me.

What developers do when this is the case is refactor the common code and reuse it across the system. Those bits that get changed are parameterized and everything else stays the same. Even DBAs do this, they just call the resulting code a stored procedure.

The result of such efforts are self-contained automated tasks that, when used to orchestrate a process, enable composable operations. You string together a set of these scripts and voila! Provisioning complete, minus the fat fingered errors or missed manual steps.

Refactoring code is much more complex than this, of course, as would also be the case within operations, but the general premise here is that the practical application of DevOps includes the process of refactoring operations. It’s about restructuring it into more composable, logical and discrete tasks that can be automated and orchestrated as a means to achieve more manageable and extensible operations. It’s about identifying those steps in a process that are repeatable and can be refactored into a single, shared executable task.

When starting an initiative that applies DevOps it is likely you’re going to have to prove the time you’re taking is worth the investment. One of the ways to do that is by refactoring those operations that are continually used and reused to enable a more consistent, predictable and repeatable provisioning and deployment process. By choosing those steps that are most often repeated and refactoring them into their own composable scripts, you can gain efficiencies as well as reduce errors (and eliminate the time they require to find and fix).

August 18, 2014
DevOps makes coding more social, social coding makes better code

One of the main principles of DevOps is to break down silos. What’s more like a silo than the image of a lone developer, shutting himself off from the world in order to code more and interact with people less? “I’m just here to write code and that’s all I do.” There’s nothing collaborative or DevOps about that scenario and unfortunately, it’s one that is all too frequent within organizations.

Enter social coding. This is a concept that didn’t exist twenty, or even ten, years ago and one that continues to grow in popularity. At its essence, social coding is an approach to software development that puts the focus on formal and informal collaboration. There are many benefits to social coding but the most important is probably the opportunity it presents for mutual growth and learning. Developers are allowed to collaborate in an environment that fosters discussion, creation and sharing.

When talking social coding, you can’t avoid a mention of GitHub. Take everything you love about your favorite social network and then apply those same ideas to a code repository. That’s basically what GitHub did, and continues to do, with much success. You can create a profile, follow or message someone, see what others are doing with their code (or yours) and even go so far as to subscribe to RSS feeds of activity streams. GitHub started to take off a mere five years ago and is now held up as one of the more successful examples of how social coding can be commercialized.

Companies like GitHub and SourceForge are leading the way in revolutionizing how developers commit and share their code but the foundation of social coding still rests with code review in a public forum. As Mark Chu-Carroll puts it…

“If you’re programming and you know that your coworkers are going to look at your code, you program differently. You’ll write code that’s neater, better documented and better organized — because you’ll know that people whose opinions you care about will be looking at your code. Without review, you know that people will look at code eventually. But because it’s not immediate, it doesn’t have that same sense of urgency, and it doesn’t have the same feeling of personal judgement.”

What’s next for social coding? After it’s start in the open source community, this is one idea (like many others) that is heading towards the enterprise market. Companies are beginning to recognize that even internal projects can be managed and treated as open source efforts. From wikis to crowdsourcing, there are many ways that the enterprise market can take advantage of social coding. The end user is the one who stands to benefit the most from such a change.

Dave West, a Principal Analyst at Forrester Research, has this to say about social coding:

“We’re social animals, and the development of stuff tends to be social in nature. We think a better way of thinking about it is as social development. It’s not just about coding; it’s about enabling business change and delivering business value in a social way. It will be interesting to see how companies broaden the idea from social coding to social development. Social coding is a fundamental change.”

April 25, 2014
Programmability in the Network: Stop a Bleeding Heart…

It is not often the case that a security vulnerability can get the entire Internet talking. And not just the security community on the Internet, but everyone. End-users and IT alike are looking for answers and trying to mitigate Heartbleed. It has its own web site and logo. It’s that big of a deal.

Many service providers have already patched their systems, but there are a whole lot of sites on the Internet and it’s estimated that a significant number of them are vulnerable.

Netcraft notes that Heartbleed, based on OpenSSL, “affects around 17% of SSL web servers which use certificates issued by trusted certificate authorities.” One of the most trusted sources of data regarding web server software in use today, Netcraft’s “most recent SSL Survey found that the heartbeat extension was enabled on 17.5% of SSL sites, accounting for around half a million certificates issued by trusted certificate authorities.”

But it’s not just server software that may be vulnerable. Any device in the data path that terminates SSL and is based on OpenSSL could be vulnerable. Any “thing” that might use OpenSSL to enable secure communication and accepts connections could be vulnerable. In other words, hearts are bleeding all over the Internet right now, and it could take quite a while for the Internet in general to plug the holes.

The hole itself is fairly simple – it takes advantage of the mishandling of what seems like an innocuous meta-protocol that runs atop TLS and DTLS that consists of two message types: HeartbeatRequest and HeartbeatResponse. It’s used to “ping” the other party in a secure connection to make sure the peer is alive. It can be used to offset idle times that might otherwise cause the connection to time out.

It can also, thanks to a missing bounds check on the message structure, arbitrarily and completely silently grab 64 kilobytes of the server’s memory. That memory might include sensitive things like the private key used to encrypt communications or passwords or pieces of critical documents.

Whatever might be in memory when the vulnerable software goes to grab it is leaving the building faster than Elvis – and without a trace. There’s no logs, no evidence of the theft, nothing. It’s clean and remotely executable by anyone on the Internet. That may explain the attention it’s currently getting.

The answer to any vulnerability is ultimately to patch it. Depending on how many servers or systems are impacted, that could take some organizations days – assuming patches are available immediately. In the meantime, the heart is still bleeding and the only way to prevent more information from leaking is to shut off the hose.

Except it’s not the only way.

Programmability in the data path can mitigate this, now. Right now. Like in the next five minutes.

Let’s review programmability in the network for a brief moment: you have, in the data path, a device. That device (or software) is capable of programmatically intercepting, inspecting, and transforming the data that’s traversing it. That means such a system can intercept a request to initiate SSL/TLS traffic, inspect it and, if it finds a request to include Heartbeat support it can reject the connection.

Interestingly, one of the primary motivators of early SDN architectures was the ability to support new or experimental protocols by inserting them into the data path in the network. While that’s a driver for academics, a more realistic driver for businesses is the ability to interact with protocols in flight and “fix” them – especially when the “fix” involves preventing the exploitation of a vulnerability.

Programmability in the network enables this kind of rapid response to threats. Being able to deploy a solution very quickly to prevent further data leakage provides protection for both the organization and those who interact with its properties (employees, partners and customers). It also offers time for IT to patch and update all impacted systems without remaining vulnerable or taking the drastic step of going completely dark until the situation is resolved.

By enabling a view of “infrastructure as code”, programmability not only enables a more software-oriented approach to deploying services, but also a software-oriented approach to quickly creating and deploying an appropriate response to emerging threats.

A devops approach can facilitate this capability, and push protection into the network as code.

April 10, 2014
What “Programmable” Means

One of the most often used words these days to describe emerging architectures and technology is (drum roll please) “programmable.”

Now, from my used-to-be-a-developer-and-architect-turned-infrastructure-geek perspective, I know what that means. It means I get to write code. But to what purpose that code is put to is a question that’s rarely answered. So today we’re going to answer it because, well, my used-to-be-a-developer-and-architect-turned-infrastructure-geek persona likes to talk about code.

When you hear the term programmable used in the context of Software-Defined Networks (SDN) they’re talking about the same things that “programmable” means in the context of Devops: programmatic control of both the data and the control path.

The easiest way to visualize the difference between these impressive sounding terms is to recognize that the data path is inside and the control path is outside.

What that means is if you’re interacting with the data path, you are participating in it. You can inspect it, evaluate it, and transform it and, perhaps more importantly, you can make decisions based on data in flight. That’s powerful for the network, because a variety of Devops patterns are based on this very concept and require the network for implementation. Consider, for example, the implementation of an A/B testing pattern. The basic premise of the pattern is that somehow the identity of a user is known and, based on their membership in a defined group (beta testing customers, roles within the organization, volunteers from the community, etc…) they are routed to different versions of the same application to determine how differences in the UI impact behavior.

Notice the term routing, which implies it is a function of the network. The reason programmability is critical is two-fold: one, you must be able to identify the user as belonging to a specific group and two, you must be able to dynamically route based on that information. The former, identification, could ostensibly be accomplished using fixed lists IFF (If and Only If) it is possible to map a user to some variable readily accessible to the network, which traditionally means things like IP and MAC addresses and TCP ports. Most folks will immediately recognize that today this isn’t enough (and actually it hasn’t been enough since the days of the mega-proxy, but I digress). The information that’s needed is much higher in the networking stack at the application layer, aka in the OSI model as Layer 7. Which means you’re probably going to have to dig into HTTP headers and maybe the payload to get the data you want. That requires programmability in the data path, inside.

To really make the pattern scale (and to make it flexible enough to support a phased approach to beta testing) you probably don’t want group membership to be a static, fixed thing. So now you’re going to want to extend your implementation to include making a call to a database or identity store (LDAP, AD, etc…) to get current information. That requires programmability in the data path, inside.

So what programmability in the data path means is the ability to use code to direct the behavior of some “thing” in the network, be it an application proxy or a load balancer or what have you. You can’t easily do this through configuration-based proxies because they don’t allow for the execution of the domain logic necessary. There are a limited number of interactions you can achieve with configuration-based systems because of the logic they’re hard-coded to execute,and most of them don’t include calling out to a data store to look-up arbitrary information.

Data path programmability = Code in Flight. So you now you know where the name of the blog came from. This developer-and-architect-turned-infrastructure-geek finds the intersection between programmability and networking irresistible.

Okay, so that’s data path programmability. What about control path programmability?

Well, that’s the easy part. Control path programmability is on the outside. It’s the APIs that enable automation and orchestration of provisioning and configuration. Control path programmability is how you integrate infrastructure into command control systems like OpenStack, Puppet and Chef. Infrastructure comes with varying degrees of API-enablement, some extremely granular and others less so, depending on the particular function and maturity of the vendor.

So what you can do with the control path depends on the API, but in general it enables things like provisioning new services – think configuring a new virtual server in Apache and all the settings that accompany it – or start up and shut down functions that enable rolling maintenance across a set of devices or servers to avoid downtime.

But all control path programmability is on the outside, enabling integration and management without having to learn a CLI or configuration file syntax.

SDN and Devops use both programmability to reach relatively the same goals: scale of operations, rapid provisioning of services, to decrease the possibility of human error (the fat finger factor) and to introduce flexibility into systems that were previously bound to fixed processing and configuration-based paradigms.

Programmability in the network is going to enable a variety of new patterns, processes and services and as traditional infrastructure becomes more and more programmable through the adoption of architectural approaches like SDN, Devops is going to see more and more opportunity to apply its maturing expertise in leveraging programmability to a much wider audience.

March 24, 2014
Closing the Deltas Between Development and Production

Who hasn’t heard the classic statements “It works on my machine!” or, “Man that would have been easier with real data!”? In either case, had the development happened in production, problems would have been avoided! Fortunately, in a DevOps world, the walls between production and non-production get broken down making the reality of developing and testing in production more tangible. In no particular order, here are some of the things I’ve done to breakdown those walls.

Develop on and deploy from a single code line.

Many times we find ourselves in complex processes around deployments and production bug fixes. For example, the code is branched for release candidates and deployments are done from the branches. Branching before release by itself isn’t complicated, however, when there is a production bug, engineers find themselves repeating or merging the fix into at least two, if not several, different places. Repeat that a few times over and, uh oh, now there is a problem because there are many moving parts and questions about what exactly is in production. Unfortunately, the differences are most likely unintentional.

Instead, simplify the process and have everyone work off the same code line and have that same code line deployed directly to production. Now there is only one code base, therefore no question about what code is running in production. Production isn’t static, so change will still happen after deployment. Notation is required indicating what version was deployed, so you can still reproduce production.

Let testing happen on the production database.

Ok, even I’m not that crazy, but let developers use a copy of the production database. There are two very important aspects to this. First, refresh it often. I suggest daily. I have had very good luck using EBS snapshots to get a copy of the database on a different server, but if your database is small, good old SQL dumps also will work. Second, make it safe. Put some safe guards in place to make sure a change doesn’t accidently get made to the real production database. Keep your ACLs tight, change the passwords and practice proper database user management. Lastly, a bonus outcome is that you are constantly testing your database backups and recovery procedures.

Minimize the steps between development and production.

Each step, especially the manual ones, between the developer’s machine and production is another point where unexpected changes can be introduced. It sounds obvious, but the gates code must pass through before entering production must be purposeful and prove their usefulness. For example, if code must go through a regression, but no regressions are being found, then that step is not useful and just increases the probability of unintended change. I have been a part of a process where code had to be deployed to “staging” where it was then hooked up to the live production database, but was only to be exercised in a read-only manner. After doing that for many, many releases and very rarely finding defects before production, we decided to just get better at deploying to production and eliminated the deployment to staging. In the end, we got faster and with no impacts to quality. Less can be more.

Depending on your situation, database size, regulatory compliance, etc. you might to be able to do all of them, but breaking down those walls will yield better code, fewer bugs, happier engineers and fewer sleepless nights to name a few. So, grab that wrecking ball and break down a few walls.

March 19, 2014
MongoDB Needs and Loves DevOps
DevOps is about more than just running infrastructure smarter, with more automation. It’s a means to engineering agile, robust, and modern software by blending the traditionally siloed areas of administration and development. This means infrastructure as code, and developer concern for operations. It means more than calling sysadmins “DevOps Engineers”; it means operators and developers working together seamlessly. NoSQL in general is a solution that by it’s nature loves DevOps, and it thrives under teams that take a DevOps approach, but it can really hurt you if you do not. MongoDB is no exception to this rule. It has gained enormous traction in the developer community by blending flexibility, features, and performance. Some teams have been seduced by the ease with which they set up MongoDB for development, and neglect to think about capacity planning, tuning, monitoring, and maintenance. Do not let this happen to you! Or you will pay the price at 3am when you are scrambling to figure out why your swell app has ground to a halt. Instead, take the time to learn MongoDB’s needs and capabilities, because it doesn’t just make development easy, it can truly sweeten operations with things like hassle-free replication and full access to administrative tasks via the same API used to perform data access. Let’s have a look at MongoDB from a DevOps perspective.

DevOps Wants:

Clueful administrators.

There isn’t much room in the DevOps movement for system administrators who don’t speak at least one high-level language. I personally find Ruby most appealing to to me for systems programming, but Python is also a fine choice. MongoDB admins should be practiced at performing administrative tasks in both the javascript shell and the programming language of their choice, setting the stage for a rich infrastructure monitoring and automating codebase. Administration best practices are extensively, if not exhaustively, documented in the administration docs, so start there. Pay particular attention to the Operational Strategies section of the manual, which provides a checklist of concerns you should address as you plan your system’s design and deployment.

Developers who know the ramifications of their infrastructure.

The age of throwing applications over the wall to operations is mercifully drawing to a close. Developers must understand the nature of their infrastructure, how to best utilize it, and how it handles failures. MongoDB affords developers the ability to code to infrastructure details without coupling to particular instances, or even hostnames. In this, tags, write concerns, and read preferences are your friends. Code that runs reports (which should be kept away from your production instances) can specify that it will read only from instances tagged as “reporting”. Admins can add instances tagged as ‘reporting’, or swap which instances have that tag, with a few lines of code. Refer to the end of this article for example code.

Infrastructure state to alert cleanly, and for trends to be visualized.

The MongoDB Monitoring manual section provides an overview of the ways to monitor your instances, including 3rd party solutions, both self-hosted and SaaS. MongoDB Management Service is MongoDB, Inc.’s free monitoring, trending, and alerting service, which you should set up from day one to monitor every single instance you rely on.

Infrastructure to be programmable.

Ok, you are monitoring your instances, why not work on automating your infrastructure to respond to faults and load requirements? While MMS is a no-brainer, it’s not going to plug into your infrastructure code. You can roll your own integration using MongoDB’s handy exposure of its internals via database commands, and for replication, the local database. For example, you can check on replica set lag in Python like so:
```
import pymongo
# three mongods running on my laptop
c = pymongo.MongoReplicaSetClient(
    'Kusanagi.local:29017,Kusanagi.local:29018,Kusanagi.local:29019',
    replicaSet='test'
)

rs_status = c.admin.command('replSetGetStatus')

lag_threshold = 120 # up to two minutes of replica lag is ok for this example
primary_optime = [ member['optime'].as_datetime()
    for member in rs_status['members'] if member['state'] == 1 ][0]
secondary_optimes = [ member['optime'].as_datetime()
    for member in rs_status['members'] if member['state'] == 2 ]
for secondary_optime in secondary_optimes:
    if ( (primary_optime - secondary_optime).total_seconds > lag_threshold):
        # do stuff
```
Provisioning MongoDB instances programmatically on Amazon EC2 is easy with the official AMIs, but automating MongoDB infrastructure goes beyond that. Learn to use the database commands to configure replica sets, because you can script joining new instances to a set. If you use sharding, you can update tag ranges for sharded clusters to migrate chunks to shards specifically provisioned to warehouse data.

Disaster recovery to be robust and automated.

There are numerous ways to back up MongoDB, and the one you pick should be appropriate for your deploy. So read through your options in the backups section of the MongoDB Manual. You should probably read that section twice.

To aid in release management.

By being schema-free, MongoDB frees enterprises from the hassles of schema alterations and their attendant maintenance windows. Instead of altering a user table to include a new column for “birthplace”, a new release of your application will just begin to write the user document with that attribute included. Naturally, your user profile rendering will not assume this value is set and throw an exception, it will just not display “birthplace”. This does not excuse you from change management! In a DevOps shop, everything about monitoring that applies to production applies to your integration environment, which I sincerely hope is continuous. Configure all the monitors you’d have in production, and alert developers when metrics go out of range.

Closing Thoughts

In a DevOps culture, admins read code, developers plan infrastructure, and they both attend the same planning meetings. Engineers who work with MongoDB, both in operations and development, should read the WHOLE manual. If your developers read the development section and your admins read the administration section, you’re really only doing Dev and Ops.

Example Code – Tags

Tagging an instance of a replica set looks like this in the MongoDB shell:
```
test:PRIMARY> config = rs.config()
{
    "_id" : "test",
    "version" : 2,
    "members" : [
        {
            "_id" : 0,
            "host" : "Kusanagi.local:29017"
        },
        {
            "_id" : 1,
            "host" : "Kusanagi.local:29018"
        },
        {
            "_id" : 2,
            "host" : "Kusanagi.local:29019"
        }
    ]
}
test:PRIMARY> config.members[0].tags = { "use": "production" }
{ "use" : "production" }
test:PRIMARY> config.members[1].tags = { "use": "reporting" }
{ "use" : "reporting" }
test:PRIMARY> config.members[2].tags = { "use": "production" }
{ "use" : "production" }
test:PRIMARY> rs.reconfig(config)
Fri Mar  7 18:35:25.040 DBClientCursor::init call() failed
Fri Mar  7 18:35:25.042 trying reconnect to 127.0.0.1:29017
Fri Mar  7 18:35:25.042 reconnect 127.0.0.1:29017 ok
reconnected to server after rs command (which is normal)
```
Here’s Ruby code, assuming the same config:
```
require 'mongo'

# I'm running 3 mongods on my laptop
rset = Mongo::MongoReplicaSetClient.new(["Kusanagi.local:29017",
                                        "Kusanagi.local:29018",
                                        "Kusanagi.local:29019"])

# using the local db to get current config
config = rset['local']['system.replset'].find_one() 
admin_db = rset['admin']
config['members'][0]['tags'] = { "use" => "production" }
config['version'] += 1  # mongod avoids reconfig conflicts by insisting on an increasing version value
verify_version = config['version']
begin
    admin_db.command({ 'replSetReconfig' => config })
rescue Mongo::ConnectionFailure # expect your connection to close during this operation
    actual_version = rset['local']['system.replset'].find_one()['version']
    raise unless actual_version == verify_version
end
```
Specifying tag sets only applies to reads from secondaries (since there is only one primary to choose from). In Ruby, it looks like this:
```
rset = Mongo::MongoReplicaSetClient.new(["Kusanagi.local:29017",
                "Kusanagi.local:29018", "Kusanagi.local:29019"])
analytics = rset['analytics']  # say we have an 'analytics' DB
cursor = analytics['page_views'].find({'page' => 'home'},
                                        :read => :secondary,
                                        :tag_sets => {:use => 'reporting'})
```
This query specifies, with the :read option, that only secondaries should answer, and with the :tag_sets option, that only secondaries tagged with 'use': 'production' can be used. If I were to shut down members[0] (Kusanagi.local:29017) while it was primary, and members[1] (Kusanagi.local:29018) became primary, the above query would fail, because as primary it’s no longer eligible to answer queries directed at secondaries, even though member 1 is tagged as use: 'production'. This is exactly what you want, to keep your reporting queries from hosing your production instance. After taking stock of the situation, you might decide to assign 'use': 'reporting' to member 3, and the above query would then again work. Note that the MongoReplicaSetClients created by the Ruby driver do not by default refresh replica set state (as of v1.9.2), they use the config that was present when the connection was established (this is not the case for all drivers, make sure you determine your driver’s behavior). This means that if you reassign tag values, existing connections will not notice the change. If I were to change the tags for member 1 to 'use': 'not reporting', the above query would not fail, it would route the query right to member 1. So if you are using tagging, and Ruby, instantiate your client with the refresh_mode option:
```
rset = Mongo::MongoReplicaSetClient.new(["Kusanagi.local:29017",
            "Kusanagi.local:29018", "Kusanagi.local:29019"],
            :refresh_mode => :sync)
```
See the MongoReplicaSetClient documentation for details. The Python driver handles this by launching a background process to monitor Replica Set health and config changes, but the API does not document any means to configure its polling interval.
March 14, 2014
Hello World

The DevOps Cup o’ Joe blog appears a few times a week with a look around the DevOps blog world, highlighting the authors choices for top blog posts of the period.

Hello World! It’s been many years since I thought it was pretty cool to have a computer print that during my own days of coding back when punch cards were all the rage. Though the technology has changed a lot since then, the business of technology has not changed as much. The quickest way to figure out whether a practice or process in IT has become accepted and here to stay is to find out how many are writing about it…and what they are saying.

That’s what I am going to write about…. I am going to be the nosey neighbor in the DevOps blogs neighborhood, sort of like a Gladys Kravitz. Some of you are probably saying; who is Gladys Kravitz? If you don’t know I will tell you later.

As I look across DevOps blogs landscape I find many simply use their blog as a support tool……not very insightful but still better than those that have no blog at all. Others use their blog as a shill for the marketing department. I will be searching the world of devops blogs for thought leaders. Though I must admit I will also shout out some blogs simply because they grab my attention. The best bloggers transcend the web pages of their blogs and give you a glimpse into their souls and the souls of the companies they work for. They are truly storytellers. Many of these bloggers are already blogging for devopsy.kinsta.cloud, but I am going to highlight others.

Enough with the posturing and let’s get started. My favorite blog yesterday was from the PagerDuty blog. You really need to read this heartfelt story written by the bride of a PagerDuty customer, “I Married an On-Call Engineer”. The post tells a story many on this site can relate to. This is a test to see if devops can survive marriage, or is it marriage survive devops. I will let you decide.

Puppetlabs is seeking talking proposals for PuppetConf 2014 by March 18; http://puppetlabs.com/blog/deadline-nigh-submit-your-puppetconf-puppetcamp-talks . Puppet Labs conference in San Francisco is one of the big devops events and time is short to submit. That is my community service for this edition.

Speaking of devops events, JumpCloud organized a DevOps State of the Union in Boston with some major analysts and writers in early March. The event is covered in a series of blog recaps and worth the read. Pay particular attention to the Three DevOps Insights.

It’s early and there is enough great content here on devopsy.kinsta.cloud, so that is all I am going to give you today. Stay tuned as I will be recapping the best of DevOps blogs a couple of times a week here on DevOps Cup o’ Joe.

BTW, to answer the earlier question Gladys Kravitz is the nosey neighbor of Darren and Samantha Stevens. If your next question is who are they, then you are too young to be reading this without a note from Ops.

If you would like something mentioned here or have something you think I should Shout Out; email parker@devops.com and don’t forget to comment here.

March 13, 2014
Let the Code do the Heavy Lifting

One of the first jobs I had when I started in IT was as the IT administrator for a dot.com startup. IT admin at a smaller company like that was typically an umbrella term used to describe a one-man IT department, which is essentially what I was.

The company was growing exponentially—both in terms of customers and revenue, as well as traffic to our website. What that meant for me was a steady stream of UPS shipments from Dell, and a virtually full-time job of mounting servers on racks in the data center, getting them properly configured, and deploying them to the network.

Dealing with 50 or 100 hundred servers in 2000 was a daunting task, but dealing with hundreds or thousands of virtual servers in 2014 is—for all practical purposes—impossible. That is, it would be all but impossible doing things the old-fashioned way. However, the DevOps revolution changes the rules of the game because it uses code to do the heavy lifting, and automate those tasks which can be automated so the IT admins of today are free to engage in more productive tasks.

If we were still dealing in physical servers, the benefits of automation that you get in a DevOps environment would still not be possible. What allows so much of the traditional “racking and stacking” today to be automated is that we are dealing primarily with virtual servers. Although a virtual server may act like a server, and fulfill the same duties as a physical server, it is essentially just a software program. The fact that a virtual server is comprised of nothing more than code means that virtual servers can be created, configured, and deployed automatically through code as well.

It’s a good thing, too. The fast pace of a DevOps culture would be challenging to maintain without the efficiency of automation. The ability to create, provision, destroy, or move virtual servers around on a whim is part of what facilitates agile development and accelerated execution by DevOps teams.

There are also some adverse consequences of automation. People have lost their jobs in manufacturing plants as machines have come along to automate tasks they once performed. There is a risk that automating many of the routine tasks normally delegated to IT personnel could result in fewer people being necessary, but that is not a reason to not embrace automation. Automation enables people to develop more advanced skills and focus on more important tasks—those functions that can’t be easily automated.

One of the primary aspects of functioning in a DevOps environment is a faster development cycle, and more frequent deployments. Traditional organizations might deploy updates once a quarter, or once a month, but DevOps teams often deploy updates at least daily—sometimes multiple times per day. Automation reduces the possibility of error—it is a simple, repeatable process that works reliably and predictably.

A survey of DevOps Conference attendees published by JumpCloud and SoftLayer found that about 75 percent of the respondents use a configuration automation tool such as Puppet or Chef. More than half use scripts of some sort to execute tasks, and about one-third reported using a patching tool of some sort. Many of those surveyed admitted that they have the tools they need, but there simply isn’t enough time to get things done.

Perhaps that explains why 82 percent of survey respondents plan to automate more. Breaking the data down a little farther, nearly a third are looking to increase automation of configuration and deployment, and about 10 percent want to automate user management, and a variety of tasks like log management, scripting, testing, patching, and monitoring all came in around five percent or below.

It’s still possible to create virtual servers, and configure them manually one at a time. That may work if you’re bored, or nostalgic for the “good ole days” of IT, but it’s impractical, and frankly just dumb. It’s also possible to cut your lawn with a pair of scissors, or paint your walls with a Q-tip, but it is highly inefficient and ill-advised.

Don’t waste time and resources performing tasks that can be automated. Identify the functions you routinely perform that are consistent and repeatable, and embrace automation to take care of those things for you so you can focus on more important things.

March 13, 2014