devopsy.kinsta.cloud

Tag: VisualOps

A Simple Service Discovery Solution for Docker
The ambassador pattern is an approach for multi-host Docker application deployment. It helps containers on different hosts to discover each other and communicate. Despite of the variants, the idea can be illustrated as the following:

(web) –link–> (web proxy) —network—> (db proxy) –link–> (db)

Instead of the web connecting with the db directly,
1. A local web proxy is linked with web and web is configured to send all traffic to localhost, which is received by the web proxy
2. Alongside this, the db proxy announces the db container with a predefined key in a key/value registry (etcd, consul, etc). The key is also used by the web proxy to discover the db container in the registry
3. Once discovered, the web proxy will route the traffic to the db proxy, which is linked with the db container.
As a result, the connection is setup.

Pros & Cons

This pattern has a number of advantages:
1. Non-intrusive: The Docker image can be used without any modification, as it is configured statically
2. Dynamic: Containers can discover each other in a dynamic way, particularly valuable to the autoscaling/load balancing case
3. Resilient: Failover is made transparent to the container
However, there is also some amount of tradeoff for the above benefits:
1. Complex: There are more components to learn and manage
2. More moving parts: A wrong key or value could lead to some bizarre results, which may be particularly difficult to debug
3. Inflexible: The proxy is good for 1-to-1 or a load-balanced connection, but it couldn’t handle other topology, such like Dynamo ring
Solution

Compared with the ambassador pattern, another solution of service discovery is to leverage the file mount feature to mount the application configuration file from the host instance into the container:

$ sudo docker run —rm -it -v ~/.bash_history:/.bash_history ubuntu /bin/bash

The approach is:
- Simple: No proxy, no registry, easy configuration files
- Non-intrusive: No code changes or image modification
- Flexible: Works with any application configuration
- No moving parts: Easy to reason about
One may argue that this is a static configuration of the containers, which does not cope with scenarios like autoscaling and failover. The answer to this is an orchestration engine. The engine need to keeps a watchful eye over the cluster and re-generate the configuration file to make sure the containers always get the correct connection whenever something happens.

Technically, the solution can be illustrated as:

Docker Service Discovery Solution

Instead of letting instances announce and discover each other via a central database, this explicit route specifies the logical relationship between services. Then the engine renders the file upon provisioning:

mysql://root@@{db.PrivateIpAddress}:3306 — -> mysql://root@10.0.0.4:3306

Using this approach to deploy a multi-instance Docker application, all you need to do is to specify three things：
- which docker image to run, i.e. my/node
- the container setup, i.e. port, cpu, mem
- the app configuration file content and link with other containers
This solution does not look particularly sexy, but what it is, is dead simple and robust; and works an absolute charm for any type of application.
September 17, 2014
Is Docker leaving Immutable Infrastructure?

According to Chad Fowler’s definition, Immutable Infrastructure can be defined as:

“[S]ervers (or whatever) are deployed once and not changed. If they are changed for some reason, they are marked for garbage collection. Software is never upgraded on an existing server. Instead, the server is replaced with a new functionally equivalent server.”

Though the idea is powerful implementing it is actually very difficult, especially with hypervisors. Thus, the DevOps community has advocated Docker from its launch to market because of its capability of building the Immutable Infrastructure, due to its layered image approach and super fast “booting” performance.

However, last week Docker introduced us all to a new feature to allow “Writable /etc/hosts, /etc/hostname and /etc/resolv.conf”:

“You can now edit /etc/hosts, /etc/hostname and /etc/resolve.conf in a running container. This is useful if you need to install bind or other services that might override one of those files. Note, however, that changes to these files are not saved during a docker build and so will not be preserved in the resulting image. The changes will only “stick” in a running container.”

It‘s an interesting move as the Writable feature opens another door. Things are no longer static, at least not 100%. For example you could change the state of a running container. The question now being asked is: Where is Docker heading to next? Are they moving away from ‘Immutable Infrastructure’ and if so, is ‘Immutable Infrastructure’ wrong?

The answer is Yes and No.

Immutability itself is a hugely valuable way to build, deploy and operate applications. But like all patterns, there is some trade-off.

One of the most discussed benefits of Docker is the consistency across different deployments: dev, qa, staging, prod, etc. However, there is always some context-specific configuration, e.g. IP, dns name. As the result, those parameters have to be set at the runtime, instead of being hard-coded in the “read-only” images. There are various ideas on how to solve this service discovery problem [1][2][3]. Yet the issue remains that all of these solutions are more or less intrusive to the code and application, which means quite some effort in testing the theories. Then, the whole thing becomes a decision based on personal preference. Even though we of course all want better ways to manage the infrastructure, the engineering cost with the 100% immutability is a concrete no go for a lot of people.

—

The new Writable feature gives us the option to non-intrusively determine the configuration at the runtime, so that it will not cost you too much effort, while at the same time retaining the immutability. Assuming you are deploying a WordPress application, which involves web-tier containers and DB containers. You certainly don’t want to hard code the DB instance’s IP, nor bring in an etcd cluster, which adds more moving parts to manage and increases the odds of faliure . Instead, an orchestration tool could do the job by leveraging the Writable feature of Docker:

As in the above diagram, the orchestration tool will take care all heavy lifting, from instance provisioning, container deployment, to a properly setup /etc/hosts file in the web containers. More importantly, if your DB instance’s IP changed, the orchestration engine will automatically rewrite /etc/hosts to fix the DB connection. And all you need to do is just to specify the following in /etc/hosts:

Pretty easy, right?

In conclusion

While Docker lays a great foundation of code deployment and it is especially useful in keeping the unit as immutable as possible, the deployment context plays an important part in the service discovery. By leveraging the new Writable feature and the Orchestration tool, one could really benefit from immutability, whilst avoiding having to change too much.

[1] smartstack [2] etcd [3] consul

August 26, 2014
The WhiteBox PaaS

One of my favourite blog posts about PaaS is James Urquhart’s ‘Is your PaaS composable or contextual? (Hint: the answer matters)’. In the post, James distinguishes two different types of PaaS: Contextual and Composable. He additionally implied a third type of PaaS (though he categorized it as “Composable”):

“Perhaps most importantly, however, there are open source “build” tool chains being deployed directly to infrastructure services that exhibit a purely composable approach toward delivering and operating applications. Combining GitHub with Jenkins with Gradle with AWS CloudFormation and Autoscaling and so on gives a fully automated, flexible “platform” for application development and operations — everything you want from a PaaS. The catch, of course, is that you’ll need to assemble and maintain that tool chain over time (rather than letting the PaaS vendor do it for you).”

The Whitebox PaaS:

When we started VisualOps, at first we didn’t realize that we were building a new type of PaaS or rather a customizable PaaS, this was partly due to our DevOps background. As James pointed out, the value of a classic “Contextual” PaaS can be achieved by using the DevOps tool chain. There is more transparency and control implied by ‘customizable’ over ‘composable’ as all aspects of the environment are within your control. Each building block’s entire make-up is determined by you rather than merely arranged. The Blackbox PaaS removes this ability to define your own rules.

Though currently this category of tool chain is mostly homegrown, the key point here is that PaaS does not have to be a Blackbox. A Whitebox PaaS exposes 100% native interface of the underlying infrastructure, giving ultimate control and transparency. Combining Infrastructure-as-Code and Configuration Management capability as a quick and easy way to build, automate and orchestrate applications. Compare the Whitebox PaaS with the Contextual PaaS and the advantage is clearly that the Whitebox PaaS poses no pre-determined constraints on the application, and as such your apps will be general-purpose and lock-in free.

Benefits:

A key benefit of this approach is that the Whitebox PaaS is capable of both deploying and managing itself. With aWhitebox PaaS service it is possible that the customer can click to deploy a private clone, and no management effort is required since the clone is supervised by the “origin”. This removes the “operation immaturity” stated by John E Vincent in his blog:

“To be clear while I feel that most organizations aren’t ready for the operational challenges of maintaining a PaaS, the job is made harder by the PaaS software. In both cases, the operational maturity of the products themselves simply isn’t there.”

A further value is that in the WhiteboxPaaS is not th eapplication runtime. Instead,your code will run in standalone infrastructures (VMs), and PaaS only needs to help automate and orchestrate the application. Thus, even if the PaaS is down, your application will still run un-impacted. This entirely side-steps the issue of ” their downtime [being] your downtime”, as well as the security concerns raised by the multi-tenancy of the public PaaS.

Objections:

There may be some concerns regarding the learning curve necessary and intrinsic complexity of the WhiteboxPaaS as it requires your involvement with the application setup and the system architecture. It is true that it is by no means the easiest option. However if you are looking from a business minded perspective, having this knowledge will give you a great competitive advantage. If you don’t know how to do this you may lose the opportunity to build a better, faster and more robust application. Moreover, if people use PaaS to achieve “NoOps”, then it would appear that the real meaning of ‘undifferentiated heavy lifting’ is the day-to-day operations necessary to keep your application running, rather than the knowledge it was built upon. In this context using a WhiteboxPaaS absolves you of all heavy lifting, yet still leaves you with the opportunity and ability to make improvements, it is ‘customizable’.

This is not to say that PaaS must always be Whitebox. James put it well: “it’s not an either/or choice, much like stability and resiliency”. Different types of PaaS serve different markets and different purposes. From our experience, we have seen some customers begin using a Blackbox PaaS such as Heroku and grow rapidly with it until they reach a point where they want or even need something more flexible and powerful, they then choose to migrate to AWS+Whitebox PaaS such as VisualOps. This progression allows you to concentrate your time and efforts wholly into your business itself at the beginning, then upgrade to a more proficient and flexible platform as your business grows.

About the Author
Peng Zhao is the Founder and CEO of VisualOps. Previously, he initiated and led the IaaS projects for China Mobile as well as being both a manager and developer in a various companies, including Reuters and Platform Computing. He holds a MsC. from Univerisity College Dublin. His twitter handle is @gnepzhao

July 25, 2014