devopsy.kinsta.cloud

Author: Alex Knol

  • The DevOps Role: Not a One-Person Job

    The DevOps Role: Not a One-Person Job

    This is actually a follow up of the article I wrote a little while ago on LinkedIn titled “DevOps is NOT Jenkins.” In that article, I ranted about how many people and companies confuse automation tools with DevOps. Now I want to address another misconception that I come across quite often: “DevOps is a role.”

    When managers tell me they have all bases covered because their team consists of developers, designers and a DevOps engineer, many questions come to my mind. Having the awareness to at least assign that role in a team is already something. Quite a few engineering managers think that by giving an engineer a DevOps title is enough, but nothing could be farther from the truth. As I mentioned in the LinkedIn article, DevOps is a culture, a process and a philosophy. Adding a DevOps engineer to the team is like declaring someone on the team to be the scrum master and assuming agile is in place. These days, Agile is widely adopted and usually it is not seen as a role, but as a proper methodology. (Although it never surprises me when I encounter a team that is micromanaged while a scrum master is on duty …)

    Like Agile, DevOps should be practiced by the team or organization as a whole. At the core of all engineering we should not only address features but also how those things will run in production. And let’s not forget quality assurance, security and compliance when designing a feature or fix. Fixing an issue at the beginning is between 10 and 50 times cheaper than leaving it until all the testing phase is done already, according to a NASA study. “The study revealed that costs escalate in an exponential fashion.”

    I suspect that concentrating on just the feature or fix is so much more convenient because it is the immediate business value that is addressed by completing it. Thinking about how a new configuration variable will be injected in production or how this new function is going to be load tested probably takes away from focusing on a adequate and speedy completion. There is even a possibility that taking every possible aspect into account simply is too much to fit in one’s brain. I know I’ve had to force myself to not forget anything with checklists and tools.

    In Practice

    Every change, feature or bug fix should have a checklist to make sure all the bases are covered. Naturally, the requirements for the change have to be captured with user stories. But also make sure there is a user story for each of the non-functional requirements. Here are some examples of user stories addressing the various concerns:

    • As a security officer I want to be aware of any changes in potential security risks so I can adapt my (automated) security checks accordingly.
    • As a quality assurance engineer I want to be able to verify the functionality of this feature/fix in isolation and integrated in the product.
    • As an operations engineer I want the to be aware of any changes in the runtime contract so I can update deployment systems accordingly.
    • As a compliance officer I want to reduce to scope for which compliance regulations are in effect.

    There are plenty of tools out there that allow these user stories to be part of a template so it’s easy to embed into the team’s workflow.

    Runtime Contract

    Defining a “contract” for how the application is configured and run in the target environment when the application or microservices are developed removes many uncertainties at the design phase. The mere fact that such a contract exists forces you to either comply with it or negotiate an amendment, like with any contract. A runtime contract has to define how the runtime is configured and which elements are considered secrets. It also needs to describe how and in which format the service outputs runtime feedback around errors and informational messages. Metrics, startup and shutdown and health checks also must be considered. This is an example of what such a contract could look like.

    Broaden the Role

    After checklists and contracts have been established, a large part of the process is codified. I typically write these things down in a version controlled document structure so amendments can be easily tracked. However, this still doesn’t relieve us from the misconception that DevOps is just a role in the team. In fact, the DevOps culture within a team or organization probably should be guarded and or facilitated by a person or persons, much like the Scrum master is tasked with facilitating that process and methodology. Probably, the actual automation will be built in large part by the person tasked with the DevOps responsibility, but on top of that it is also their responsibility to raise and maintain awareness of all the aforementioned aspects of testing, deploying and running an application.

    As such, I believe that people practicing DevOps are required to be more than average communicators and well-versed in every step of the software delivery life cycle (SDLC). Almost all of the DevOps job descriptions that pass my eyes take great care to mention all kinds of technology and familiarity with clouds and programming languages, but don’t emphasize the communicative requirements. It’s time to take DevOps seriously and start reaping real benefits by embracing it as a methodology.

    I’d love to hear your opinions and feedback. Let’s start a discussion below in the comments or on LinkedIn/elexy.

    — Alex Knol

  • DevOps: Making Developer Experience Reality

    DevOps: Making Developer Experience Reality

    As DevOps professionals, among other things we are responsible for making developers’ lives a little bit easier. How many times have you heard, “Do you have a working vagrant box?” Or maybe you need to set aside a few days to onboard a new developer.

    Many times a development environment is created when a project is started, usually on a developer workstation. This typically means it will work on that laptop. But, as the project evolves and more engineers get involved, the need to share this environment becomes apparent. When the project grows in complexity, requiring more components and third-party services, this organically grown developer environment quickly becomes unmanageable.

    Get Ready for Production

    Another interesting aspect of the developer environment is the level of similarity with the actual production environment. Even evaluating this aspect requires that you have insight in what production looks like. If projects start on a developer laptop, the desired production setup is seldom considered. More times than I care to remember, I was called into a project when the “thing” that worked well on the development teams’ laptop had no sensible production deployment format.

    This teaches us that even if we start out small, we must think about how the project will function in a production environment and how we are going to get it there. In practice this means that any new project requires at least the following:

    • Some unit tests, they may even start out as stubs.
    • A minimal deployment target environment.
    • Continuous integration (CI), running the (stubbed) unit tests.
    • Automated delivery, preferably continuous deployment (CD).
    • At least one simple integration test, possibly just a curl to the deployment app to verify it is alive.

    This does add some complexity to your project startup, possibly beyond an individual developers’ abilities. It does, however, lay the foundation to, at the very least, be able to deploy the application somewhere. For a seasoned DevOps professional, adding a basic CI/CD only takes an hour or so and there are plenty of free cloud resources for small projects.

    When the project grows and the development environment gets more complex, things get a bit more tricky. The above steps will not really help you mitigate the possible frustration of keeping all the moving part inline. A pattern I see often is that developers try to spin up the entire production stack on their laptop. For a containerized environment, this might mean a Docker Compose-based setup or even minikube to run a local kubernetes cluster. These tools often are using shared volumes for quick iteration, which work well until a large number of files need to be kept in sync. Vagrant is a great way to quickly spin up an environment. This does offer some level of production parity, but certain parts are missing: The CI/CD system watching commits to your source code repository is not deploying to your local setup and traffic to and from it are not passing any additional infrastructure such as load balancers or firewalls, to name a few.

    Local-Remote

    When things break in production, “It it works on my laptop!” is a phrase we have all heard at some time or other. What if we could bring the laptop environment a bit closer to production? It’s not the developers’ fault if things break in production but work well on the local workstation. Providing an option for developers to develop iteratively using a remote infrastructure means we can get closer to production parity. Also, it means that onboarding new developers in the team could be simplified significantly because all the complexity is in the remote environment. This remote environment is provisioned in the same automated way as production, so we have a great degree of confidence about its state. For Kubernetes-based environments, many great tools exist including Skaffold and Draft. They both provide an easy way to build, push and deploy applications to a Kubernetes cluster. This makes it easy for the developer to take on the responsibility to ensure the application under development will run in the target environment.

    If, for some vague reason, you are not containerizing you applications but run code in a virtual machine instead, there are also options. Most modern editors provide automatic syncing with a remote server, either on-demand or when code is changed. Ksync is a tool for syncing your local code into a pod in a Kubernetes cluster on file change. Syncing code live removes the delay of rebuilding a container image and uploading it every time you want to see it in the deployment environment. As such, code sync tools are better suited for iterative development.

    Recently, I worked with a client setting up Telepresence  for a development team. This is also a tool for Kubernetes environments but it offers a twist—developers get to run their code locally with all traffic in and out proxied through a remote Kubernetes cluster. Like with sync tools, this means there is no more long delay before viewing changes, IDE debugging works as normal and all the log output is local as well. When code is committed and pushed, the CI/CD system should update the cluster with the latest version, which in turn can be used to iteratively develop on with Telepresence. It’s a relatively young project but offers great potential. An added advantage is that only a small portion of the entire application stack or landscape needs to run on the developers’ laptop, which means it can do with less memory and a cheaper CPU.

    Conclusion

    Containers and microservices are more widespread than ever, making it increasingly difficult for developers to have a production-like environment to develop in. The key to solving this problem is to not try to load everything into an evermore powerful laptop, but splitting the load and complexity between developer workstation and a remote environment.

    — Alex Knol