devopsy.kinsta.cloud

Category: Blogs

COBOL: Completely Obsolete But Omnipresent Language

The programming language COBOL is a nuisance, something everyone wants to get rid of; however, I’m about to advise you to learn it. It might be a gold mine.

The IT world is a fast paced one where obsolescence is the rule and where things evolve so fast that any person who wants to understand it has to continuously learn and train. Languages, methodologies and technologies keep on changing, evolving to make programming easier, safer more robust and above all, faster. As to every rule, there is one exception.

In 1959, convinced that the English language would be more suitable for programming than machine code or the assembly language, Grace Hopper creates the first ever human-readable programming language that would survive for over fifty years: COBOL

COBOL, 55 years old and still in its youth, not ever thinking about retirement, has become a nuisance, something everyone wants to get rid of but no one has the courage to try.

Are we ready to replace COBOL?

Nowadays, 80% of financial transactions are made through COBOL programs and most of those programs have been written decades ago and have proven to be reliable, effective and efficient. To top it all most companies which are still using it have invested a lot of money in those systems and if it is not broken, why fix it?

So why would one want to change them? Well, it’s not Java. COBOL was created in an era where IT didn’t have the meaning it has today. A program was mainly designed to replace simple repetitive tasks such as retrieving data from a database, performing a couple of operations and generating a report or updating some other data. Today’s programs have to present a front-end to thousands of users, go through different layers of security, logic and quick data retrieving before accessing the back-end to ultimately alter data and send some information back to the user. Alternatively, a program can be a tool to design objects and buildings, communicate with your friends, frag your opponents, enhance your reality, you name it… The whole philosophy behind the conception of programs has changed so much in the last two decades that having such an outdated technology that almost no one knows anymore represents a potential danger. Only few people learn COBOL these days and even though those systems are not supposed to evolve much, there is always maintenance to be performed. But are we ready to replace it just yet?

Fast, robust and future-proof language

COBOL has been on the brink of death for years but its presence overall hasn’t changed. Not only are the cost to migrate from one technology to another huge but the choice of the replacing technology itself is a million dollars question. What better than COBOL to perform typical COBOL operations? While not designed to be an Object Oriented Language such as Java or C#, COBOL has slowly adopted some of their aspects, extending itself and its functionalities to a point where almost everything is possible. Everything with the exception maybe of creating user friendly or good looking Human-Machine Interface. In that matter, nothing has changed much since Hopper’s time. Many of us still remember the frustration of spending hours to get a simple screen with a couple modifiable fields and those who still use them must be dreaming every day to replace F5 and F6 by simple buttons…

But that was never the point; the point was to have a fast, robust and future-proof language.

COBOLs main contenders

The discussion is still opened as to which language could replace COBOL but here are some pros and cons to the main contenders:

– Java is easy to learn, well spread and has a huge community of developers already but is rather slow compared to other languages despite the existence of some specialized compilers and massive improvements in the last few versions. It is also arguably still in its youth and the yearly evolutions and changes to the language don’t play in its favor. Not to mention the recent security issues.

– C++ is fast, flexible and robust but it takes time to learn and understand and has what is maybe its biggest flow, potential memory leaks. Something COBOL and Java programmers have, almost, never heard of. By conception, it offers a lot of freedom to developers but that also implies few ready-to-use components and thus it’s less cost-worthy.

Let’s say that one day we manage to find a clear winner, another major problem will suddenly emerge. COBOL was made to perform all kind of operations in one block but we have learned throughout the years that modularity is the key to success. So if we replace COBOL itself by another language, we will also need to have databases and SQL, a front end and HTML, most likely a Business Process Management Software to externalize workflows… Basically, replacing COBOL programs means creating whole new ones.

The ultimate piece of advice

While it is possible to replace this aging language, this comes at such an extremely high cost that we can anticipate its presence in at least the next twenty years which brings us to the ultimate piece of advice of this paper: if you have the opportunity to learn it and enough motivation to write white on black interfaces, go for it. The pool of competent COBOL programmers is getting smaller by the day and the demand doesn’t change, meaning that this personal investment might reveal itself a gold mine.

September 3, 2014
A Culture of Trust
Trust is by far the most critical element of a DevOps culture. It is also the most difficult to achieve. IT was not built for trust. The silo structure that is common in most IT organizations was built for specialization and territorial ownership. The silo walls are thick.

While DevOps may not be able to break down the silos, but it can encourage and groom a culture of trust.

Trust is difficult to define. We know when we feel it, we know when we don’t. In 1993, Dr. Duane C. Tway, Jr. published a dissertation called “A Construct of Trust”. In it, Dr. Tway defines trust as

“The state of readiness for unguarded interaction with someone or something.”

Dr. Tway believes that trust is actually “constructed” from three basic elements:
- The capacity for trusting
- The perception of competence
- The perception of intentions
The capacity for trusting is how your total life experiences have developed your current capacity and willingness to risk trusting others.

The perception of competence is your perception of your ability and the ability of others with whom you work to perform competently at whatever is needed in the current situation.

The perception of intentions is your perception that the actions, words, direction, mission and decisions are motivated by mutually-serving rather than self-serving motives.

While trust is individual, a culture of trust is organizational. Both are earned. DevOps is a great opportunity to look at Dr. Tway’s construct of trust and identify small but meaningful opportunities to steadily increase the trust between Dev and Ops including
- Improved and honest communications
- Increased personal interaction
- Honored commitments
- More listening than talking
- Frequent knowledge sharing
- Welcomed input and feedback
- Admitted mistakes
- Intolerance of blame
- Mutual respect
- A celebration of mutual achievements
What else could you do to increase the capacity for trusting, perception of competence and perception of intentions in your organization? Perhaps you can start with an honest dialog with and between your Dev and Ops teams. The answers may be surprising and insightful.
September 2, 2014
Being Agile is all about controlling the flow

For businesses, Agile is the new fashionable initiative as was E-business was in the 2000’s, Six Sigma was in the 90’s and Total Quality Management (TQM) was in the 80’s. If we look back across the spectrum of these initiatives, we need to examine how many of these initiatives met the goals the business was looking for and, more importantly, if they were fully adopted, remembering that many were derailed, did they deliver the value the business was seeking?

Of course we cannot answer this question with generalities—it’s a spectrum. For some businesses, these initiatives were red herrings, some did not follow through to completion, but certain changes adopted as part of these initiatives had a very positive impact, while others succeeded wildly. It’s a Harvard Business Review case study as to why a business ended up at the point on the spectrum that they did.

Clearly, one fundamental aspect that is consistent across all these initiatives is the ability for a business to embrace change. In a previous blog, “Cloud Computing, DevOps and Business Velocity,” I addressed the issue of business velocity—the rate at which an organization can embrace a particular disruptive technology change. Underestimating your business’ ability to adopt a new technology or process will most likely have a strong negative impact on the outcome. In a recent conversation with a client, the non-IT business areas has been very successful at adopting lean practices, but IT short-circuited shortly after starting. It didn’t outright fail, but after a strong start, they took a break from which they never started back up. Without a post-mortem, we don’t know why this specific initiative fell flat, but we can intimate that if something is difficult and feels unnatural, we’re more easily able to focus our attention elsewhere—think dieting and exercise programs in your own life.

This seems to be the decade of Agility. Businesses want to be more dynamic and fluid. They want the organization to be prepared to respond to market changes more quickly, introduce new products faster, and respond to problems that occur more rapidly. To accomplish this goal, the business will have to let go of some of the ways that they do things today. They will have to re-engineer processes to be more compose-able and componentized so that they are less static. For example, a manufacturer goes through a lot of planning to deliver a new product to market. They go through steps for design, prototype, test, source, manufacture and distribute. The entire cycle may take upwards of a year and is highly automated once it reaches the manufacture stage. Tweaking the product between batches would cause significant delays and pain because the process is static. It effectively amounts to restarting the entire cycle. So, what if they discovered that by simply elongating the body by 2 cm the product was far more useful their intended audience? In this case, agility might equate to being able to complete the design-to-manufacture process faster.

In the previous example, this latter step of being able to tweak the product is called continuous improvement and is a key element of fostering agility. If a product isn’t selling well in the market, many companies will just abandon the product completely due to the costs associated with making the product viable. The reason being that most manufacturers consider that each product is a project and when it reaches the manufacturing stage it moves under the control of operations. That is, the creative team that includes marketing, finance, and other stakeholders view their position as “completed” with regard to getting that product to market and they then move onto other projects. With this mindset, there’s no room for the group to review the process, think about how it could have been done better and then iterate with the improvements incorporated. Hence, they just keep taking their “this is the way we do it” to the next project.

Being agile means being able to control the flow. There’s no way to change how products come to market if there’s a culture of accepting finality once a product has reached a certain point in the cycle.

The other aspect of controlling the flow is managing how products and services enter and exit the development pipeline. I spend a lot of my time helping IT organizations unlock their potential to deliver by helping them to understand that there’s only so much that can be done to improve managing the work-in-progress (WIP). The only way to make gains beyond a certain point is to enable transparency with those that are responsible for inserting work into the pipeline or who are demanding that work completes and exits the pipeline at a certain time.

The following two figures express well the reason transparency is a requirement. I believe most people would agree that managing production is a far easier task in Figure 1 than Figure 2. In Figure 1, there is one input and one expected output. Managing WIP for this pipeline is a matter of optimizing just the process for completing that one deliverable. In Figure 2, there are many more variables at work. There could be contention for resources within the pipeline to support the multiple inputs.

Likewise, there may be a request to reprioritize what gets completed when to meet certain business goals. Perhaps a particular report is required urgently to support a key business decision. I personally dealt with this very scenario when the business I was working for years ago was suddenly engaging in acquiring another company. All focus was transferred to producing the needed financial statements and everything else was put on the backlog. In that particular case, the CEO and CFO were the customer, so no one questioned the delays to their projects, but for many IT organizations reprioritization requests are never that clear making it very difficult to manage the control of the flow. Hence, transparency and communication between those consuming the services of the pipeline and those who own the control of the flow of the pipeline is an imperative for agility.

Like TQM, Six Sigma, and E-Business, Agility will require changes to the fundamental way that a business operates. An analogy of this process can be as benign as a person recovering from knee surgery or as complex as recovering from a stroke depending upon what the business wants to accomplish and how they are organized currently. Along the way, people will most likely be the biggest hurdles to success.

When change of this magnitude occurs, employees tend to become very myopic of their own situation. To mitigate this risk, it is critical that changes are reviewed with employees prior to change, during implementation and after the change is enacted. Management must gauge if employees are struggling because the complexity of the change or because they personally don’t agree with the change. The former can be overcome with education and support while the latter should be dealt with quickly through removal.

Businesses can succeed with their Agility initiatives given they identify and mitigate key risk factors prior to starting and are willing to commit to looking critically at the way they have been operating the business. In agile software development, developers are trained to think about ways their code can break and then write unit tests to ensure that the software is sustainable in light of these possibilities. Creating an agile business means looking at ways to ensure sustainability in light of market changes, new competitors, supplier changes, and changing customer demands.

August 28, 2014
Moving to DevOps in a traditional enterprise
Introduction

One of the challenges of introducing DevOps in the setting of a traditional (i.e. siloed, waterfall etc) Enterprise is where to begin.

It’s certainly something I wrestled with alongside the CTO for quite some time.

It’s not possible to impart years of experience of moving a large company to a DevOps ways of thinking but I hope to give the main 3 pillars of how I’ve approach it previously.

Note…none of it mentions technology and tools! DevOps is primarily a cultural idea and therefore affects people above anything else.

Step 1 – bring in the Rock Star

If you are moving to DevOps in your organisation, then bringing together different people from different silos and then expecting them (under their own steam) to magically cooperate and agree on everything is a little unrealistic.

If you are dealing with historical teams and employees – which I suspect most of you are – then breaking down silos is likely to be a significant challenge.

Aligning incentives makes things a lot easier. If both your devs and ops guys are paid a bonus depending on how much “change” they do in a year is a simple way of getting them to agree on whether they should automate releases for example. That’s a little too simplistic but hopefully you get what I mean.

This may well come at the cost of something else though. The first thing that springs to mind in the last example is quality.

Let’s put everything straight into live! We get paid for it!

Ok so that isn’t going to work.

Sometimes, at least to begin with, you need to bring people together and ultimately empower someone to make decisions when one can’t be reached through consensus. I’ve seen a real danger in DevOps of death by committee. The more silos you bring together then potentially the greater the possibility of them not making a decision.

You may have heard about this idea of DevOps rockstars. Guys (and gals) that get DevOps inside and out. People that understand all sides of the coin (development, DBA, infrastructure etc).

They try and influence through education and persuasion and point people in the right direction. They help the silos reach the right decisions themselves rather than a top-down “do what I say”.

All said and done sometimes this open inclusiveness doesn’t always work and someone needs to just stand up and be counted.

So what’s my point?

If you are trying to move towards DevOps then hire a rockstar (I doubt Rod Stewart is suitable btw). Someone that has done it before. Someone with a very mature sense of what DevOps is and preferably someone that has done most of the jobs he/she is trying to influence.

Moreover, hire someone who’s first instinct is to try and bring people around to their way of thinking. And lastly, someone who isn’t ultimately afraid to put their privates on the line and make a goddam decision.

Step 2 – look at your people structure

The traditional definition of DevOps is to break down the silos between Development and Operations so that they accomplish shared goals.

However, I’m a keen advocate that DevOps isn’t just Dev and Ops (see http://enterprisedevops.blogspot.co.uk/2014/07/devbiztestreleasethingyops.html).

It’s about all the functions that come together to achieve meaningful change.

The devs, the ops guys, the business and techies should all have shared goals and incentives. Where possible they should be sat around the same desk. The team should rise and fall collectively (there is no “us and them”). The team should manage change from cradle to grave (requirements, build, test, deploy, support).

I think we should rename Devops to “DevBizTestReleaseThingyOps”. Catchy.

To that end, DevOps is as much to do with “people” than it is about new tools and processes.

With that in mind, potentially you are talking about some fairly radical changes across a wide number of people of various skills, experience and personality types. I guess it really depends on what your organisation is structured like, the skills of your team and many other factors. For my company (a large UK insurance company) however, it’s meant some fundamental changes.

We started by aligning change functions (devs, testers, BA’s etc) to individual business units (rather than monolithic shared group services).

That fundamental change has been sponsored on the back of some early and small wins by implementing things like Continuous Integration and labeling it “DevOps”. Ok so it’s not DevOps but if using that term starts to get visibility and movement within your organisation then I don’t think that’s a bad thing. I always believed we needed a banner for the modernization of the department to fly behind. If that banner reads “DevOps” then so be it.

That early small success was really vital to us.

When we started out we had a lot of push back and negativity….
- Why are you telling me how to do my job?
- You are using the wrong tools
- I don’t see the benefit
- I know more about this than you do
- What you are trying to do is easy
- We don’t have the right people
- This is a hipster idea for the likes of Facebook
- There are regulatory reasons this won’t work
Basically everything under the sun.

That’s stressful when you are trying to do the right thing and move the incumbent forward.

I don’t think we helped ourselves too much to begin with. There was too much of my team trying to pick up and change other team’s technologies and processes. Not enough leading the other team to water and helping them implement something themselves. Lesson learnt.

Step 3 – create a team to help others

I mentioned earlier that DevOps is cultural idea and not a job title. Definitely not. Unimaginable.

So how is it I run a “DevOps” team? Surely that’s a collective of “DevOps engineers”?!

My team “Platform Services” supports the idea of the change platform at my company. The change platform consists of tools such as Jenkins, Puppet, uDeploy, Teamcity, TFS and a few others.

Someone ultimately needs to “own” the tools making sure they are maintained, creating best practices and establishing a roadmap for the future.

My team goes beyond this and helps different change functions adapt to the change platform and are evangelists for things like DevOps and Continuous Delivery.

With DevOps being concerned with breaking down silos its contradictory to have a team that bottleneck of knowledge and skills. To this end, Platform Services is the silo-buster – we help others pick up the tools & processes and implement them.

For example, we don’t do deployments ourselves but we do sell the benefits to other teams and help them to adopt the tools and processes.

I think this is a good model and helps spread the success of DevOps around the IT function.

It helps that consultancy from my guys is effectively free to other parts of the IT team (thanks to an enlightened CIO & CTO).

I hope you found some of that useful. Feel free to check out my blog on my move to DevOps: http://enterprisedevops.blogspot.co.uk/

Good luck and get in contact if you have any questions or comments.
August 4, 2014
The Continuous Delivery Pipeline — What it is and Why it’s so Important in Developing Software
Continuous Delivery (CD) is a software strategy that enables organizations to deliver new features to users as fast and efficiently as possible. The core idea of CD is to create a repeatable, reliable and incrementally improving process for taking software from concept to customer. The goal of Continuous Delivery is to enable a constant flow of changes into production via an automated software production line. The Continuous Delivery pipeline is what makes it all happen.

The pipeline breaks down the software delivery process into stages. Each stage is aimed at verifying the quality of new features from a different angle to validate the new functionality and prevent errors from affecting your users. The pipeline should provide feedback to the team and visibility into the flow of changes to everyone involved in delivering the new feature/s.

There is no such thing as The Standard Pipeline, but a typical CD pipeline will include the following stages: build automation and continuous integration; test automation; and deployment automation.
- Build automation and Continuous Integration
The pipeline starts by building the binaries to create the deliverables that will be passed to the subsequent stages. New features implemented by the developers are integrated into the central code base on a continuous basis, built and unit tested. This is the most direct feedback cycle that informs the development team about the health of their application code.
- Test Automation
Throughout this stage, the new version of an application is rigorously tested to ensure that it meets all desired system qualities. It is important that all relevant aspects — whether functionality, security, performance or compliance — are verified by the pipeline. The stage may involve different types of automated or (initially, at least) manual activities.
- Deployment Automation
A deployment is required every time the application is installed in an environment for testing, but the most critical moment for deployment automation is rollout time. Since the preceding stages have verified the overall quality of the system, this is a low-risk step. The deployment can be staged, with the new version being initially released to a subset of the production environment and monitored before being completely rolled out. The deployment is automated, allowing for the reliable delivery of new functionality to users within minutes, if needed.

Your Pipeline Needs Platform Provisioning and Configuration Management

The deployment pipeline is supported by platform provisioning and system configuration management, which allow teams to create, maintain and tear down complete environments automatically or at the push of a button.

Automated platform provisioning ensures that your candidate applications are deployed to, and tests carried out against, correctly configured and reproducible environments. It also facilitates horizontal scalability and allows the business to try out new products in a sandbox environment at any time.

Orchestrating it all: Release and Pipeline Orchestration

The multiple stages in a deployment pipeline involve different groups of people collaborating and supervising the release of the new version of your application. Release and pipeline orchestration provides a top-level view of the entire pipeline, allowing you to define and control the stages and gain insight into the overall software delivery process.

By carrying out value stream mappings on your releases, you can highlight any remaining inefficiencies and hot spots, and pinpoint opportunities to improve your pipeline.

Don’t Add New Functionality Until You Get The Quality Right!

Continuous Delivery is about enabling your organization to bring new features to production, one by one, quickly and reliably. That means that every individual feature needs to be tested prior to rollout, ensuring the feature meets the quality requirements of the overall system.

In a traditional environment, development teams typically try to implement an entire new version in one go, addressing software quality properties (such as robustness, extensibility, maintainability) only when the project is close to completion. However, as deadlines loom and budget pressures grow, quality is often the first thing that is compromised.

Poor system quality, low-user satisfaction and endless “quality band-aids” can be avoided by adopting the principle of not adding new functionality before getting the quality right. You should always first meet and maintain your quality levels and only then consider gradually adding functionality to the system.

With CD, each new feature is required to meet the level of quality expected for the system as a whole, right from the start. Only once this quality level has been reached can the feature be moved to production.

Getting Started With Continuous Delivery

Obviously, organizations cannot and should not rush into adopting Continuous Delivery all at once throughout all their business units. The best approach is to focus on improving your biggest delivery bottleneck. CD will automatically show you what the next bottleneck is. This puts you on a measurable improvement path.

The main goal of using Continuous Delivery is to roll out new features and functionalities that are better than previous iterations — gradually incorporating and refining the CD principle throughout the organization. Go slowly, go smoothly — and watch the improvements!

Much of the material here is covered more extensively in the book, “The IT Manager’s Guide to Continuous Delivery” which is available at go.xebialabs.com/IT-Managers-Guide-to-CD.html

About the Author

Andrew Phillips heads up product management at XebiaLabs. Andrew is an evangelist and thought leader in the DevOps, Cloud and Continuous Delivery space. He sits on the management team and drives product direction, positioning and planning.
July 29, 2014
Five Top Tips for DevOps At Scale
Since I published my blog explaining the topic of ‘Big DevOps’ late last year, the concept of Enterprise DevOps has exploded, with commentary, experience, and advice (some good, some awful) popping up from many different quarters. Fortunately, many pundits have now come to agree that application delivery in a traditional enterprise is very different from in a typical web-scale/startup environment.

As a result, a growing body of work is filling a significant and important gap with unique insight into Enterprise DevOps, from many excellent commentators and practitioners who understand how large enterprises are different, and how to address those different challenges.

As Enterprise DevOps takes off, I seem to field questions about it on an almost daily basis, so I decided to contribute to the body of work with a webinar explaining my answers to five of the most frequent challenges I hear about DevOps at scale. It was titled ‘Big DevOps: Five Top Tips When Implementing at Scale’ and you can catch the full replay on YouTube.

The top five issues I chose to address are below:

My five ‘Big DevOps’ tips for overcoming these issues center mainly on cultural change, because DevOps is primarily a cultural and attitudinal shift. However, it is also undoubtedly true that certain technologies can play a large part in enabling that change, especially in the more complex world of larger enterprises.

With that in mind, my top five tips to address these challenges are as follows:
- Facilitate a culture of teaming and collaboration – this is at the core of devops in any environment, large or small, but especially for large organizations with bright lines between teams, this is essential to overcoming the tyranny of complex and dispersed silos
- Establish agile development as a shared discipline – agile development is brings code to market faster, but can result in just filling the clogged pipes of PMO, HR, training, security, and other departments faster. ‘Agile’ needs to be common throughout the SDLC – in dev, ops, and beyond
- Automate relentlessly to enable rapid devops response – automation allows faster execution throughout the system, enabling downstream processes to keep up with speed of DevOps. Automation also adds audit and governance, especially important to larger organizations
- Push smaller releases faster, measure and remediate impact – large organizations especially are can avoid big errors by avoiding big releases. Adopt faster, smaller cycles with closer scrutiny to achieve a DevOps approach while managing the risk of faster cycles and fewer controls
- Manage user identities to enable shared access – in smaller organizations it might be acceptable to give out root access like candy, but not for large enterprises. Identity and access management can enable a new DevOps approach that is safe, secure, auditable, and well-governed
While these tips may not be the perfect answer for every organization, I have seen how these approaches have made the journey to DevOps easier, safer, and more effective for big enterprises. I hope they help you on your journey.

If so, you will want to stay tuned to DevOps.com. Since starting work on Enterprise DevOps concepts, I have collected many questions on how large organizations can best succeed in their DevOps transformation. I will be sharing these questions along with my answers here on DevOps.com in a regular series, to give everyone the chance to understand how large enterprises can overcome the challenges of adopting DevOps at scale.

So keep your eyes out for my posts over the coming days, weeks, and months in the brand new occasional series, ‘Enterprise DevOps Q&A’. I hope you will keep reading, and maybe even help me out by discussing your experiences and ideas in the comments below.

Finally if you have any questions about Big DevOps, Enterprise DevOps, or anything to do with adopting a DevOps approach at scale, let me know in the comments below too. I would love to hear from you, and answer your questions if I can, or source guest opinions from the DevOps community.

Meanwhile, please stay tuned for more Enterprise DevOps Q&A. I am excited to continue the discussion on what it takes to transform application delivery at scale.
July 24, 2014
Is SDN just DevOps for the network?

The answer depends on your definition of SDN.

(more…)

July 22, 2014
Infrastructure-as-code, new rules for the old game

There’s been a lot of discussion regarding DevOps and its role in fostering quality and continuous improvement through increased levels of automation. In turn, this automation is fostered through emerging infrastructure and software architectures that are exposing control plane and management programming interfaces, which has been dubbed in many circles as Infrastructure-as-Code.

This movement allows development and operations personnel to have real-time interactions with the underlying environment—the network, servers, operating systems, and storage—that are used to deliver applications and data. It also means that as the environment becomes more programmatic the business must treat this this effort as a first-class application development process and ensure that the corresponding assets follow common coding practices and meet with a consistent and sustainable architecture.

Those that are unfamiliar with the activities of system administrators and data center operations may have an ill-conceived notion after reading emerging literature on DevOps that operators have been running the business on the backbone of purely manual processes. Even as a life-long IT person, until I started working with infrastructure and operations as part of my role, I was unaware of what happens behind the great curtain that keeps the digital bits flowing through the organization.

The truth is that operations and administrators have been developing scripts for years to automate routine tasks. These scripts have typically been used to dynamically generate or update configuration files that were then applied after a restart. This change to Infrastructure-as-Code is significant because now these configuration changes can be made on the fly and are often immediately acted upon without having to take the system offline. If you’re now putting your hand over your mouth upon the realization of the potential disastrous effects this can have, you understand the impact of this power.

Thanks to Software Defined Networking, hypervisors, and even Software Defined Storage, an operator can programmatically change the entire environment underlying the application without having to notify the application of this change. Architecturally, this is a wonderful thing. Pragmatically? Well, there’s a lot of bad code out there that makes assumptions about the underlying infrastructure and is about to be exposed when migrated to these modern infrastructures.

Hence, building, testing and deploying infrastructure control applications must become part of the overall Software Development Life Cycle (SDLC) and begin to be built, tested, deployed and managed using the same controlled techniques that are hopefully being used for any applications that are being developed by application engineering.

You may be asking, “Wait, doesn’t that mean my QA environment must have the same infrastructure components as my production environment?” To wit I respond, “Maybe!” In some cases this may be advantageous for ensuring that production deployment can support continuous deployment processes, which will ensure limited human intervention from the point of user acceptance through deployment, which will, hopefully, lead to limited opportunity for production failures. However, because we’re moving to a more programmatic infrastructure many vendors can also supply virtual instances of their infrastructure code for purposes of testing and this may be satisfactory for your needs.

Of note, there are many businesses out there that do not develop their own software. They primarily use Commercial-Off-The-Shelf (COTS) applications. Thus, these businesses may not have an established SDLC practice. Likewise, these businesses may not believe DevOps is relative for them since they do not do “development”. If your business falls into this category, the move to Infrastructure-as-Code and Software Define Data Centers should be perceived as a driver for DevOps adoption as well as formally adopting one of the well-defined SDLC approaches. Even if these components won’t impact you for a few years until your next refresh cycle, the advantages DevOps brings around improving quality and consistency in delivery is of definite value to your business now.

Infrastructure-as-Code is a significant change in capability that operations has for delivering compute and storage resources to the business. It will allow them to have greater insight into the hardware allowing them to do predictive maintenance, migrate applications to various resources based on demand, and tune the overall environment to meet the business’ needs at a given moment. In short, it will allow IT to deliver compute services with a high degree of availability and security. It will also expose shortcomings in existing applications that have made assumptions about the underlying environment on which it is operating. Thus, if operations and application engineering are not working together and the entire application environment inclusive of the infrastructure control apps is not tested and deployed as a unit, the likelihood for failure will be high.

June 30, 2014
The Double-Edged DevOps Job Title

Last month at GWO in Atlanta I met Nathen Harvey with Chef. I liked his DevOps presentation, “DevOp With Me!”, especially how he used “Walk This Way” by Run DMC and Aerosmith as an example for how collaboration in a DevOps environment often ends up playing out. They literally break down walls in the music video.

While not all of Nathen’s presentations are safe for work ( many include some explicit language. It’s safe to assume so unless clearly labeled otherwise) they are all entertaining and informative. I’d encourage you to check his stuff out, especially if you’d like to know more about how and why Chef and DevOps mix well.

I asked Nathen if he minded me linking his deck and giving him a shout out in a blog post. I even promised to say nice things about him. The answer he gave me only reconfirmed he does really get it:

Of course! (even if it’s not all nice things, as long as they’re honest)

If I’m being completely honest I have to admit this changed the topic of this blog post. I liked the idea of setting devOps to music so much I was going to post a DevOps playlist. I may still write or talk about that another time. It ended up being pretty easy to think of other songs that would fit for one reason or another. It actually even makes for a decent drinking game in the right crowd.

Instead I’m going to talk about the one thing I know Nathen and I don’t see eye to eye on. Nathen’s an advocate for doing away with using the word “DevOps” to describe a role or a team. I, being a DevOps Engineer, strongly disagree with this perspective. When I asked Nathen about this he responded with a single question:

Can you describe what you do without using the word DevOps?

It’s true there are a lot of folks with the word DevOps in their title that are adequately described as a systems administrator or a build automation engineer. Coming up with job titles that don’t contain the word DevOps is also a decent drinking game in the right company. A good serious answer for me would be ombudsman or public advocate. I think KTLO (keep the lights on) custodian would also accurately describe how I’m earning a large part of my keep these days. Still, the job I interviewed for was DevOps Engineer, and for now I’m going to keep using those words to describe what I do. I like Nathen’s Socratic response to why we shouldn’t be describing people and teams with the word DevOps, but what I am doing isn’t accurately described by other job titles where I work. If another title is a better fit for the work you do where you’re at, then I would encourage you to take Nathen’s advice.

The more common reason I’m given as to why I should have changed my job title on day one is really pretty disappointing. I’ve read more than once that I should be concerned that my head will be the first one on a pike should the devOps movement fail. This really isn’t a concern I have.

First of all, DevOps does work. I’m not sure who would dispute that the CAMS model for DevOps is made of things any organization should be trying to improve Kaizen style. I’d even say that improving Culture, Automation, Measurement, and Sharing together results in a sum that is greater than any of its parts.

There are however places that aren’t ready to embrace the change needed for this type of improvement. I caught a quote from @filler when @RealGeneKim retweeted it last November:

If you’re concerned about a lack of progress in echo chamber, go offer yourself up in a different forum. And bring cookies.

Advice I took quite literally. I found a job with a DevOps title at a place that was actively trying to change. I regularly bring in food. Just this past week I baked six dozen toll house cookies from scratch for a DevOps round table meeting I hosted. Why? Because I genuinely give a damn. I want people to come to that meeting to listen and to share. People who are passionate about technology and have faith that there’s always a better path. This group should find such a round table conversation quite enjoyable. The cookies help us remember this is supposed to be fun. They help keep the mood collaborative instead of conflicting. Such a job shouldn’t be left to baked goods that have been sitting on a shelf for weeks. If I want the best from the people I’m working with then I should give them my best whenever I can. It’s often the smaller details that could be neglected or ignored without issue that are noticed and have a more lasting impact.

Some of you right now are in an environment where the bigger details that are making life difficult for everyone are actively being ignored. You may have been told not to worry about them or that the real solution is for everyone to work weekends. This is garbage and you know it. Please do not change your title to something safe and hide under your desk while in the office. Be the change you want to see in the world. There are lots of opportunities to help organizations make this transition. Find one that’s a good fit for you and make the jump.

My advice for businesses is to be very careful when you describe a position as a DevOps role. We know this to be a very specific thing, but if you’re simply shoving the word DevOps into a job title in an attempt to attract talent there’s a good chance you’re in for a rude awakening. The person applying for that position knows there’s always room for improvement. They know that any transition is going to cause some friction but are willing to work through it. They’re also not scared to rock the boat. So, if your organization is not ready and willing to collaborate as one big team it won’t take too long for your devOps staff to go somewhere else on their own. Threatening their job security in an effort to maintain the status quo will only accelerate the process. You may very well end up with a lower head count at the company picnic as warned, but if it’s because all of your talent moved on to greener pastures you’ve effectively turned a bad situation into a terrible one.

If you’re a DevOp-er that has a work environment in that gray area between heaven and a coal mine I’d encourage you to push the envelope. Share your honest opinion with someone you know doesn’t agree with you. Set up a meeting to discuss conflicting perspectives diplomatically. Drop an F bomb in a conference room if it will wake the room up and get some blood pumping (use sparingly and with caution). Don’t expect people not to get upset. Instead, try really heard to shut up, listen to everything they’re trying to say and do your best to give them what they need. If DevOps is anywhere near your title then you have some responsibility in getting these conversations started. If your current employer didn’t know this when they used DevOps as an adjective then shame on them.

May 23, 2014
DevOps resilience: going active-active with an existing application

What does it take to run an active-active architecture? It boils down to data, and how you interact with it. The most common disaster recovery strategies I encounter with folks are either “hope and pray” on a single location for running an application, or “we have a backup facility we can switch to in a disaster” with an active-passive configuration. This is a shame, as running an active-active architecture is not science fiction, and while it takes some work, for many business use cases the pros greatly outweigh the cons.

The Devil’s in the Data

If you work your architecture from the outside in, an active-active architecture is relatively easy to understand: you have more than physical datacenter or cloud region where you run your application, you have your application running in each, and you have traffic management to load balance users to the closest, fastest and most available location for them. Where folks get stuck is in the database. How do you make sure a change made in one location gets propagated to the others? What happens if a user “bounces” from one facility to the next and attempts to read back data they just wrote? How do you avoid race conditions where user A at location 1 and user B at location 2 make a conflicting change to the state of the application?

The key to handling the scenarios above is understanding the business requirements of your data, and treating each type of data appropriately for an active-active architecture. If you try to “make all data available everywhere”, you’ll quickly run into either CAP theorem constraints or financial constraints on the high costs of this approach. Instead, let’s break the bigger problem down into smaller, more tractable problems.

Three Classes of Data

For each type of data your application accesses or produces, let’s classify it according to one of three types, using a banking metaphor for simplicity (and for easier explanation to your business peers):

1) Is this data like a bank account balance? It changes frequently, and it’s incredibly important to make sure two conflicting changes don’t “break” the balance. We don’t want users A and users B making a withdrawal from two locations at the same time, thereby potentially overdrawing the account.

2) Is this data like your address and mailing information your bank keeps for your account? It changes, but not frequently, and during a change, it’s OK if there’s some short period of time where the multiple locations that host your application are out of sync for this data. They’ll eventually gain consistency.

3) Is this data like your banking statements? Once produced, they don’t change. They can be archived.

Account balance. Mailing address. Historic statement. Three categories, and for each, we have a data replication strategy that can support active-active. By categorizing the data and treating each with a different replication strategy, we’re accomplishing two very important things:

1) We’re being thoughtful about what data REALLY needs near realtime replication and race condition protection. By doing so, we’re reducing the amount of overall data that needs high priority replication. The less data to replicate, the better and more efficiently replication works.

2) Rather than try to replicate all data with the same strategy, we’re willing to use the right tool in the toolbox for each type of data, which means we’ll save costs by using archival and eventually consistent strategies where they make sense.

For each class, we’ll use an appropriate replication strategy:

1) For “account balance” information, we’ll replicate changes as quickly as possible, and make sure our application is aware and capable of resolving change conflicts. This is certainly one of the hardest pieces to get right, but by reducing the scope of how much data we need to address with this strategy, the problem is much easier to solve than by trying to apply this strategy to all three categories of data.

2) For “mailing address” information, we’ll use an eventually consistent replication strategy.

3) For “historic statements” information, we may choose NOT to replicate this to all of our facilities. We may revisit some old assumptions on what’s important to save and what’s not, and decide to discard some of this information. We may choose to simply regenerate the statements should they be lost. We may choose to bulk replicate this data at certain times of the day.

The Top Three Benefits

After we migrate an application to active-active, we can expect three key benefits:

1) Faster and more reliable failover in the event of a disaster. As opposed to an active-passive disaster recovery strategy, which depends on idle infrastructure coming up to full production speed in the event of a disaster, we avoid the risk of “we thought it was ready to handle production but something broke in between disaster recovery drills”. In an active-active scenario, we’re sending production traffic to each location all of the time, so we don’t have an “idle to full capacity” ramp-up problem; instead, we’ll have a “50% to 100%” (for two locations going to one) or “33% to 50%” (for three locations going to two” capacity challenge. We’ll need to make sure we always have spare capacity in an active-active configuration to handle the additional load during a failure.

2) More flexibility for making changes to applications and infrastructure. During a site maintenance event, we can shift all traffic away from our facility, perform our maintenance, and then pull the traffic back, without downtime.

3) Improved user experience. By connecting users with an application instance that is near where they are in the world, they’ll receive a better and faster user experience than if they were to be all sent to a single location.

There are others, but those are the biggest three.

The Top Three Drawbacks

Of course, it’s not all sunshine and kittens, and it’s not without work and investment:

1) You’re going to need spare capacity available, and be diligent and have conviction about maintaining a safe buffer for spare capacity. If you run two facilities each at 75% load, you’re going to have a big problem when one fails. Know your limits, and stick to them. Don’t fall behind in investing in this capacity.

2) You’re going to need to change your application. If you were looking for a “drop in solution” to active-active, this is not it. You’re going to need to change your application. The most common scenario is you’re going from a single database instance on a single database technology to three separate data services that treat each type of data with the appropriate data management strategy. The most common problem is “I used to be able to JOIN these two tables to get a result, but one was an ‘account balance’ type, and the other was a “mailing address” type, and now they’re stored separately”. In this case, your application will need awareness to retrieve the data from each appropriate location, and “JOIN” in software.

This sounds hard and time consuming, but there’s a great strategy for how to implement these changes. Presumably, your application accesses data using a core set of libraries (often an object relational mapping or ORM approach). In these libraries, you’ll want to make the changes for “what type of data is retrieved from where”. This will help minimize the impact to the rest of the application in terms of the interfaces to retrieve data, and keep the “data load balancing” functionality in one location in your codebase. Add in helper functions as appropriate to implement the JOIN functionality you need that previously was done in the database. Keep in mind, if you have a LOT of JOINs across categories of data, this strategy as a whole may not be viable for you.

It’s highly recommended you take a dark architecture approach to these modifications. For some period of time after migrating data to a new strategy, have your application perform reads and writes to BOTH the legacy and new implementations for a single data interaction use case. Compare the values and results, and log/alert when different, while returning the legacy result. This gives you operational experience with the new data strategy while minimizing risk, since the application is still using the legacy implementation for all functionality. After you have confidence that the two approaches are functionally equivalent, you can start using the values from the new implementation, and turn down the legacy implementation. For more details on dark architecture, see here: http://gigaom.com/2013/06/20/making-it-change-less-scary-using-dark-architecture/

3) You’re going to be running multiple database and data storage implementations. There’s a definite cost here in training your team, licensing, infrastructure, and having the monitoring and associated operational infrastructure required to run a dedicated strategy for each type of data. You’ll have to understand for your business whether the benefits outweigh the costs.

There are others, but those are the biggest three.

Your Next Step

How important is uptime, infrastructure agility and user experience to your business? Are these investments worthwhile? Only you will be able to tell for your business. Once you quantify the value of uptime (e.g., how much money do you lose and damage to your brand is there during an outage?), the value of agility (e.g., how much faster could your team evolve your application if they had more flexibility to make changes anytime to production without affecting customers?), and the value of user experience (e.g., how much faster could your user experience be by splitting your infrastructure into multiple active-active locations, and how much would your users appreciate that improved experience?). Come up with those figures, and compare them to the technical investments required above, and you’ll have a pretty clear GO/NO-GO for evolving your application to active-active.

May 7, 2014