devopsy.kinsta.cloud

Tag: sysadmin

Red Hat Enhances Insights to Simplify RHEL Management

This week at its Red Hat Summit event, Red Hat introduced an expanded set of capabilities to Red Hat Insights that simplify management of Red Hat Enterprise Linux (RHEL).

Managing Linux across the hybrid cloud is complex, and it’s made even more so because of current macroeconomic conditions and the challenge of finding and retaining highly skilled DevOps and IT professionals, said Gunnar Hellekson, vice president and general manager, Red Hat Enterprise Linux, Red Hat.

The enhanced Insights capabilities add a layer of abstraction to help reduce enterprise Linux complexity across the hybrid cloud and make RHEL management easier by lowering skill level requirements for managing Linux server estates, the company said.

Using a single UI, entry-level systems administrators and IT support teams can access Red Hat Insights’ predictive analytics to identify potential bugs, misconfigurations or security vulnerabilities and remediate them. Systems administrators of all skill levels can detect, assess and push fixes for potential problems without a deep understanding of Linux management systems like Red Hat Satellite Server and without having to interact with the command line, the company said.

These enhancements help organizations maintain services and systems even in uncertain macroeconomic times when hiring is constrained. Red Hat research also claimed that these enhanced capabilities enabled IT organizations to find IT issues up to 90% faster and remediate them nearly 66% faster across the hybrid cloud.

A Red Hat Insights image builder service also helps lower the skills level needed to manage Linux estates by enabling IT teams to build standardized, optimized operating system images, including the creation of their own ‘golden images’ that adhere to organization-specific needs for security and compliance. These images can be deployed across public clouds, hybrid clouds, data centers and the edge using the same console.

As the economic downturn continues, organizations are looking to do more with less and identify ways to make the most of the resources, talent and skillsets they already have. Red Hat is making the case for adding a layer of abstraction to allow less-highly-skilled IT and DevOps team members to contribute to managing Linux estates across hybrid, multi-cloud and edge deployments.

“The demands of managing Linux across the hybrid cloud are stretching the ability of resource-constrained IT departments to keep pace,” said Hellekson in a statement. “CIOs need to be able to extend existing skillsets and lower the skill barriers for overseeing Linux estates, which is exactly what the management capabilities of Red Hat Insights are designed to do.”

May 24, 2023
Ops: The Other DevOps

Much of the conversation around DevOps, including right here on devopsy.kinsta.cloud, is about helping developers get easier access to production systems, increasing agility by shortening release cycles. Most of the time, this focus on developers leads to a concentration on so-called “Day One” tasks, the Dev side of the equation, to the exclusion of “Day Two” operations—the Ops side of things.

Photo via Damien Kuhn – https://unsplash.com/@producteurslocaux

Recent moves in the DevOps market underscore the fact that the weight is still very much on the Dev side, and not so much on Ops. Microsoft buying GitHub is huge news, don’t get me wrong, not least because many people are still stuck in a very Nineties conception of Microsoft, and this acquisition should help clarify that this is not your parents’ Microsoft (full disclosure: my employer, Moogsoft, has a strategic partnership with Microsoft). News on the Ops side of the house, meanwhile, rarely achieves the same prominence.

The reason for this imbalance is to be found in DevOps’ origin myth. The story is all about developer frustration with risk-averse operations procedures which prevented them from deploying as rapidly and frequently as they would have liked to. As with most good stories, there is a kernel of truth here, but the truth is more complicated. Ops people are risk-averse for good reason: When they mess up, they don’t have a safety net, and they get a lot of very public blame.

That said, it is certainly true that a reasonable aversion to risk can be taken too far. Couple that with a very human tendency to define oneself by one’s job, and you wind up with people putting hands to keyboards for even basic, routine tasks, becoming bottlenecks for huge processes.

What Does DevOps Mean To Ops?

The biggest benefit of DevOps from an Ops perspective is that it has forced that hard conversation about moving from artisanal, handcrafted IT infrastructure, to industrial and scalable models that don’t require a human in the loop for routine operation. This cultural shift began with the famous pets-versus-cattle analogy, but the transition is far from being complete.

A big sign that there is still work to be done is the relative lack of sophistication around tooling in IT Operations. There is a lot of talk about software-defined infrastructure and desired-state automation, but relatively little practical uptake, especially when it comes to operating such frameworks day-to-day. These new models will require a wrenching transition from traditional Ops models, with pervasive automation working to augment human sysadmins’ capabilities.

This is not the old style of automation, with cron jobs and shell scripts, which any BOfH worth their etherkiller has been doing for decades. The new serverless frameworks do away with the last vestiges of persistent coupling of infrastructure components to individual business services. What vMotion promised in 2003 is now coming to pass, with workloads moving, growing and shrinking to support changes in user demand, and infrastructure reconfiguring itself top to bottom in response.

Key Questions To Ask About Operational Automation

In this context, operational automation can no longer work on the old model of “set it and forget it.” Every process set in motion needs real-time contextual awareness of what is going on around it and how that is constantly changing. Without that visibility, backup scripts may go on faithfully backing up what used to be a high-priority server, but is now a backwater of the application infrastructure. Meanwhile, the cloud services where the bulk of the application’s logic runs are not meaningfully integrated into the disaster recovery (DR) plan.

All of these facets need to be reviewed regularly, in terms of need, function and ownership.

Who Needs This Automation?

Anything that runs should be linked to some kind of business requirement, however indirectly. “This ensures that our infrastructure can scale and flex seamlessly in response to user demand”: Good. “This keeps our operating systems patched”: A valid goal, but do we need to do it in production if the average lifetime of provisioned operating systems is shorter than the patch cycle? Just patch the templates and you’re done. “We’ve always done it this way, and I’m not sure what might happen if we stop”: Time to set some chaos monkeys loose, and see what happens.

What is the Function of This Automation?

We are in the middle of huge technological transitions, and the waves of innovation keep coming closer and closer together. As an industry, we had just about come to terms with virtualization by the time cloud computing came along, but that was far from settled when containerization got underway, and it looks like that may be undermined by serverless architectures before it even comes to its full fruition. Watch out for hidden assumptions in terms of the way you go about a task, and check back in regularly. Ops types should take a leaf from Dev’s book, and regularly consider refactoring their automation to remove cruft and take advantage of new developments.

Who is the Owner of This Automation?

Someone has to own every piece of automation, so that they can maintain it over time according to the previous two points, and so that everyone else knows who to call if (or realistically, when) it breaks. Again, developers have been talking about technical debt for a long time: the impact of older, unmaintained code. Meanwhile, sysadmins used to pride themselves on the uptime of individual servers. That attachment needs to move from the individual server to the overall infrastructure and the automation and management that develop over time to ensure its availability and performance. They’re cattle, not pets, remember?

Ops is the Other DevOps

As Ops becomes more and more about managing the automation, the distinction between Dev and Ops starts to blur. With Dev roles “picking up the pager” and taking responsibility for production operation of their software, Ops roles develop more and more code to fulfill the promises of those applications. With traditional Ops tasks evolving or even disappearing, new roles will appear. The job of automation architect will become much more relevant than any job that requires someone to ssh into an individual server or router. In this transition, old-school Ops people can learn a lot from what our Dev colleagues have been up to over the last few years.

One day, the term DevOps will no longer describe developers and non-developers, but two different types of developer, one focused on business logic and one on architectural underpinnings.

— Dominic Wellington

July 10, 2018
AIOps: Managing the Second Law of IT Ops

Or, why IT systems keep breaking

The tension between the two halves of DevOps is long-standing, going back long before the term was even invented. What underlies the split between Development and Operations is a fundamental difference, like an actual physical law. Let’s explore that analogy together.

IT Ops is fundamentally a constant fight against entropy, meaning disorder. Sysadmins expend significant energy trying to keep everything up and running, in a process which can sometimes feel like trying to shore up a sandcastle while the tide is coming in. Systems are constantly breaking down; users need changes made that introduce risk; and, as if that wasn’t enough, the whole shebang is constantly under attack from malicious outsiders. It’s no wonder that sysadmins have a reputation for being a bit grumpy!

It’s not just a good idea, it’s the law.

In contrast, Dev actually increases the level of entropy. Every new release, be it as small as a one-line fix or finding that errant semicolon, is a change that has to be made, reducing the order that Ops fights so hard to maintain. Dev is entropogenic, embracing change as a positive force.

The interaction between Dev and Ops is analogous to the second law of thermodynamics: Entropy can only increase over time, unless energy is introduced to the system from outside. In other words, unmanaged change will eventually cause something to break, unless people put in continuous effort to prevent those breaks.

Thermodynamics: Lots of Heat, not Much Light

These conflicting goals, taken to extremes, can get unhealthy, with Ops actively fighting change or Dev going around Ops to increase their velocity of change. Left unchecked, this tension can turn into a vicious cycle, as frustration with the “Department of No” drives developers into unmanaged “shadow IT.”

The good news is that the analogy with hard laws of physics is not perfect. The process of increasing entropy is not irreversible, and does not require constant inputs of energy to prevent. New agile approaches to IT Ops are emerging that are light and flexible, taking their structure from that tension. In the physical world, this is called “tensegrity,” and was originally formulated by Buckminster Fuller, one of those annoying polymaths who seem to make breakthroughs in every field they touch.

Tensegrity structure

IT Ops needs its own equivalent to tensegrity, a flexible, holistic structure that is able to embrace change while maintaining its shape. Structures that try to resist change through rigidity are brittle and ultimately fragile.

Today, this flexible strength is enabled by AIOps, a term coined by Gartner to describe a new way of operating IT, based on augmenting human skills with automated analysis enabled by modern algorithmic and machine-learning approaches.

Complexity Breeds Complexity

Modern IT is too complex for any single person or even team to keep track of, instead requiring multiple specializations. In turn, each of these functions requires its own specific support tools that enable that particular role. Beyond a certain point, it becomes very difficult to combine these partial views to gain a holistic understanding of what is actually happening in the environment.

Because of the scale of modern IT environments and the sheer rate of change that businesses require, old-style rules-based “Managers of Managers” can no longer keep up, requiring more effort to build and maintain their worth. Especially as infrastructure becomes more and more self-modifying in response to changing conditions and demand, IT Operations needs to accelerate just to keep up.

AIOps brings developing issues to the attention of sysadmins before they become visible to customers and end users, and gives them the tools to collaborate effectively between different teams and specialist roles to avoid outages or restore service quickly and efficiently. In this world, automation is used to augment the capabilities of the human operators to make them more effective, letting people concentrate on adding value, not just fighting fires.

This is how Ops can get out of the business of fighting Dev, and instead become a supportive partner. AIOps can catch issues that are affecting services and flag them to the right people early on, including the DevOps engineers that are responsible for production support.

For developers, one of the fears around DevOps is that in “picking up the pager,” they are opening themselves up to a constant stream of irrelevant alerts. Old Ops hands nod wryly at this, or maybe try to share their personal approach to filtering the noise. Instead, AIOps ensures that only valid, actionable alerts get as far as being shown to a human being and taking up their valuable time.

Eyes on the Prize

It’s not all roses. As the AIOps market begins to mature, there is a certain amount of “algorithm-washing” going on, with more and more vendors trying to claim that either their legacy tool was AIOps all along, or that their one-trick pony is actually a whole three-ring circus. What gets lost in the excitement is what all of those algorithms are actually supposed to achieve.

Me too!

In considering a shift toward AIOps, the goal should not be simply to build a better mouse trap, meaning a smarter filter for monitoring events. The success of the project can be measured not so much in the mathematical results of the algorithms, but in the smoother exchange between the human users. This easier collaboration is especially important across team boundaries, whether these be different specialist units within the Ops function or developers and software engineers who need to work with Ops to understand the performance of their code in production.

The benefit of transforming operations this way—moving from sequential, waterfall Ops to abile, collaborative Ops—is to turn DevOps-enabled teams into a sort of perpetual-motion machine, able to continue operating without requiring constant massive inputs of energy from outside the system. Modern applications and the complex infrastructures that support them cannot be managed with the sorts of rigid, cobbled-together, industrial approaches that were appropriate in the mainframe days. AIOps is the agile, flexible approach that can not only accomodate that complexity, but thrive on it.

— Dominic Wellington

September 22, 2017
SRE vs. DevOps — a False Distinction?

Just a few days before he died at the beginning of the 1990s, a wise man taught us that “the show must go on.” Freddie Mercury’s parting words have long provided the guiding light for many, if not all, ops teams. In their eyes, the production environment should be exposed to minimum risk, even at the expense of new features and problem resolution.

About 10 years ago, Google decided to change its approach to production management. It took the company only a few years to realize that while R&D focused on creating new features and pushing them to production, the Operations group was trying to keep production as stable as possible—the two teams were pulling in opposite directions. This tension arose due to the groups’ different backgrounds, skill sets, incentives and metrics by which they were measured.

Trying to bridge this gap between the two groups, one of Google’s ops leaders, Ben Treynor, thought of an innovative solution. Instead of having an Ops team built solely from system administrators, software engineers—with an R&D background and mentality—could enrich the way the team worked with the development group, change its goals and help with automating solutions.

Site Reliability Engineering, or SRE

And so, the Site Reliability Engineering position was created. According to Google, SRE engineers are responsible for the stability of the production environment, but at the same time are committed to new features and operational improvement. Google decided its SRE teams should be composed of 50 percent software engineers and 50 percent system administrators. The engineers were driven to use software as a way of solving problems and perfecting what had historically been solved by hand. They integrated easily with the development team, and encouraged code quality improvements and automation testing.

Understanding the value of SRE, several organizations of various sizes decided to embrace its principles. Some, such as Dropbox, Netflix and Github, are well-known for being at the forefront of technology leadership.

Wait, Isn’t That DevOps?

DevOps is a more recent movement, designed to help organizations’ IT department move in agile and performant ways. It builds a healthy working relationship between the Operations staff and Dev team, allowing each to see how their work influences and affects the other. By combining knowledge and effort, DevOps should produce a more robust, reliable, agile product.

Both SRE and DevOps are methodologies addressing organizations’ needs for production operation management. But the differences between the two doctrines are quite significant: While DevOps raise problems and dispatch them to Dev to solve, the SRE approach is to find problems and solve some of them themselves. While DevOps teams would usually choose the more conservative approach, leaving the production environment untouched unless absolutely necessary, SREs are more confident in their ability to maintain a stable production environment and push for rapid changes and software updates. Not unlike the DevOps team, SREs also thrive on a stable production environment, but one of the SRE team’s goals is to improve performance and operational efficiency.

Google tried a few approaches to implementing the SRE process before finding the one that suited them best. One of these approaches attempted to tie the number of permitted releases to the product’s stability. The principle underlying this process is that new releases are green-lighted based on current product performance.

For each service, the SRE team sets a service-level agreement (SLA) that defines how reliable the system needs to be to end users. If the team agrees on a 99.9 percent SLA, that gives them an error budget of 0.1 percent. An error budget is exactly what the name suggests: the maximum allowable threshold for errors and outages. Here’s the interesting thing: the development team can “spend” this error budget in any way they like. If the product is currently running flawlessly, with few or no errors, they can launch whatever they want, whenever they want. Conversely, if they have met or exceeded the error budget, and are operating at or below the defined SLA, all new releases are frozen until they reduce the number of errors to a level that allows the launch to proceed. This process ensures that both the SREs and developers have a strong incentive to minimize the number of errors in production.

Another interesting approach recommended by Treynor, which is more related to SRE professionalism and efficiency, is allowing SREs to move between projects. Moreover, he suggests allowing SRE engineers to move to development, and even the other way around. As the work done by both teams is similar, these transitions help the Ops team gain better and deeper knowledge of the product and code, and bring the Dev teams into the production space to help them understand its challenges. This strongly promotes a team atmosphere, rather than one in which an individual feels that “I’m on the SRE team for this product.” As part of this approach, Treynor had Dev teams handle 5 percent of the operations workload. This, according to many organizations, adds to the SRE team’s motivation and effectiveness.

Riddles in the Dark

Well, SRE seems perfect. As already stated, several large-scale organizations have chosen to move some of their production operations from old Ops to SRE. However, there are still a few questions that need to be asked:

Is the position of SysAdmin (production/ops) no longer relevant for SysAdmins?

Historically, almost all system administrators have come into their roles through tech support and similar work, or even just running Linux on their desktops and then transitioning into server work. It should be pretty clear that the same path is not available into SRE. To retain their positions, SysAdmins should now be more code-oriented, have better technological knowledge and be receptive to new methods of conducting the work they already do.

Can an SRE team prevent production incidents?

One of the strengths of a professional site reliability engineer is the ability to handle the growth of production load and traffic in-house. Monitoring and analyzing processes and logs with platforms such as the ELK Stack is part of the day-to-day workload, and the team should be able to identify problems as they occur, and even foresee risks to software stability. The power of this position—based on the skill set it requires—lies in developing solutions for these problems and risks. As Ciara, a software engineer in Google’s cloud storage SRE team, described it in a “Life in Google” post, “We solve cooler problems in cooler ways.”

Are we doing DevOps or are we doing SRE?

This question is usually asked by people trying to position themselves in the Operations world. According to many companies that implemented SRE in a slightly different way than Google, you don’t have to decide. At Reddit, ops engineers work on reducing toil, improving deployment and scaling processes, but they are referred to as “DevOps.”

At Logz.io, we bridge the gap between developers and production through the use of automated monitoring and performance stress-testing. And we personally refer to it as “DevOps” rather than “SRE.”

So, is it Just a Question of Context?

In her blog, Charity Majors talks about SRE and DevOps being two different operational approaches that any organization can choose to work by, but insists on emphasizing that there is no “correct” approach. Although Google and Dropbox have decided on SRE, this does not mean the rest of the world should do so as well. What fits Google’s needs and organizational philosophy does not necessarily work for other orgs—at any scale. Moreover, Charity believes that DevOps, having grown and evolved within a broader variety of software organizations, is the more flexible, collaborative and adaptable approach, and will work better for most software organizations at all stages of development.

On the other hand, SoundCloud senior engineer, Matthias Rampke issued a series of tweets on how SRE and DevOps were basically the same, with only one difference—high management support. Though in contradiction to her blog, Charity lent weight to this opinion during an AskMeAnything event that revolved around DevOps and SRE. She shared a story about a friend who was hiring for a startup, which, as an experiment, posted the exact same job description twice, the only difference being that one listing was titled “DevOps engineer,” and the other “SRE.” At the time of relating the story, DevOps was winning by 10 percent or 20 percent. However, all this means is that the job title is irrelevant, and naming an SRE team does not magically bestow qualities upon it. Rather than focusing on the title, organizations must focus on the work being done.

To summarize, we can quote Matt Simmons, a technologist with many insights into this topic, who says, “Not every infrastructure needs an SRE, but every infrastructure could use an administrator who acted more like one.”

— Asaf Yigal

May 18, 2017
‘Admins’: The Movie

In the IT industry, where we are exposed to various levels of incompetence, it is quite tempting to become a little cynical. A few years ago we hit rock bottom. Our patience—not to mention our sanity—lay in the balance. Like many of you, we sought refuge in laughter. We found comfort and comic relief through Kevin Smith’s vulgar yet oddly accurate low-budget movie “Clerks.”

Now, it’s time to pay it forward. As longtime fans of Kevin Smith, we wanted to honor his work by creating an IT spin of a very similar story. That is the genesis of “Admins.” It’s a day in the life of a disenchanted government contractor surrounded by incompetence, idiots and debauchery.

The story centers around Dan, a system admin working for a government contractor, who is forced to come into the office on his day off. His day gets progressively tougher as he discovers mounting technical issues and the reality that his position may be in jeopardy due to cutbacks. Fearing unemployment, Dan sidesteps additional contract work and pursues the holy grail of guaranteed lifetime employment (a.k.a. “a government job”). The allure of job security leads Dan to the crossroads of an ethical dilemma where he and his co-worker, Randy, engage in morally questionable and professionally reprehensible activities.

About the Authors

AARON GOODMILLER – Director | Producer

With a lifelong passion for movies, Aaron got into film making in 2001 when digital video had just started to take off and it became something he could afford to do. Having written and directed three shorts and been a producer on 11, he worked with Eric Espejo on producing their first full length feature together “Ghosts Don’t Exist.” Their current film, “Admins,” is something they have talked about doing for several years. After this they have several other projects is various stages of production.

ERIC ESPEJO – Writer | Producer

Eric has been involved in film since 2002. After writing and directing eight short films, he wrote and directed his first full-length feature, “Ghosts Don’t Exist,” in 2009. He continues to oversee a number of creative projects all while writing screenplays and serving on the George Mason University advisory board for its Film and Video Studies. Eric started delving into indie film by attending the Hollywood Film Institute, where under the tutelage of Dov Simens, he acquired the valuable knowledge of the artistic and business sides of filmmaking. An avid and talented screenwriter, he is currently writing his next movie.

April 22, 2016
George Jetson Automation: Chasing the Dream

“His job was one simple thing—press a button. That’s all he had to do; press one button and sit back.” – Joe Barbera

I wanted to be George Jetson when I grew up. Back in the days when watching Saturday morning cartoons in footed pajamas was the high point of my week, George’s life looked as sweet as a bowl of Super Sugar Crisp cereal. Flying cars, robots, and a two-hour-a-day, three-day work week lightened even further by 100% push-button automation.

Contrariwise, my first tech job out of high school was a classic study in manual labor. From 9pm to 5am, I was solitary “Night Backup Operator” for a small data storage company, lone man responsible for continuously swapping reel-to-reel tapes on and off a literal wall of old magnetic drives.

It was repetitive, monotonous and boring, and the tapes were prone to jamming and tearing. A lot.

One night I was thumbing through a tech catalog and came across a listing for a new cassette backup system. I mentioned to my boss the following day that our reel drives could potentially be replaced by a unit the size of a large book. He took it to heart, made a purchase, and one evening shortly afterwards I came to work, popped in a blank cassette, hit a button, and kicked back for the next eight hours as 100% of my work was done for me. It was a George Jetson moment. The future had finally arrived. I was blissfully complete.

They fired me, of course. It actually took them a few weeks to realize that I’d been automated completely out of a job. To their credit, they were appreciative and regretful in parting, but there was hardly any point in paying me to sit on my can reading pulp SF for eight hours every night.

Regardless, I’ve been chasing that George Jetson moment ever since.

The magic behind the button

In the decades since, with a number of (slightly more professional) IT automation projects behind me, I’ve discovered that the “automate-everything” button is usually a lot tougher to achieve, and near-impossible to buy off the shelf. To even get close, you often end up building a significant amount of infrastructure up from scratch.

But say you could buy a “George Jetson” button; a self-contained, perfect automation solution off the shelf; what do you look for? Setting aside the nuts and bolts of a specific IT problem or project, what does an “automate-everything” button need to do, in essence?

1.   “Do No Harm”

At the first sign of an incident, Mr. Spacely (voiced by Mel Blanc, “Man of a Thousand Voices”) pops up and begins motivating with the bellow “JETSONNNNN!”. (I once worked outage calls with an IT director who had the same management style, I kid you not). We don’t want this to happen. So our “automate-everything” button needs, first and foremost, to mitigate risk.

Security should be the first consideration in risk management, yet it’s an area where (oddly enough) a number of automation solutions fall flat out of the gate. Managed user accounts (ideally, integration with established account management systems), role-based permissions (for pushing the button in various situations, IT environments, and scheduled times), and sensitive information handling should all be covered.

The ideal automation solution’s back-end infrastructure should be robust, flexible, and professionally engineered. If any pieces are open-source or community-designed, what they do and don’t do should be well-documented, including an understanding of potential situations likely to result in unintended consequences.

We also want the automation button, once pushed, to never perform the action again if there’s no need to do so. This is idempotence, a valuable concept that’s been around for a long time but has risen to prominence over the last year. Idempotence is particularly important once you start to consider auto-remediation (why push the button manually if it can run automatically?), and efficiency (why spend effort building an automation action that can only be run once?).

2.   “CYA”

“Cover Your Automation.” George isn’t particularly smart for a computer operator. But maybe he doesn’t need to be. Let’s say he’s supported by a lower floor full of smart engineering robots. The robots set up the automation actions and push them into an approval queue with just enough information to make a decision on whether an action is needed or not.

George evaluates each situation and pushes the button if everything looks good to him. Now, the “automate-everything” button becomes more of an “approval” button in a formal change management system. There might be more approvers besides George, each laying eyes on the proposed automation action and passing it along to the next approver as appropriate.

Most operators aren’t as dumb as George, but everybody has “dumb moments”. Change management needs to be a core part of any automation system.

Accountability supported by thorough logging is also a key “CYA” aspect of the ideal automation system. “What” happened, “when” did it happen, “who” did it, and so on… if your automation system doesn’t track all essential information, you might not be able to return to your job after Spacely mistakenly fires you.

3.   “KISS”

“Keep It Simple, Stupid.” Dealing with complex automation workflows is tricky if they involve a lot of step-by-step actions, as they traditionally have in the past. When complexity clashes with modern, large-scale IT environments, situations arise that are literally impossible to manage manually. “Scale x Complexity > Effort” is the equation desperately slashed across whiteboards in planning meetings these days. Modern automation systems often attempt to reduce complexity and equalize scale through a form of orchestrated states.

The idea depends on two parts. A desired state is a recent Microsoft-championed concept where the steps required to reach an end result are black boxed in favor of defining a system’s state itself. The usual example: rather than considering a step-by-step recipe for a birthday cake, you describe “a white cake with pink frosting” as your desired outcome and pop your request into an automated system to make it happen. Recent configuration management solutions have made excellent headway in this area.

But desired states aren’t enough in and of themselves. They almost always need to be orchestrated to reach the final goal. An orchestrated states effort runs more along the lines of: “I need a white cake with pink frosting at the same time as a bowl of orange punch, before the time of the birthday party, but after the house cleaning. Also, yellow balloons.” Real-life IT infrastructure requires at least this level of complexity and timing.

For a complete “automate-everything” solution, our ubiquitous button needs to orchestrate the states of all necessary systems and services in a way that’s easy for George, our hapless operator, to process, and considers both transitory states and final states as well as the interactions between them.

When evaluating whether an IT automation solution is right for you, you may not be able to find the ideal “automate-everything” button, but make sure you consider it from these three perspectives. The perfect solution may, in the end, just be a dream to be chased by the laziest as we move from one IT automation project to the next. But it sure is fun seeing how close we can come to that blissful, futuristic George Jetson moment.

April 27, 2015
Service virtualization – The hows and the whys.

Close your eyes and take a deep breath. You are on an island, smelling the iodine coming from the sea, the sun is gently caressing your cheeks and you are sipping on a cocktail in the middle of the afternoon. Your phone is off, in a place you forgot and you do not have direct access to the internet. You are a sysdamin and for the first time since the beginning of your career, you take a break, an actual break to reload your batteries and you are not even thinking about work because you KNOW. You know it is all going to be alright, as it has been since you implemented service virtualization in a really DevOps oriented way.

What is service virtualization?

First of all, let me say that service virtualization is not something new but many are still unaware of what it actually is and only a few good products are available.

You can find a really good explanation of what it is in this post from Chris Riley: https://devopsy.kinsta.cloud/blogs/service-virtualization-window-advanced-devops/ but I will try to sum it up.

In order to avoid confusion, it is also necessary to briefly explain mocks and stubs. Mocks are usually programmers written tests that check the behaviour of a specific piece of code like which methods are called, in which order and if sub processes were created. Stubs on the other hand check that for any given input, the expected result is outed. I often refer to them as logic tests and I/O tests.

Service virtualization has nothing to do with mocks or stubs. It mimics the entire behaviour of an application as if it was an actual one deployed on a server and connected to a database, backend etc. In order to enable this, data will be recorded from QA (and anonymized) and the whole behaviour of the application will be recorded. Going through security layers, calls to a database, updating some values, even timings can be recorded for performance testing. Absolutely every single operation is stored and if the test data is well chosen at the time of recording, all cases can be covered.

For those of you who are into gaming, think of it as an emulator. You do not need a SNES, Genesis or Neo Geo to play your favourite oldies, instead, all you need is an emulator software and the ROMS. The emulator will behave like a gaming console on your computer, reproducing inputs and outputs with high fidelity and translating CPU instructions into different ones your computer can understand.

If you are not into gaming, and that is a shame if you ask me, think about virtual machines. When you install a Fedora in a confined environment on top of your favourite edition of Windows (or Mac OS), what you do is pretty similar to service virtualization as well. Your Guest OS will send instructions to the different components of your computer which the virtual machine will catch, translate into commands your Host OS can understand and send to the concerned components. It sounds complicated but it works like a charm.

What it can do for you.

As it operates on local servers or even better, on containers (check out Docker if you have not done so yet), it is virtually always on. Downtime is a thing from the past and updates can be scheduled and performed when people are not at work. How much of your project you want to cover is up to you and your needs but once you have started the process, why not go all the automated way? If you do, that means that your programmers will have access 100 percent of the time to environments that are up, running and that reflect actual services.

What is the point of having test environments then you might ask. And here is the answer you can give stakeholders: none. You can achieve good environment resilience, actual data tests and higher-than-ever response time through service virtualization. If well applied, and I really want to emphasize on that, service virtualization makes good ol’ test-environments obsolete.

Once again here, the benefits mentionned by Chris Riley will speak for themselves: https://devopsy.kinsta.cloud/blogs/service-virtualization-window-advanced-devops/

Taking DevOps one step further.

Like all great things, service virtualization comes at a cost. In order to fine tune it to your requirements, it is going to take great efforts. You have to decide which services, ideally all of them, you will provide your programmers, which data sets are representatives of a functional application and of course, install and configure it all. The biggest benefit that you can identify here already is that developers, testers and operations have to be involved and work in a close collaboration. This is your opportunity to make different teams understand what others do. Get them all together, make them work together in a form they are not necessarily familiar with: cross-teams brainstorming.

Brainstorming, if well done, can be a very powerful tool to come up with creative solutions to problems that were not identified as such beforehand. It has to be directed though. To brainstorm doesn’t mean talking nonsense about non topic related things. Having a neutral person in the room who focuses the discussion around a few pre-established points and who keeps track of time is the way to go. Scrum masters are generally good at this since they have an overview of the project and its problems and can quickly identify the right persons for a task.

Getting developers to update their own service virtualization server with the latest data from QA means better communication with testers. Operations have a critical role here too since they setup the environments and give the how-to to the different teams.

Another glass sir?

Once your teams work together on an almost 24/7 available server and understand how it works, all you have to do as a sysadmin, is to relax a bit for once and focus your brain and time on how to improve processes even more, enhance the quality of deliverables and tighten up the now-existing bonds with your colleagues. Oh yeah, and if you’re on holidays, stop worrying about that potential catastrophic phone call, just worry about yours and yourself.

March 20, 2015