devopsy.kinsta.cloud

Category: Enterprise DevOps

Will Automation Fill Gaps Left by the ‘Great Resignation’?

Between burnout, job-hopping and the ongoing pandemic, labor shortages will surely get worse before they get better, and automation is increasingly sought after as a way to mitigate the ripple effects of understaffed IT departments.

For DevOps, automation has several benefits, from freeing up employees to work on more constructive tasks to minimizing software bugs and technical debt.

However, starting off slow and learning how automation will help address staff shortages without complicating processes is a critical factor.

“There’s still a lot of teams that are struggling with far too many manual or repetitive tasks,” said Rich Lane, chief strategy officer at Netenrich, a digital IT and security operations company. “Obviously, trying to do your day job of delivering features and functionality into the wild can be challenging with the speed that business needs demand.”

Lane, a former senior research analyst at Forrester Research, pointed out that slowing down in the present to do foundational work that can lead to higher levels of automation in the future can be hard to sell to digital services stakeholders.

“And yet, without doing so, it becomes increasingly difficult to meet commitments in the event a team becomes shorthanded or delivery requirements increase,” he said.

Generalist or Specialist?

Cody Michaels, application security consultant at nVisium, said when it comes to hiring people for a DevOps environment, you have one of two options: You can hire someone with a wide range of knowledge that spans the entire process or you can hire someone with expertise in one core area like programming or deployment.

“At an individual level, a generalist will be better than an expert, however, a team of experts will outperform a team of generalists,” he said. “Not only can automation fill in gaps between an expert’s area of influence, but it can also serve as a permanent fixture on the team. As a result, the turnover rate won’t affect any automation so long as it’s implemented properly.”

From Lane’s perspective, the main areas DevOps teams should be looking to automate are continuous integration and continuous delivery (CI/CD), IaC and AIOps-enabled incident management platforms.

“By taking the manual nature of day-to-day work off of DevOps engineers’ plates, they are freed to focus on digital transformation,” he said. “The number-one stumbling block is not starting with process.”

Lane noted unless you understand all the steps in a procedure that you’re trying to automate, it is very difficult to maximize the power of automation tools.

“Much of the process that is still adhered to today is outdated for the digital age,” he said. “Spend the time up front to map out what you hope to achieve with an automation project, what all the touchpoints are and how one can measure the quality of automation when it’s implemented.”

Michaels added that while the internet is flooded by companies shouting they have the “best” tools, that proclamation of “best” is going to be determined by budget and known languages.

“The top tools that I would recommend are Jenkins, Docker, Git and Jira, since these four play very well together with regard to automation,” he said.

Git, for example, has hooks that can trigger upon performing certain actions, like freshly submitted code, while Jenkins has automation as its middle name and it’s used in building, testing and deploying, which in turn facilitates CI/CD.

Meanwhile, Docker provides virtualization of operating systems and running applications in a secure manner in conjunction with containers.

“Lastly, Jira is one most don’t think about, but the platform has a lot of macros and filtering functionality that can update the work process automatically from events like code passing a test before it’s deployed, for instance,” he said.

Michaels said a couple of core considerations organizations need to make when approaching the idea of automation and AI are time as well as safety nets in place.

“Automation will take time to set up before you can benefit from it, but in the case of smaller teams, it won’t be too time-consuming to configure it all in one fell swoop,” he said. “But if you’re working with a lot of technical debt, unfortunately, it’s going to be payback time. If this is the case, my best advice would be to start small and not overhaul everything at once.”

Minor Automation Improvements Have Major Benefits

He said he thinks incremental improvement will be the best route for most organizations, and that even minor improvements will compound far faster than waiting a year to see if a redesign will work for you.

“As I mentioned previously, it’s imperative to have safety nets in place,” Michaels said. “Make sure you have a contingency plan if something goes wrong in the process to recover back to a working state. However, strictly speaking, everyone should make it a practice to have a recovery plan in their software development life cycle (SDLC).”

As the DevOps world moves into 2022, organizations choosing to embrace automation could find themselves in a win-win situation—for both developers and operations.

“The more automation takes over the ‘housekeeping’ items of the process, the more time will be freed up to improve the process as a whole,” Michaels said. “As a result, taking time to go back and clean up technical debt can now be set aside between sprints and maintenance.”

He pointed out that senior roles can tackle optimizing the architecture, which will ensure fewer bugs and a healthier work-life balance for all: Just imagine a world where developers can focus on perfecting their craft—instead of waiting hours or days for server reboots or disrupting their flow to update a ticket status.

“It really comes down to the fact that organizations can no longer make automation an afterthought,” Lane added.

He pointed out that given the difficulties in recruiting and retaining staff today, DevOps leaders must make their teams as efficient as possible—and automation can help achieve that goal.

“The number-one way to achieve this is applying as much intelligent automation throughout the IT environment as possible so that the highly-skilled engineers can focus on transforming the business,” Lane said.

December 1, 2021
GraphQL as a Meta Layer
There’s a lot of interest in GraphQL lately. The query language is excellent for data retrieval, and it could act as a more usable mechanism to evolve the traditional REST API integration approach. But as adoption grows steadily, architects are now considering the implications of managing GraphQL at scale.

Certain issues may hinder GraphQL in practice. To review, security concerns require a nuanced approach to authorization. This is because it wraps multiple data retrieval points into a single request. Without native caching, performance could also suffer. Changeability is also required to continually update the graph, which may ruffle feathers with those seasoned in the world of REST where a guiding tenet is to build for longevity using web-compliant methods.

According to Jens Neuse, CTO, WunderGraph, there is a simple way to mitigate most of these issues and harmoniously work with various API styles. To Neuse, the core problem is that GraphQL is not meant to be exposed over the internet. Instead, he advocates for interacting through a gateway. Such an intermediary would enable more implicit calls and proper authentication and authorization for specific underlying functions and resources. Using GraphQL as an internal integration layer could also help combine disparate APIs into a similar language, exposing GraphQL-like usability for frontend developers while allowing them to hit any API style on the backend.

Below, we’ll consider the drawbacks of exposing GraphQL directly and see how an intermediary layer could benefit development. We’ll also uncover what open source software is on the horizon that could power this setup.

Why GraphQL Shouldn’t be Exposed Publicly

Facebook, the creator of GraphQL, uses an internal GraphQL API to power all of its apps across Android, iOS and web browsers. (This is not to be confused with their public Graph API). Interestingly, if we look at Facebook’s use of GraphQL, we see these endpoints are not exposed to the public web at all.

It’s true. Open up Developer Tools in Chrome, and you’ll see there is not a single mention of a root HTTP endpoint like https://api.graphql/. Functions name GraphQL, sure, but they call it more implicitly. Other companies, like Twitter and Airbnb, use a similar abstracted approach to interact with a private GraphQL.

In fact, most GraphQL APIs are not public. APIs.guru only charts 35 public GraphQL APIs—notable services include GitHub’s GraphQL API, Shopify Storefront and GitLab. This number is minuscule compared to the total number of public APIs, at over 24,000 according to ProgrammableWeb.

A Layer In Between

If we treat it as Facebook intended (and how they use it in production today), it could reverse the issues above, said Neuse. “These problems come down to our core belief that GraphQL is not supposed to be exposed over the internet,” he said.

But if clients do not access a GraphQL endpoint, what are they interacting with? According to Neuse, the ideal architecture involves a layer between the client and the server. Such a layer would validate queries and represent verbose calls with simple functions. The process would involve a handshake as follows:
1. Take query, store it on a server.
2. Validate it; make sure it’s not malicious.
3. Send back a handle to the client.
4. Then, execute the query.
Such a process could also enact universal policies on a per-function basis, thus solving potential authorization issues when fetching data from multiple backend services with a single query.

“The issue is that using the query language comes with a cost,” Neuse said. “Not exposing GraphQL directly but ‘compiling’ it to a JSON-RPC API when deploying solves these problems.”

Thus, the scalability issue isn’t so much an issue of size but of access. “GraphQL can be very valuable, even in small environments,” said Neuse.

The Next Evolution of a Virtual Graph

Another potential benefit of using GraphQL as an intermediary is combining APIs; a “virtual graph,” as Neuse calls it. This virtual graph offers a common integration language to interact with various API styles, whether REST, GraphQL, gRPC or webhook, in a GraphQL-like manner.

Consider the ease with which package managers like npm enable developers to instantly install dependencies. For example, installing a package like React.js is as simple as initiating the command npm install react within Terminal.

On the other hand, the API integration is very cumbersome. Users must read through documentation, create an account on a third-party site and download an SDK to call the service in their preferred language. So, why isn’t looping in dependencies like APIs and cloud services as simple as other common dependencies?

According to Neuse, one reason is that the REST paradigm is obsessed with the resource constraint—aside from HATEOAS, it lacks a concept of relationships between data and fields. A virtual graph could enable a straightforward process similar to package managers, said Neuse. Interestingly, the WunderGraph team plans to open source a project that allows just that ability.

GraphQL as a Meta Layer

Many folks assume that GraphQL and REST are mutually exclusive. However, as we’ve seen, there are ways to use them in combination. Plus, with an abstraction layer, the REST versus GraphQL comparison becomes just an underlying technical nuance.

“If you compare GraphQL to other specifications, like OpenAPI spec, you see GraphQL has the fastest-growing community,” said Neuse. As of 2021, JSON Schema is the most popular, used by 75% of developers, followed by Swagger 2.0 (54%), OpenAPI 3.0 (40%) and GraphQL (31%) according to the Postman State of API Report. While this number may be small now, 35% of respondents also said they plan to use GraphQL in the coming year.

“It’s a very, very interesting movement,” said Neuse. What began at Facebook as a framework for data fetching for React apps has now expanded into something much bigger. Within this second new movement, largely driven by companies like Apollo and Hasura, GraphQL becomes the new data layer, he said.

“Still, everybody is looking at GraphQL thinking they must migrate from REST—this is the wrong direction,” said Neuse. Instead of migrating all existing REST APIs to GraphQL, he predicts the industry will coalesce on using GraphQL as the integration language for querying REST APIs. A meta layer, if you will.

Of course, meta has been a bit co-opted lately with Facebook’s rebranding …
November 30, 2021
Fugue Marries Compliance-as-Code Tool to AWS Well-Architected Framework

Fugue has infused the best practices defined by Amazon Web Services (AWS) within its software-as-a-service (SaaS) offering for provisioning infrastructure using its infrastructure-as-code (IaC) platform.

Fugue CEO Josh Stella said that IT teams can now evaluate templates for provisioning AWS infrastructure that were created using AWS CloudFormation or Terraform tools to ensure they comply with the AWS Well-Architected Framework.

The AWS Well-Architected Framework spans five pillars that AWS has defined for designing and operating reliable, secure, efficient and cost-effective systems in the cloud. AWS has defined a set of best practices for each pillar and rewards organizations for following those by making available additional cloud credits. The goal is to improve overall cloud security by providing guidance that ultimately reduces the number of misconfigurations so that cybercriminals can’t discover and exploit them.

The Fugue platform complements that effort because it is based on Open Policy Agent (OPA), a general-purpose engine for managing compliance-as-code that is being advanced under the auspices of the Cloud Native Computing Foundation (CNCF). Fugue extended OPA by creating Regula, an open source tool that evaluates IaC files for potential security and compliance violations. Fugue then created a SaaS platform that makes extensive use of the AWS Lambda serverless computing platform to make it easier to enforce compliance policies.

Those policies include turnkey coverage for specifications such as SOC 2, NIST 800-53, GDPR, PCI, HIPAA, ISO 27001, CSA CCM, CIS Controls, CIS Docker and CIS Foundations Benchmarks for AWS, Microsoft Azure, Google Cloud and Kubernetes.

Stella said Fugue has now incorporated the technical aspects of the AWS Well-Architected Framework. However, it’s still up to each IT team to navigate the cultural aspects of the shared responsibility model of compliance and security that cloud service providers like AWS follow.

Unfortunately, developers that employ IaC tools to provision cloud infrastructure often make some faulty assumptions about the level of security maintained by the cloud service provider. The result is often a raft of security and compliance issues that arise mainly because most developers don’t have a lot of domain knowledge expertise in either area.

It’s not clear how many organizations have adopted the AWS Well-Architected Framework, but as various tools make it easier to implement these best practices defined by AWS the more secure and efficient cloud computing environments should become. In fact, in many cases, those tools are being embedded within a larger set of DevSecOps best practices that are being implemented across multiple clouds.

Stella said he doubted there will ever be any standardization when it comes to security and compliance frameworks that could be applied to multiple clouds; however, it’s clear that cloud service providers are at least making available frameworks that borrow concepts from one another.

Regardless of the approach, the overall state of cloud security and compliance will continue to improve as more guardrails are automatically implemented. The issue then becomes not only finding a way to ensure that cloud infrastructure is secure when deployed and also determining how much of the infrastructure being relied on today is less secure than it should be.

November 29, 2021
A Pro Football Approach to Agile Vs. DevOps
Enterprises and professional football teams have a lot in common. The pressures of being more agile, more resilient while ensuring performance is felt both on and off the field. For CIOs, surviving and thriving requires a serious focus on operations tied to business mandates and growth.

I am frequently asked questions around infrastructure and operations (I&O) with organizations looking to solve ITOps, DevOps and DevSecOps challenges.
- How do we shift from legacy ITOps models to transformative digital operations?
- How do we align process, people and tools for closer integration, consolidation and correlation?
- How are we structured as teams across functions—ITOps, DevOps, SecOps?
- How do we transform to analytics-driven operations at speed and efficiency?
Historically, Agile and DevOps teams face off on the IT playing field as adversaries; both sides want to advance their plans, execute plays and reach the end zone. Disconnects happen when teams are not aligned, goals are not defined and resources are constrained.

To effectively deliver digital services, teams must focus on the essentials of clear communication, unified culture and measures to ensure resiliency, performance and availability if they are to be effective.

Traditional I&O teams often operate independently, and knowledge is not shared nor available to cross-functional teams. Attempts to respond and resolve critical incidents can take longer. With DevOps or even DevSecOps, the ideal is for groups to align across functions by breaking down silos of tooling and responsibility.

In a hybrid ops environment, a lack of integration and coordination between DevOps roles can easily misalign efforts and priorities. Whether it’s hybrid in terms of IT environment or hybrid in support model, there are unique challenges that must be addressed. Not being able to huddle around the same data sets while troubleshooting incidents poses a huge challenge when implementing proper analytics comes into play. Legacy IT operations management (ITOM) tools simply cannot support DevOps’ use of cloud-native services nor support remote IT teams working globally.

For successful DevOps and traditional Agile teams to coexist and be successful, IT leaders need to change existing mindsets and implement new practices in order to scale.
- DevOps and ITOps groups have to feel supported by the entire IT organization.
Often DevOps starts as an experiment with a small group of people working on one application. Everyone in IT, from the leaders to IT workers, needs to understand what the end goals are for integrating new methodologies. Deep realization sets in when DevOps finishes a project and pushes to production which is still supported in traditional ITOps models. It becomes obvious that operations needs to adapt to frequent change.
- Plan and implement DevOps practices to avoid disruptions.
With legacy systems still in practice for delivering digital services, DevOps teams cannot disrupt critical day-to-day operations by driving new support models without taking into account migration considerations. Instead, teams will need to work in a hybrid model using application development delivery methods.
- Collaboration and planning tools are essential.
The use of collaboration and planning tools alongside automation capabilities is essential in hybrid organizations. Start by automating alerts to generate tickets for the appropriate team(s) to handle versus having Level 1 support process all alerts independently.
- Build or buy scalable data analysis systems.
These systems will aggregate and correlate data across infrastructure and digital/cloud-native technologies to deliver contextual insights to problems and provide corrective steps to resolve them.

Scaling and modernizing operations and creating agile practices is an ongoing process that needs constant planning, testing and adapting. Streamlining DevOps and ITOps processes using automation starts the process as teams begin to work together versus working in traditional silos.

For winning football franchises, dynasties weren’t built in a single season; the same is true for your digital ops transformation. It takes challenging the status quo—taking a hard look at the deeply embedded ways ITOps has always been done—and reimagining the whole process focused on data-driven digital operations.
November 29, 2021
WTH? We Wanna WFH | DoD Dual-Sources JWCC | More Nvidia ARM Woes

Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

This week: Working from home is de rigueur, JEDI redux, and more about Nvidia/Arm.

Survey Says: Devs Stay Out of Office

First up this week: GitHub’s annual survey of developers has some fascinating data on the workplace (ahem) “new normal.” The pandemic has radically changed devs’ expectations of where they can do their jobs.

Analysis: WFH is SOP

As with other roles, Dev people want to continue to work from home. But does this pose a threat to your DevOps culture? Make sure you can do Ops with equal flexibility.

Paul Krill: Dev productivity is back to pre-pandemic levels

GitHub’s “2021 State of the Octoverse” research … sees signs that work rhythm is returning to pre-pandemic levels. GitHub also found that 46 percent of developers who worked co-located with teammates now expect to work fully remotely or in a hybrid environment.
…
Based on data culled from more than four million repositories and surveys of more than 12,000 developers … JavaScript and Python remain the top languages, followed by Java and TypeScript. [And] GitHub found that development team performance can increase as much as 87 percent when reusing code and as much as 43 percent when using … CI/CD.

Zeroing in on the WFH angle, it’s Richard Speed:

Respondents who were in the office either full or part-time dropped from 42 percent before the pandemic to a mere 10.7 percent now. … That’s a lot of empty desks.
…
Fans of the ctrl-c, ctrl-v combo will be delighted to note that GitHub found that when code reuse was made “frictionless” at work, developer’s performance increased by up to 87 percent. … It’s all splendid stuff.

But beware of statistical fallacies, thinks Henrik Ingo—@h_ingo:

#WorkFromHome may have been more common than anyone realized. [But] is the population of GitHub users/respondents somehow skewed toward pioneers/early adopters? … This might be a case of correlation is not causation.

Defense Dept. Orders New Clouds

The DoD issued its latest cloud contract opportunity notice, with the snappy title of “Joint Warfighting Cloud Capability” (JWCC). Reading between the lines of the notice, it all sounds very DevOps-cum-Agile.

The Pentagon has backed away from its previous insistence on a single supplier, and now wants at least two cloud providers.

Analysis: JEDI redux, because AWS wants a do-over

JWCC is the follow-on from the aborted JEDI procurement. Amazon Web Services was “disappointed” to have been passed over in favor of Microsoft. Now it appears the DoD wants them to share.

What can you learn from such a multi-vendor DevOps strategy?

Jordan Novet: Pentagon asks Amazon, Google, Microsoft and Oracle for bids

Amazon and Microsoft were the finalists for a single Joint Enterprise Defense Infrastructure. … Microsoft won it in 2019, Amazon filed a protest and the Pentagon canceled the contract in July.
…
The new effort [is] known as Joint Warfighting Cloud Capability. … JWCC differs from JEDI because it allows the Pentagon to rely on multiple cloud providers.
…
The U.S. General Services Administration … said only two U.S. cloud infrastructure providers, Amazon and Microsoft, appear able to comply with all of the Pentagon’s requirements. … The value of the new contracts is not known, but the Defense Department estimates it could run into the multiple billions of dollars.

In summary, heed this Anonymous Coward:

Competing suppliers will try harder than one non-competing supplier. The DoD has had a lot of problems with One-Ring-to-rule-them-all systems like the F-35. … JEDI would have been the same.
…
Smaller steps with upgrades while testing/using offers better returns. Done right they can get a better deal by making the companies compete against each other in making sure the products are actually useful.

But look carefully at the DoD contract opportunity, says bustinbrains:

They’ll all get contracts. Then DoD will simply use the one that they actually like the best. No one can complain because it’s open-ended with no guarantees.

Arm/Nvidia Deal on FTC’s Radar

Last week, we talked about Nvidia’s goal to acquire Arm Ltd from SoftBank, and the regulatory challenges it poses—notably in the UK, home of the chip architect. After we went to press, the other shoe dropped.

Analysis: Just Give Up Already

As we said last week, Nvidia can’t be allowed to do this. Not only are ARM chips found in 99% of smartphones, but they’re an increasing fixture in the datacenter—especially places that value “performance per Watt.”

Richard Waters: What’s next for the tumultuous takeover of Arm?

Nvidia’s acquisition of UK chip design company Arm from SoftBank has provoked serious opposition on both sides of the Atlantic. … The US Federal Trade Commission had its own worries. China, meanwhile, is waiting in the wings.
…
The deal has shone a spotlight on the unique position Arm plays in the chip industry. [Nvidia] offered a guarantee that it won’t block any other companies from licensing Arm’s designs. The offer has fallen on deaf ears, with both the EU and UK ruling it inadequate. … Post-Brexit politics have also come into play.
…
At what point might Nvidia and SoftBank call it a day? … A stock market listing [is] the most likely alternative, with the UK a favoured venue.

How’s it viewed from the inside? Here’s a hint from fivemack:

I was at ARM when Softbank bought them; it came as a surprise to everybody, because we were absolutely confident that we were unbuyable because anyone interested in us would want to use differential licensing as a weapon against their competitors, and we … were confident that regulators wouldn’t accept that.
…
Lots of people went home with a year’s pay check in free money. And not a few people … went home with a decade’s pay check in free money.

More and more DevOps workloads are running on ARM silicon. Here’s Biophoton:

The low power expertise of ARM has to be an advantage in the data centre (DC) market. … Plus Linux is available on ARM so architectural lock-in of Intel could be eroded in the DC market.

The Moral of the Story: You are known by the company you keep

You have been reading The Long View by Richi Jennings. You can contact him at @RiCHi or tlv@richi.uk.

Image: Lukas Mann (via Unsplash)

November 24, 2021
Digital Transformation Driving VSM Adoption

The number of organizations using value stream management (VSM) to bridge the divide between DevOps and the rest of the business is expected to dramatically increase over the next two years. The shift is due, in part, to investments in digital business transformation initiatives.

A survey of more than 500 executives, IT and business leaders conducted by Dimensional Research on behalf of Broadcom, a VSM platform provider, finds the percentage of organizations employing VSM will rise from 42% today to 70% in the next two years. Nearly three-quarters of organizations (72%) that are not currently using VSM plan to do so in the next 18 months or more.

The survey also finds that 85% of organizations are collecting some subset of value stream metrics today as part of an effort to deliver solutions faster (56%), improve decision making (55%) and increase transparency (54%).

Laureen Knudsen, chief transformation officer at Broadcom, said that as organizations embrace DevOps best practices more broadly to drive digital business transformation initiatives, they need more visibility into the application development process. VSM surfaces the insights required to not only identify bottlenecks but also track how changes in application delivery timetables will impact key performance indicators (KPIs).

VSM traces its lineage back to lean manufacturing methods, which called for each step of a manufacturing process to be continuously measured. As software development has evolved from being a craft to an automated process using DevOps best practices—and as organizations realize just how dependent they are on software—an appreciation for the value of monitoring the impact of missed software development deadlines on the business is growing steadily. VSM platforms automatically collect technical metrics from DevOps tools that are then correlated against the KPIs the organization decides to track.

The Broadcom survey found more than 96% of respondents’ companies are already in the process of adopting digital transformation. Strategic objectives include increasing efficiencies (70%), improving product quality (67%) and improving customer value (63%).

Most respondents (92%), however, are anticipating product development challenges for 2022. The biggest product development challenge cited is a lack of visibility throughout the product life cycle (45%) followed by inefficient processes (44%), lack of resources (42%) and enterprise-wide lack of understanding of how products deliver customer value (42%). A full 89% of the business leaders surveyed also noted they continue to struggle with organizational silos between strategic development and operations; 93% noted that better alignment between teams would improve their organization’s overall return on investment (ROI) in software development. A total of 95% said they believed VSM would optimize their product life cycle processes.

Knudsen said that, in time, artificial intelligence (AI) technologies will increasingly be infused into VSM platforms to enable automated feedback loops between the business and DevOps teams. In the meantime, DevOps teams should be prepared to adjust project priorities based on the needs of the business and to endure a higher level of scrutiny as business stakeholders become more involved in setting the software development agenda.

November 24, 2021
Some Change is Bad
Some time toward the end of the year, I start considering the volume of change we, as practitioners, have been under for … literally years, even decades. And I start to try and think of ways to keep you all from drowning in change. And then I write something that I hope helps you all focus on good change, and keeps moving your environment forward.

This year I’m going to say “consider the reasons for the change.” I will attempt to find examples that do not require mentioning a specific product or vendor, so apologies if these read a bit convoluted as I attempt to illustrate this point without castigating.

There are three basic reasons that a market or marketers want you to consider a change. There are, of course, variants and cases where it is a bit of this reason and a bit of that, but most demands that we change what/how we are doing things can be lumped into “mostly one of these”. They are:
1. A great idea that is really a game-changer for some subset of the market.
2. A shifting of who is in control.
3. Locking customers in or expanding revenue.
All of these happen at the vendor/project level, and all of them happen at the market level.

Number one is best illustrated by any of the waves of server virtualization that have come through. From virtual machines to containers, each has solved problems and changed the way we do business. Notably, I’m only talking about server virtualization efforts here. Most of the other virtualization attempts have really not fit the “great idea that is a game-changer” category, though some have limited usefulness. Personally, I think the new version of storage virtualization will be in the game-changer category but am waiting to see it in use.

Number two is one to watch out for and is most obvious when big-name OSS projects are forked. It is rarely about the technology being implemented (after all, a fork is a copy of the original; at first it is not at all about the technology), but rather about who is calling the shots. There is a bit of this in every public cloud vendor implementing their own version of container management (though this example is more about lock-in, in my opinion).

Number three is multi-pronged. If your vendor went from selling you hardware or software to selling you a service, they want that recurring revenue and they want you to be linked into that service so deep that you can’t walk away. The much-reviled 1990s mantra of “embrace and extend” is the poster child for this kind of change. There might be a value-add in there somewhere and sometimes the added functionality makes the move a no-brainer. Often, though, the change is simply to the vendor’s advantage, not their customers. As most major Linux distros have shown, open source is not immune to this one, as most of them are making OSS users the de facto beta testers for their paid offerings.

None of this is to say that the change in front of you is bad—even if the reason it was originally spawned is bad. It is reminding you that, after a constant stream of change, we’re more open to it than maybe we should be and you should remember the only important thing about new changes—do they make our environment more secure, faster, more agile? Everything else is just noise. Yes, all the cool kids are switching to coding in [insert flavor du jour], but does it offer your organization anything new? Yes, it seems everyone is using [insert current CI/CD flavor], but does it address the weaknesses of your current toolchain?

You all are nailing it. Keep doing it. And don’t be afraid of change, but make sure you know if that new OSS database system is really new, or just a lot of installing the same code under a slightly different name. Because someone didn’t like the new owners.
November 24, 2021
Top 10 DevOps Things to Be Thankful for in 2021

Thanksgiving has been observed in the U.S. since 1861 and was officially designated as an official holiday in 1863 by president Lincoln. Of course, the tradition itself traces its lineage to the early days of the Pilgrims.

This past year, of course, has been one of recovery from a pandemic that caused the deaths of millions of people around the world. Vaccines have saved the lives of countless more, so there is much to be thankful for despite all our recent trials and tribulations. In that spirit, there are more than a few things for DevOps professionals, in particular, to be thankful for this year starting with:

Digital business transformation: Organizations of all sizes have had to adjust their business models because of both major changes in buyer behavior and an ongoing supply chain crisis. DevOps teams are at the core of any major digital business transformation initiative, so the demand for DevOps expertise continues to outstrip the available supply. Going forward, it’s been predicted we’ll see more digital transformation in the next 10 years than in all the last 40 years combined. The vast majority of those initiatives would never see the light of day if it were not for the expertise of developers and engineers working hand-in-hand to build and deploy applications faster than ever.

Low-code platforms: As the demand for applications to drive digital workflows spiked in the wake of the pandemic, professional developers relied more on low-code platforms to decrease the time required to build an application. While many complex applications will still be built using procedural code, it’s clear not nearly as many applications could have been deployed without relying on low-code platforms to build them. It remains to be seen how many of the next generations of low-code applications will be built by so-called citizen developers that may soon be flooding DevOps pipelines with application code.

Microservices: As an architecture for building applications the core concept of employing loosely coupled services together to construct an application goes all the way back to when service-oriented applications (SOA) were expected to be the next big thing in the 1990s. Microservices have, of course, been around for several years themselves. However, with the rise of containers as a software artifact for building microservices, it’s only now becoming apparent that building an application constructed using microservices not only takes less time but is easier to maintain and also more resilient.

Observability: As a concept, observability traces its lineage to linear dynamic systems. Observability in its most basic form measures how well the internal states of a system can be inferred based on knowledge of its external outputs. In the past year, a wide range of IT vendors introduced various types of observability platforms. These make it easier for DevOps teams to query machine data in a way that enables them to proactively discover the root cause of issues before they cause further disruption. That’s a major advance over monitoring platforms that typically only provide predefined metrics to identify when a specific platform or application is performing within expectations.

Biden’s executive order: In the wake of a pernicious attack against IT service management software from SolarWinds, the Biden administration published a sweeping executive order that requires federal agencies to implement a wide range of security best practices. Foremost among them is guidance concerning the security of software supply chains; which resulted in many organizations reviewing their entire software supply chain to make sure similar breaches will not occur.

DevSecOps: As an extension of any DevOps best practice, DevSecOps generally describes an overall shift left in terms of making developers more responsible for application security. While still in its relative infancy, DevSecOps is starting to gain traction as DevOps and cybersecurity teams look to bridge a cultural divide that is as wide as any in IT. The ultimate goal is to deliver more secure applications without reducing the speed at which they are being built and deployed.

GitOps: At its core, GitOps is an operational framework that takes DevOps best practices used to automate application development and extends them to infrastructure. The basic idea is that the configuration software created to manage infrastructure-as-code is just another software artifact to be stored in a Git repository. The approach provides a means for IT teams to achieve a continuous delivery goal that, more than a decade after DevOps best practices were defined, remains elusive. It’s still early days as far as GitOps best practices are concerned, but this has been the year when GitOps become part of the mainstream DevOps conversation.

Kubernetes: Simultaneously one the most powerful and most complex IT platforms ever to come down the proverbial enterprise IT pike, Kubernetes has emerged as a de facto standard for building and deploying container applications that can more efficiently scale up and down as required to consume IT infrastructure. The challenge (and the opportunity) now is to make Kubernetes much more accessible to mere mortal IT administrators that typically don’t have the programming skills of a site reliability engineer (SRE) and which are both hard to find and retain.

GraphQL: Employed as a query language for application programming interfaces (APIs), GraphQL provides a more efficient means of accessing data than the REST APIs that have been employed by developers in the past. Originally developed by Facebook, GraphQL adoption is still in its infancy but it’s clear APIs based on GrapQL will one day be as plentiful as REST APIs. That’s especially critical for microservices-based applications that rely on APIs to access data across a highly distributed computing environment that requires a more efficient means for transferring data.

Venture and Wall Street capital investments: Investment firms are helping fuel DevOps advances as they continue to make capital available for companies vying for their share of a market that is expected to see compound annual growth rates of 24.7% through 2026. There will, of course, eventually be a wave of consolidation as merger and acquisition activity inevitably ensues. In the meantime, however, there is a wide range of DevOps technologies that might not otherwise be as broadly adopted if it were not for all the cash being poured into DevOps companies.

The most important thing, of course, is that so many DevOps professionals are still here to build and deploy applications. It’s been a trying year for all concerned; however, as the pandemic crisis continues to recede, we can all be thankful for the opportunities that lie ahead.

November 22, 2021
Ensuring Network Connectivity With SD-WAN and AIOps
Recently, Facebook experienced an outage. If companies relied on that app to do business, their IT teams were likely swamped with end-user complaint calls.

How would teams traditionally respond? They’d dive deep into the rabbit hole, analyzing issues on a case-by-case basis, asking countless questions, including:
- Could there be a problem with the enterprise network?
- Is the service provider experiencing issues?
- Are just a few users affected or is the issue more systemic?
Traditionally, this manual process requires significant knowledge sharing and time to complete as teams break the problem into smaller pieces to triangulate an issue’s root cause.

Oftentimes, it’s an enterprise network that experiences connectivity issues—not apps like Facebook.

Some companies pay an arm and a leg to leverage highly reliable WAN links, such as MPLS, for connecting to the internet and connecting branches to their data centers. Conversely, they may save a buck or two by using commodity links, like cable or DSL, which deliver less reliability. Both choices introduce tough tradeoffs—could there be a better way to ensure enterprise-wide connectivity?

Enter SD-WAN.

Combining the same service-level agreement (SLA) as a dependable WAN link with the affordability of a commodity link, SD-WAN specializes in handling brownout situations, rerouting link traffic to maintain network connectivity — ensuring users’ access to their must-have apps.

And by incorporating artificial intelligence for IT operations (AIOps), teams can harness the power of analytics to radically reduce their response time, ensuring that if problems do happen, they get fixed fast. In fact, 90% of IT professionals acknowledge that powering network management with AIOps will improve their companies’ business outcomes.

Drowning in Data: Understanding Traditional Network Issue Investigation Headaches

To determine the root cause of a service disruption, teams spend considerable time assessing the landscape of all the different apps that people use and the links they follow to use them. These apps could be located anywhere—forcing teams to find needles across vast haystacks of data to determine what went wrong.

And this data is exploding at such an extraordinarily high rate that it’s practically impossible for anyone to keep track of it all. Only by leveraging an advanced analytics system like AIOps can teams gain the necessary insights to swiftly solve problems.

SD-WAN and AIOps: A Problem-Solving Dream Team

How do SD-WAN and AIOps work together to enable a highly reliable, always-connected network?

SD-WAN quickly addresses all potential local issues and optimizes performance and connectivity across multiple WAN links—or even just a single WAN link that may be unreliable. SD-WAN also ensures that business-critical apps are prioritized correctly and are offered the precise amount of error correction that avoids any issues or disruptions as much as possible over the WAN.

Where does AIOps come into play? It analyzes data at a far more granular level for extended periods over a global topology. What is it looking for? Everything. It’s examining emerging patterns that are impacting individual apps. It’s searching for systemic issues that affect multiple SD-WAN edge devices simultaneously, worldwide. It could also be investigating how edges or apps are being affected over a particular time period.

By surfacing all that data, AIOps analyzes much more than SD-WAN can. And by deriving insights from that data, it highlights issues affecting everything from a lone user on a single application to a systemic issue that’s impacting an app across a far broader cohort of users. In certain cases, AIOps can cue the SD-WAN to heal itself and automatically fix the problem, then verify that the issue was fixed. As a result, end-users remain connected to their network.

Inside a Wholly Adopted SD-WAN/AIOps Environment

Imagine you’re an IT admin, supporting an extremely complex enterprise network, spanning thousands of different sites that use diverse security services for connecting to apps. Even if you’re on a huge team that’s charged with monitoring the network, it would be nearly impossible to have full visibility across the network to solve problems as they occur.

By adopting SD-WAN together with AIOps, you gain a superpower. No, you won’t be able to fly or become invisible, but AIOps will make you omniscient, enabling you to:
- Harness powerful analytics firepower to find and fix network trouble spots instantly and precisely before users complain—in some cases before they even notice
- Understand which sites perform better than others to pinpoint where improvements can be made
- Propose configuration changes that help the network self-optimize and self-heal
- Detect and optimize degraded links to boost user experience
And if issues ever do arise, AIOps advises you immediately as it monitors the network 24/7, providing you with full situational awareness and insights, solving problems as they occur.

SD-WAN & AIOps: Maintaining Connectivity by Self-Healing

The Facebook event was unusual in that it globally impacted the world’s users, who couldn’t access the app through any Facebook data center. In that scenario, AIOps simply identified the app disruption and noted the root cause.

But what if the situation had been different? For example, imagine if Facebook’s U.S. west coast datacenters were deemed unavailable but its east coast datacenters were operational. By combining SD-WAN with AIOps, a similarly affected company could have performed diagnosis and ensured app connectivity.

How could that have played out?

First, AIOps would have detected the issue and its root cause. It would have reported that the application was unreachable to users on the west coast but available to users on the east coast. Finally, SD-WAN would have self-healed by rerouting end-user application traffic to the eastern data centers to maintain app connectivity.

Using the old-school method of manually monitoring and troubleshooting networks is nearly impossible for most IT teams. Instead, teams are relying on SD-WAN and AIOps. This combined capability gives teams an unbeatable superpower, enabling them to substantially improve connectivity across the enterprise and rapidly respond to issues as they occur.
November 22, 2021
SAP Extends Developer Tool Portfolio

At an online SAP TechEd conference this week, SAP announced it has added additional low-code/no-code tools to the SAP Business Technology Platform (BTP) to enable both professional and citizen developers to build applications that invoke application services provided via the company’s cloud platform.

The effort to increase the number of custom applications invoking those services revolves around a no-code development and automation platform dubbed SAP AppGyver, launched this week, and an existing set of low-code tools dubbed SAP Business Application Studio that the company also enhanced. SAP Process Automation, a robotic process automation (RPA) platform for automating workflows, is also being previewed.

JG Chirapurath, chief marketing and solutions officer for SAP BTP, said the goal is to make it easier for organizations to build applications faster using tools that provide higher layers of abstraction through which they can invoke, for example, backend RPA and AI services via an application programming interface (API). SAP makes it possible to achieve that goal via an SAP BTP platform it provides to IT teams. The BTP platform allows teams to build applications that can be integrated with SAP application services running on a public cloud or managed by SAP on behalf of the customer. In either case, the goal is to make it possible for developers to spend more time building business logic versus building and maintaining integrations, he said.

As part of the effort, new capabilities were also added to the SAP Conversational AI service and the company has promised to launch a personalized recommendation service based on a neural network.

At the same time, the company also revealed it has updated the SAP Integration Suite for SAP BTP to provide access to additional prepacked integrations that are available on the SAP API Business Hub.

In addition, the company said by the end of this year it plans to make the SAP Integration Suite available on the Google Cloud Platform alongside support it provides for other public clouds.

Finally, a revamped training site, dubbed SAP Learning, will make it simpler for developers to advance their skills.

The challenge SAP application developers are trying to navigate is determining which business processes are best served using packaged cloud applications versus ones that require actual customer code. The number of processes packaged in a suite of enterprise resource planning (ERP) applications has greatly expanded over time. With each successive update, ERP platform providers such as SAP identify new processes that could be beneficial to the bulk of their customers. For example, the processes that organizations use to track invoices and issue payments are all pretty much the same. It doesn’t make a lot of sense to write a custom application to manage a set of tasks that are already well-defined within a packaged application.

Custom application development is typically where organizations have an opportunity to differentiate themselves. After all, if every organization is relying on the same core processes provided by a packaged application vendor, there is no real difference from one organization to the next. Custom applications, on the other hand, built using no-code or low-code tools or procedural code written by developers, should either be used to extend a workflow in some way an application doesn’t or should be used to create an application that provides a sustainable competitive advantage. That challenge—and the opportunity—is first identifying those processes and then finding the simplest way to build and maintain the applications that drive them.

November 18, 2021