devopsy.kinsta.cloud

Author: Bernard Brode

5 Steps to More Sustainable DevOps
The rise of remote workforces, cryptocurrencies and super-sized data centers brings to the forefront a concern in tech development that doesn’t get enough attention: The impact of computing and advanced technologies on the environment.

About six months into the pandemic, people began to notice cleaner air, clearer water and an overall calming effect on the environment in the reduced presence of industry. Memes were created and widely circulated that suggested maybe we were the problem after all.

That lasted for about six weeks before cabin fever set in, and we became impatient and ready to get back to business as usual.

In the time between the last quarter of 2020 and the sudden waxing and waning of NFTs, a few timid voices again raised concerns about the impact of super-computing and other technologies on the environment. Those concerns eventually became muffled amidst socioeconomic and political turmoil, fears about inflation, staff shortages and logistical issues surrounding supply chain disruption.

App development tried to fill some of those gaps and new data centers appeared as we struggled to find our footing again in a post-pandemic world.

While environmental issues have been a matter of growing concern for decades, the past few years have put a particularly intense focus on the issue, with entire industries growing up around everything from climate-friendly investing to carbon offsets, and more. The environmental impact of technology has become an issue yet unresolved in spite of a growing focus on sustainability.

Addressing the Elephant in Your Staging Area

The typical DevOps journey focuses mainly on how app development will impact an organization. Rarely does it address how development impacts the environment.

In addition to factors like improving time-to-market, reducing human error, debugging and making financial resource management more efficient, part of your DevOps strategy should include ways to reduce your carbon footprint and impact on the planet.

First, it helps to understand exactly how technology affects the climate and depletes natural resources so you can incorporate sustainable development into your plan in a meaningful way.
- Carbon emissions: It’s estimated that in 2022, there are already more than 16.5 billion mobile devices in use worldwide. Just one iPhone emits up to 86kg of CO2 over its life cycle, and 80% of that occurs before it even gets into the hands of consumers. This is just the tip of the carbon emissions iceberg, and it will only get worse.
- Power consumption: All of that carbon comes from somewhere, and taking a look at energy consumption from just one slice of technology provides a glimpse into just how heavily computing contributes to the problem. As of 2020, data centers used about 416 terawatts of energy annually. The Bitcoin network alone consumes more power than all global data centers combined. A single NFT transaction consumes 82 kWh of power and emits 48 kg of CO2.
Even relatively passive use of technology, like video streaming and remote collaboration, for example, has the potential to consume more power than entire countries. Add up all of the hours of Netflix, Zoom meetings and other remote activities and ponder the implications.
- Waste: Despite the number of devices in use, very little hardware is recycled. Electronic waste comprises 70% of toxic waste worldwide, and it’s estimated to total 40 million tons annually.
- Water consumption: You don’t often consider mixing water and electricity, but all of those data centers need to be cooled somehow. Internet usage alone uses 3,000 liters of water per person each year and emits 2,000 kg of CO2.
Cleaning up Your DevOps Process for the Good of the Climate

To become part of the solution, it’s important to do more than just invest in green businesses. Here are some steps you can take to clean up your DevOps process and become part of the solution rather than contributing to the problem.

Identify Areas of Vulnerability

Software engineering and DevOps are increasingly embracing new territory and emphasizing sustainable engineering. The first step in the right direction is to become aware of how power and resources are consumed during your development cycle.

Consider:
- How your power is generated. For example, are you dependent on fossil fuels or more eco-friendly methods?
- How often are you replacing or recycling hardware and other physical components? This is a goal in waste reduction.
- How are you cooling data centers and computing equipment? Are your efforts at maintaining optimal temperatures wasting resources?
Balance the Three Areas of Sustainable DevOps

In the realm of sustainable engineering, the process can be broken down into three categories: Technical, operational and environmental. The goal is to balance these aspects to monitor and optimize the development process and do the least amount of damage possible.

Enforce Sustainable Development Practices

Once you’ve identified areas of risk and set goals for sustainable software engineering, use the results to create and enforce a set of development best practices that better support environmentally responsible design and production. If you need some help or ideas, you can model your own initiatives after these emerging standardization principles.

Become Part of the Green Tech Community

In 2021, more than 200 countries signed the Glasgow Climate Pact, solidifying their commitment to reducing their carbon footprint and resultant environmental damage from tech development and use. The conference brought together stakeholders from businesses, government agencies and others to share information and create standards and best practices for sustainable computing and tech development.

Incorporate Tools That Support Sustainable Development

Tech giants like Google, Amazon, and Microsoft are creating tools to help DevOps and IT monitor and measure the environmental impact of everything from cloud computing to app development. Seek out and incorporate these tools into your own set of best practices or work on creating your own solutions.

Final Thoughts

Tech development is intended to improve our quality of life and reshape how we conduct business. There’s no reason that intention shouldn’t extend to the world in which we live and engage in commerce.

The first step toward combating climate change without negatively impacting development is understanding how technology affects our environment. Then, we can make informed decisions that support responsible, sustainable DevOps.
March 21, 2022
How Developers Can Handle the FinServ Data Deluge

A decade ago, if you were to ask developers what the biggest challenges in commercial banking were, they would likely cite the difficulty of accessing and collating high-quality data on the day-to-day operations of banking institutions. Today, that’s completely changed.

Or it has to some extent, at least. Developers working in the banking industry certainly have access to more data than ever before. But this has created challenges of its own. With so much data being collected, stored and at least partially processed, some developers have started to talk of a “data deluge”—an unstoppable torrent that overwhelms the developer’s ability to process and use it.

In this article, we’ll try and make sense of this. We’ll also explore what it is being used for and what this means for the future of the banking industry.

Promises and Challenges

Before we get too deeply into the specifics, however, let’s take a moment to note that the data deluge is not entirely new and that it will be with us for quite some time to come.

In fact, most guides to technology in the banking sector note that many of our current issues are caused by the sheer amount of data being collected. It has become acutely apparent that our data harvesting capabilities have grown far more rapidly than our ability to ensure accuracy or to label it in accordance with legally-mandated compliance frameworks.

And, of course, this is only going to get worse. No one knows yet how the global COVID-19 pandemic and ensuing economic downturn will affect the global payments market, but it was previously predicted to reach $2 trillion by the end of 2025, with a compound annual growth rate of 7.83%. Even realizing a portion of that growth will create real challenges for developers in the banking sector.

Using the Data Deluge

With all that said, however, let’s also recognize that this is a ‘good problem.’ It’s not necessarily a bad thing that we have access to so much data—the issue is making sure that it’s accurate, safe and useful.

And when it comes to doing that, three opportunities are already apparent. Here they are.

1. Increased Velocity

Many of the new approaches to leveraging data in the banking sector are focused on increasing speed. Indeed, the ability to access real-time intelligence in banking operations was one of the original promises of the big data revolution, albeit one that has taken some years to come to fruition.

There are several reasons for that, but one of the most important is the fact that banks have lagged when it comes to updating their hardware and software to correlate with the amount of information they are collecting. Most banks now undertake significant levels of quantitative analysis on business and customer data, for instance, but few do so in real-time.

Instead, many banking systems are still run on monthly data-processing schedules, meaning that it can take even the most technologically advanced bank 30 days to spot consumer or business trends. Implementing new technologies such as Apache Spark is one potential solution to this problem. Like Hadoop, it’s an open source big data analytics engine, but it’s faster, more scalable and easier to use.

2. More Reliable Governance

Another challenge when it comes to managing the deluge is that of governance. With so much data flowing in, it can be a challenge for developers and operations staff alike to keep track of its source and how it should be managed within data compliance frameworks.

This is one reason, as we’ve long argued, that banking should consider DevOps. By uniting developers and operations staff, both can come to a deeper understanding of data compliance and how this must be integrated into every aspect of the contemporary banking industry.

This is one area, however, in which more data is not necessarily a bad idea. Rather than pausing data harvesting because of concerns that it will create compliance issues, developers should aim to build systems that can carefully and reliably label this data throughout the custody chain.

3. Focus on the Customer

Finally, developers shouldn’t lose sight of the ultimate aim of collecting data—improving the customer experience. Data can be used to do this in two primary ways.

The first is to put data to work in your customer-facing systems. This is the approach recently pioneered by Western Union, which offers an omnichannel approach that tailors and personalizes customer experiences by processing more than 29 transactions a second and integrating all that data into a single platform for statistical modeling and predictive analysis.

Secondly, data can be used to inform strategic decisions. Just as consumers are now more savvy about the way that their data is used, leadership needs to understand that this type of data-based decision-making is the future of the industry.

In other words, banks must be able to move faster to transform their data into intelligent insights and then put those insights into action to improve customer service, connect customers to information and products when and where it’s needed most and protect sensitive data and customer accounts from threats.

The Bottom Line

The data deluge creates challenges for developers and leaders alike. These challenges are only going to grow as our ability to harvest consumer and industry data continues to expand. If we are going to realize the promises of technology-driven banking, we need to improve the way in which data is managed and processed and ultimately improve our ability to extract meaningful insights from that data.

And just as this challenge is not new, neither is the outcome—a better banking experience for the customer.

If you want to learn more about financial services and FinTech development, operations, architecture and leadership through expert-led talks, panel discussions and keynotes, check out FinCon DX—now available on demand!

December 10, 2021
Out of the Silo: Big Data and DevOps Meet in the Middle

Given the myriad benefits they provide, it seems crazy that big data and DevOps have been implemented independent of each other

If you have been working from home during the recent lockdowns, you probably don’t feel like you are making many new connections, but when we come to look back on this time in a few years, it might prove to have been one of the most effective networking sessions of the past decade.

Nowhere is this more true than in the twin worlds of big data and DevOps. It seems that being forced to work from home, far from locking engineers in both disciplines deeper within their silos, has brought them out of their shells. In recent months, we’ve seen an outpouring of speculation on how DevOps and big data can work together, and even a number of impassioned manifestos on why it is time to bring DevOps ideas to big data.

In this article, we’ll look at why this crossover is suddenly so popular, how these two disciplines can engage in a symbiotic relationship with each other, and how they are already doing so.

The Pandemic and the Silo

For consumers, the experience of lockdown has accelerated internet usage trends. While some of the changes that have recently swept the online world were easy to guess, the speed at which they have taken hold has been astounding. It’s therefore no exaggeration to say, as the New York Times put it recently, that the virus has changed the way we use the internet.

These consumer trends are driving changes throughout the software development industry, forcing firms to adapt to rapid release, testing and update cycles.

One of the biggest changes in the way that developers collect data over the past decade has been the source of the data. This year, and largely due to the COVID-19 pandemic, we passed a notable watershed: Mobile devices and smartphones now account for more than half of all internet traffic, resulting in an increased data flow that boggles the mind and creates issues of its own. This increased data load from consumer mobile devices, rather than from corporate data acquisition systems, is forcing teams to work more closely together.

In the words of Bill Detwiler, editor in chief of TechRepublic, in a recent CBS interview: “The COVID-19 pandemic has accelerated the blending of data analytics and DevOps, meaning developers, data scientists, and product managers will need to work more closely together than ever before.” For most firms, the way in which these consumer trends are fed into the development life cycle is via analytics teams. This has meant that, during the pandemic, analytics teams were generating actionable insights at an increased rate, and this has been the driving force behind the newly intimate relationship between big data and DevOps staff.

Waterfall vs. DevOps

To understand the way in which big data and DevOps teams can best work together, it’s instructive to visualize the way in which developers, operational staff and data analytics agents work.

A decade ago, most software developers were using some form of waterfall development methodology. As the name suggests, this approach was essentially linear—the specifications for software were set in advance, and after this point multiple teams worked in tandem to deliver a continuous “stream” of software updates.

The DevOps framework changed that by encouraging communication and collaboration between operations and development staff. The idea, in short, was to ensure that software underwent continual testing—not just in terms of usability and security, but also against the requirements of the operations staff who were charged with maintaining it.

Now, we might be facing a shift of similar scale. Many firms have now adopted big data frameworks, but these systems generally have been seen as a way of harvesting and collating market insights. This was always a waste of potential for big data because as long as it was regarded as part of marketing, it’s utility in other parts of an organization were difficult to discern.

Not any longer, it seems. The changes caused by the pandemic—both in consumer behavior and in the way that organizations communicate internally—have given developers an appreciation of the value of the data that analytics teams hold.

Pioneers Show the Way Forward

It’s also worth noting that big data and DevOps teams working in tandem is not a utopian vision; rather, this way of working was already being explored within leading companies even before the pandemic.

In retrospect, in fact, it seems crazy that these two teams ever worked independently. If you’re a transportation company, the ability to track your loads on the road and the health and safety of the cargo that they’re carrying has become mission-critical. If you’re in the armed forces and using drones on the battlefield to conduct and report reconnaissance in real-time flyovers, the ability to adapt software parameters on-the-fly can be the difference between success and failure.

The military is, in fact, the place to look for examples of how big data and DevOps will shortly become one process. Big data in the military has long been more advanced than that in the civilian sector and is beginning to inform everything from IT predictive analytics to real-time usage analytics.

The Future of Big Data and DevOps

There remain, of course, significant challenges to be overcome before the promise of DevOps working with big data teams can be realized. Perhaps the most thorny of these is the fact that computing performance is crucial for big data in a way it hasn’t been for development: The latter teams may need to build capabilities to make use of the insights generated by big data teams.

Still, communication can only be a good thing, and if the pandemic has forced some of us to emerge from our bunkers and start to work together, then at least one good thing will have come from it.

December 4, 2020
DevOps Was Supposed to Kill Developer Jobs. Here’s Why it Didn’t

The rise of DevOps is more an evolution than a revolution, and developer jobs aren’t going anywhere

Development died a long time ago and was replaced by DevOps. Or, at least that was the way it was supposed to happen.

In reality, developers are doing just fine. There are currently almost 150,000 development jobs right now on one major job site, compared with only about 16,000 positions for DevOps. It seems that the revolution in which developers would be replaced with DevOps engineers never actually happened.

In fact, and as we’ve previously argued, many people find that having DevOps on their job title actually harms them when it comes to finding the right job. In this article, we’ll look at why.

The Death of the Developer

The “death” of the developer has been prophesied for quite some time now. Ever since the creation of the concept of DevOps, in fact. The most vocal and prominent instance of this argument was offered by programmer Jeff Knupp back in 2014, who used a blog post to argue that the popularity of DevOps meant that programmers were increasingly unable to focus on what they do best: writing code.

Instead, argued Knupp, the tyranny of DevOps was forcing coders to become generalists rather than specialists. They were being forced, he said, to include on their CVs an ever-growing number of “skill sets,” some of which were pretty tangential to their core role within an organization. Instead of traditional developers, it was feared, an army of “DevOps engineers” would rise who could just about do a wide range of tasks, but do none of them very well.

In the interests of balance, it’s worth pointing out that Knupp—and many who have repeated his argument—have a vested interest in “defending” traditional coders against the rise of DevOps. It might be true that a thorough knowledge of GDPR compliance processes is not technically necessary for a developer, but some of the skills the DevOps highlighted—not least backend security—are best addressed at the development stage. Developers, in other words, shouldn’t be given a free pass when it comes to thinking about the way in which their creations are actually used.

Adapt and Survive

Nonetheless, the popularity of the concept of DevOps has forced many developers to adapt their skill sets, or at least the way they present them. Concepts such as CI/CD, segmentation of internal communication, automation and “continuous everything” have become central to the way many developers work and are highly sought after. We have DevOps to thank for that.

In addition, the rise of DevOps has made room for an ever-expanding set of novel approaches. Five years ago, it seemed that a new approach to IT management was being developed once a month. First came DevOps, then DevSecOps, then QAOps. This explosion of new approaches does seem to be calming down, but they remain influential on at least two levels.

The first is that, although none of these approaches (and arguably even DevOps, as we’ll see below) have developed into distinct disciplines, they have subtly redefined the way in which developers are understood by the average organization. The second is that, in this context, many developers have begun to stress that they have cross-cutting expertise. In other words, they have had to adapt the way in which they present themselves to survive.

The Death of DevOps

Because of this, we should see the rise of DevOps more as an evolution than a revolution.

At the broadest level, DevOps has had a hugely beneficial effect on the process of software development. There is an expectation that developers will have at least a passing acquaintance with the systems and processes that IT teams use. This was not always the case in the past. Developers today have to know how things such as cloud IAM frameworks or APM platforms work, and why they are important for deploying and managing the applications that developers write.

In this context, it seems strange that some coders ever worried that their job would be replaced. While it’s true that developers are now expected to have much broader skills than they did 10 years ago, organizations still need people who can actually code, and will do for decades yet (until we are all replaced with AI, that is).

It seems that some of us were too eager to see a dramatic revolution, whereas the reality has been more subtle and much slower. And indeed, in many cases the “transition” to DevOps has been almost invisible: new graduates are now trained in the basic principles of DevOps, without this ever having become a discrete college course.

Many of these new graduates, in fact, would be hard-pressed to define what DevOps actually is. But that’s not because it’s gone away; quite the opposite. It is now such a fundamental part of the way that most of us do our jobs that it is losing its identity as a separate idea. This means, eventually, that it will not be “development” that dies, but “DevOps” as a distinct addition to it.

The Future

In short, DevOps has not killed the developer, and there is no evidence that it ever will. Some of us, it seems, were being a little dramatic. On the other hand, DevOps has had huge effects on the way that developers work, because it points to the skills that the best developers have. For that reason, there remain tens of thousands of jobs for developers in the conventional sense.

In short, developers work alongside DevOps engineers; they don’t compete with them for the same roles. So instead of seeing ourselves in competition, let’s try a little collaboration.

July 17, 2020