Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.
This week: The OpenSSL project has egg on its face, and half of Twitter’s staff are for the chop tomorrow. (more…)
Category: Enterprise DevOps
-
OpenSSL Fiasco: What can DevOps Learn? | Elon Fires ‘50%’ of Twitter
-
Know What You Want and Need
Over the years, what we purchase from vendors has changed significantly. It used to be that IT would define a need, go out and find a bit of software to solve it, install and configure the software and, in most cases, spend the majority of time on the project getting it to work correctly in their environment. Those days are largely gone. First, we started buying appliances for some functionality, then we started using MSPs for some or all functionality, then some vendors wanted to deliver in a VM where they could make configuration much easier by controlling the environment. That was the first step to virtualizing purchases. Then we started looking at infrastructure-as-a-service (IaaS), then SaaS, then some vendors offered PaaS … Currently, all of the cool vendors are offering containers—which are an improvement over VMs, and offer a virtual appliance that is quick to get running and tested to do what it was purchased for. Or at least tested to do what it was sold for. We’ve all got stories about what we did with these purchases that were “off label”—things that a product really wasn’t designed for. Along the way, we also introduced open source vendors; a bit of an oxymoron that really described service orgs that base their product on an open source project. Some far more successfully than others.
And all of that, dear reader, is why we are here. Before purchasing, there is a lot more to find out than there used to be. What, actually, is the product this software vendor is selling? In the end, IT is looking to solve a problem. Most of the time, whether that problem is solved by SaaS, local install or cloud instance is less important than how well the problem is solved. But implementation—and licensing—are important to an organization if not immediately, then over the long term.
When purchasing an open source “product”, you’re … actually not. You are getting the product and purchasing support. Since open source can be hacked to run just about anywhere given enough time, more than one OSS vendor has pretended that makes them superior to commercial products targeted at a specific platform. That is blatantly untrue—unless the OSS vendor is supporting the product anywhere and everywhere which, of course, they are not. I can port the OSS project MyFunApp to any platform I want without paying for support that won’t do a thing for me when I get it up and running on OS/2 (as an extreme non-existent example).
SaaS has more issues when it comes to TCO. While pay-as-you-grow is cool and all, locking in a monthly payment forever is a big step and should be considered. And at an enterprise, “forever” needs to be considered realistically. Implement a particular SaaS CRM and you are unlikely to move to another, even if you claim that you will in a heartbeat. It’s work. A lot of work. It is definitely possible, but for most organizations, unlikely. So know what you are getting into. And while you are evaluating, don’t forget to consider variable charges, which most SaaS products have. Make certain you understand what the actual monthly costs are likely to be. IaaS suffers this same variability problem, as many of us have learned.
Containers are a very sweet solution to appliance-like functionality—drop a container into your Kubernetes flavor of choice and move along. The problem here is a combination of proliferation—how many are required? One per pod can get expensive over time, for example. Those resources aren’t free, no matter the platform. The other issue is updates. Automatic updates are the norm for container instance-delivered products. From a security perspective, you should have questions. Updating things in your network should probably require your approval.
I’m not suggesting that one delivery method is better than others here; I’m simply suggesting that we all need to be more deliberate about those choices than we’ve needed to be in the past. Knowing not just SLAs but the cost and support implications of every choice is important both today and down the road.
Keep kicking it, and keep picking the solutions that best solve the organization’s problems. Just be aware that sometimes, the issues a delivery method introduces undermine the effectiveness of the solution.
-
Making App Dev More Efficient
Applications of all kinds are continually popping up across various industries to help improve consumer customer experiences, provide entertainment, make certain tedious or error-prone tasks easier for workers and to help businesses improve their operations overall. In fact, there are over 5.4 million applications available across the Apple App and Google Play stores as of this year.
As more and more individuals and organizations turn to applications for various experiences and to drive enterprise efficiencies, we need to take stock of the process we deploy in application development.
The Inefficiencies of Application Development
Today, it is estimated that there are more than 26 million software developers worldwide. Each of these developers brings a unique combination of creativity and technology acumen to the organizations they work for and the applications they create. However, the current application development process in many enterprises is riddled with inefficiencies that often require these talented developers to focus on tedious, mundane, non-innovation-related tasks.
Technical Debt: Renowned programmer Ward Cunningham first introduced the concept of technical debt in 1992, stating, “Shipping first-time code is like going into debt. A little debt speeds development so long as it is paid back promptly with a rewrite. Objects make the cost of this transaction tolerable. The danger occurs when the debt is not repaid. Every minute spent on not-quite-right code counts as interest on that debt. Entire engineering organizations can be brought to a stand-still under the debt load of an unconsolidated implementation, object-oriented or otherwise.” What many didn’t realize at the time was that technical debt would grow to be a major issue in software and application development years later. Recently, a new Scandinavian study found that developers dedicate 23% of their time, on average, to addressing technical debt. While this statistic alone should be alarming to enterprises—given their development teams are spending a quarter of their time on an issue that could have been avoided had a different approach been taken—the study also found that developers often have to introduce new technical debt in the process.
Maintenance: As with all technological equipment, applications require routine maintenance to ensure they remain working for the end user, provide ongoing value and are secure. However, unlike a car or kitchen appliance, applications see far more frequent updates from operating systems (OSes), cloud or hardware providers that need to be addressed by development teams. According to research from Stripe, developers spend more than 17 hours per week on maintenance issues like debugging or refactoring. For instance, take a look at Apple. From the time iOS 15 was released in June 2021 as a beta, there were more than a dozen updates made to the OS until early September 2022. That means that every single time the updates were released, developers had to spend time ensuring their applications were working correctly, were bug-free or complied with new security measures. When you add this to the fact that the average enterprise app has 110 third-party integrations, that is an extreme amount of maintenance.
The Reinvention of the Wheel: Of all the applications available today, many of them consist of the same foundational features and functionality—login, chat, social sharing, payments and more. What’s astonishing about this is that developers are frequently recreating each of these common features and functionalities from scratch every time they develop a new application.
It is clear that these tasks are highly inefficient, and it begs the question, “What can be done to address these inefficiencies?”
Prioritizing Innovation
It is not enough to just introduce some sort of technology in the hopes that it makes application development more efficient. It also requires an overall mindset change by enterprises. Yes, it is still of the utmost importance to get applications to market quickly and keep up with the pace of competition; however, innovation doesn’t have to be sacrificed to do so.
Innovation must be the key driver for enterprises and their development teams. Providing unique value and experiences to end users will truly set enterprises and their applications apart from the competition. But, the reality is that nearly 80% of technology leaders from around the world believe basic application maintenance and support are the biggest barriers to innovation.
Enterprises must be able to balance innovation, time-to-market and development processes that are free of technical debt and outrageous maintenance support.
Making Application Development More Efficient
One way in which enterprises can balance innovation, commercialization, and development is through an emerging approach to application development known as features-as-a-service (FaaS).
A FaaS approach to application development enables enterprises to implement an agile and accelerated engineering process—similar to that of low-code/no-code without the constraints often seen with those vendors and their functionality today—without sacrificing quality, code or time. More specifically, it combines pre-coded “commodity” features with continuous and automatic updates that empower development teams to spend their time and creative minds on innovation. In most cases, this approach can help enterprises bring applications to market within weeks instead of months and limits the headaches on the backend for development teams down the line.
As enterprises work through the fourth quarter of this year, I implore them to reconsider their development process at a fundamental level and bring innovation to the forefront before the beginning of 2023. Time and resources are of the essence, and something must be done now to address the inefficiencies of application development.
-
BMC Extends Scope of ServiceOps Initiative
At the BMC Exchange conference this week, BMC revealed it has updated its Helix IT service management (ITSM) platform to include mapping and discovery capabilities. Those updates make it simpler for cross-functional teams to correlate incidents, alerts and data and pinpoint root causes faster.
In addition, BMC is now applying pre-trained artificial intelligence (AI) models to observability data and the service models it tracks to assess service health, predict service impacts and proactively manage IT environments as the volume of metrics, events, and alerts continues to increase.
Finally, BMC also announced that the BMC Helix Control-M orchestration engine now supports open source Apache Airflow, a workflow management platform for defining data pipelines using a Python programming tool.
Margaret Lee, vice president and general manager for digital service and operations management at BMC, said these updates extend a unified ServiceOps initiative the company launched last year to unify ITSM, DevOps and customer service desk workflows. While each of these disciplines may have distinct objectives—such as speed and reliability—that are often in conflict with one another, Lee said it’s apparent IT organizations need to reduce the current level of friction that is often encountered when managing IT.
As a software-as-a-service (SaaS) platform, Lee said the BMC Helix platform achieves that goal via a consumer-like user interface that can be easily personalized. The goal is to make it simpler to create virtual agents, knowledge bases, live chat and tickets that make it easy for customers and employees to request IT help, she added.
Achieving that goal, however, will require IT organizations to rely more heavily on AI models to automate routine tasks as IT environments become more complex. There are simply not enough IT professionals available to manage IT environments using legacy processes, she noted.
In addition, end users and customers are expecting service requests to be automated in way that allow them to resolve issues via a mobile device, added Lee. In many cases, how cumbersome IT processes now has a direct impact on both customers and employee retention. The primary goal is to reduce the number of interactions required to complete a task such as changing a password or provisioning a laptop whenever a new employee needs to be onboarded.
While the pace at which IT management is evolving will vary greatly from one organization to the next, it’s apparent that many DevOps principles are being absorbed into traditional ITSM platforms. The challenge is that cultural divide between DevOps teams and centralized IT teams is still very wide. Most organizations are still trying to determine how far left they want to shift responsibilities toward application development teams versus continuing to rely on IT administrators using graphical tools to centrally manage IT workflows. It’s rarely going to be an all-or-nothing proposition. Rather, the issue is finding a way to strike a balance that enables organizations to be as agile as possible without introducing unacceptable levels of risk to resiliency.
-
Where Does Your Data Go?
One of the most interesting developments in security and compliance in recent years is the ability to follow a piece of data through an application from input to consumption and see each bit that touches it. For me, the reason this is so interesting is that it allows postmortems to actually determine exactly what was lost in a data breach. “Oh, that library was compromised? Okay, let’s see what inputs that library actually touches …” It’s so very cool, given the free-for-all that data has traditionally been in applications.
I’m not a huge compliance fan for a variety of reasons, but this is one great result of mandating compliance. I don’t think that we would have this level of data tracking without compliance requirements driving it.
So what can we do with this information? That’s the interesting part that I think we’ll see developing over time. If we know everywhere that a given piece of data is used, we have a powerful bit of development information. It is still early days for implementation–this is not easy stuff–but think about it. I could see a future where all redundant data access is eliminated. If we’re getting the same set of data from multiple sources, and it is being fed to the same internal consumers, then we can standardize the you-know-what out of it. A singular library to standardize formatting, a singular library to standardize look-ups, etc.
For the most part, we currently validate and format on input. Like straight away as the data is brought into the system. That is a great idea for a system where we are uncertain where the data is going, and are sure that some attacks are coming through those inputs. But what if we discovered 10,000 more lines of code to maintain than is necessary? What if data tracing allowed us to eliminate that code by centralizing those functions? We would have to harden the routine doing the validation and formatting–because it will be getting dirty data–but no more so than we are doing literally everywhere today.
If we can reduce source code by using a tool that is designed to improve security, is that not the ultimate shift left? The volume of source code that handles data entering the system is staggering, also. For enterprise systems, most of what we develop is actually data processing, no matter how convoluted it gets. The data store is the central repository and apps are simply accessing and modifying it. So shift left with regard to data is huge.
Consider it. I know that you are all super-busy with other projects, but less source code lightens the load on everyone; take a look at data tracing functionality in products you already own (mostly available in vulnerability management and source code scanning toolsets), and see if it suits your needs. It might save everyone time and will help with some data-centric compliance requirements, too. That leaves you more time to make systems better.
-
Fire at Data Center Causes Chaos | 20% Costlier Cloud
Welcome to
The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.
This week: A South Korean conflagration leads to a ridiculously long outage, and the price of public cloud is skyrocketing. (more…)
-
Cloud: Why Must We Be Confusing?
When I was a kid learning to play Dungeons & Dragons (D&D), the concept of level was never confusing—though the authors should have seen that it very well could be. For those unfamiliar, D&D has many uses of the word level—character level, caster level, spell level, dungeon level and, increasingly, as time went on, monster level. I can almost guarantee that I missed some uses of the word level in the game, but you get the idea. In fact, if I missed your favorite, feel free to add a comment.
We used context to determine what was meant, and we clipped along without any (game) world-shattering consequences. Though some people were horribly confused.
(Side note: I worked with D&D co-creator Gary Gygax for a while and he knew about the over-use but didn’t feel that after D&D’s surge in popularity it could be fixed. His later RPG creations did seek more clarity and less word reuse, but most RPG systems available today still have many uses of the word level.)
I am comfortable declaring that those of us who grew up on RPGs are now running vendor companies and doing analysis. Otherwise, why would we have so many uses for unique words? I mean, we no sooner got DevOps than we made a total mess about what, exactly, you meant when you said “CD”. Was it continuous deployment? Continuous delivery? Both? Similarly, we malformed continuous integration. Let’s call it CI—and then make an entire market around it really being continuous test while it is still used to refer to continuous integration, because that is not at all confusing.
But then we jumped to cloud, and spent some time being really busy trying to confuse everyone about what “cloud” actually means. Is it only public cloud, or does it include private cloud? If private cloud, does that include virtual private cloud only, or does it include cloud in a co-host or data center? Is SaaS actually cloud, or is that a completely different beast?
And that brings us to the reason I came here to write this particular blog post. We are now being barraged with “cloud” that includes Docker hosted on Kubernetes. I think of this as akin to one too many uses of the word “level”.
Kubernetes is not cloud. It offers similar infrastructure in a smaller format, but the key to Kubernetes is that it supersedes cloud. Kubernetes doesn’t care about your cloud (or lack thereof) once it has been implemented. Cloud simply becomes infrastructure-as-a-service (IaaS) when Kubernetes is deployed. This is terribly important for a variety of reasons, not the least of which is that if you drop Kubernetes onto a cloud provider, you have two sets of infrastructure to lock down. Two sets that have a lot of internet exposure—because it’s not called “public cloud” for nothing.
I do understand where referring to Docker hosted on Kubernetes as ‘cloud’ comes from; there are a ton of similarities. But Kubernetes needs a platform—hardware or virtual—to run on. And hand-waving the implications of that nested relationship is asking for trouble. And when cloud is either a thing of the past or so totally commoditized that people don’t think about it, Kubernetes or its successors will likely be the reason. So let’s take it easy re-using cloud too much.
In the short term, though, my advice is that if someone in the industry mentions cloud in their product offering or analysis, ask which use of ‘level’ they mean. Because honestly, Kubernetes, if implemented on hardware or IaaS, can be picked up and moved to whereever it is needed. And that’s not “cloud”.
Keep kicking rear. Doesn’t matter where your apps are running, it matters that you’re keeping them running and locked down. Keep it up, and call things what they are—container orchestrators are not cloud platforms, no matter which functionality they share. And to all my RPG peeps, keep having fun. There are always new adventures to be had… No matter the level.
-
Fear of the Unknown
We IT folks are pretty fearless. Because it is newer technology, and change in IT is easier than in areas like heavy manufacturing, we are regularly thrown new tools, products and methodologies and expected to be able to master them. And we do. And it works. If your career is more than a couple of years old, you are not doing what you started out doing. DevOps and Agile have made this process more pronounced because they increased the rate of change, but this phenomenon has been going on since the PC became readily available. Someone somewhere is currently developing your next toolset. And when they do and it gets thrown at you, you’ll figure it out and run with it.
The problem is that our architectures are never black and white, and we should fear the unknown more. The unknown residual cruft that is all over our networks. Doesn’t matter if “networks” means data center, cloud or both, there’s residual cruft on it. Some organizations even have a full-time position finding cloud instances and services no longer in use and eliminating them. But it is much more broad than cloud instances; there is garbage all over your network and it is time to try to clean it up.
Most of us believe that we can’t possibly get a handle on everything that is out there. Oh, zombie containers and VMs are relatively easy, but the modules and libraries we loaded up and then, through the natural evolution of the software, stopped using, or whatever was bundled in with the OS chosen for that particular project, but was never used—it’s all over, it’s hidden and lots of it isn’t easy to find. That doesn’t even touch on the software that one employee from accounting absolutely had to have—but that employee is long goneand no one in accounting is willing to remove the software, either because they’re unaware of its existence or because they are unaware of what purpose it might be serving.
There are tools to help with some of this, like SBOM generators for excess software on instances, and application maps that work much like network maps to help you understand what is interrelated. But they’re not perfect. They can’t be. You’ve been building this spaghetti application infrastructure for years; it’s going to be tough to work out. Get a list of what’s deployed, tag a person as responsible for it–that app in marketing? The person who requested it is responsible for it (or their manager is, if the original person is gone). IT is responsible for cataloging and, in some orgs, managing the app, but the business unit can explain why it is in use, and if it is still necessary.
It’s a ton of work and, as a former enterprise architect, I can confidently say that it won’t be easy getting names attached to apps. Some apps you might have to turn off to see if they are still in use … and the work will likely never be complete because the infrastructure is changing non-stop as you work.
But it’s worth it. It saves the org money; it reduces IT footprint and it reduces security exposure. It may even save IT staff time, which is one of the most precious commodities your organization currently has. So, start wrangling. Start with one business unit and ask them to help you–after all, they are almost guaranteed to have running apps or SaaS that IT is unaware of. The goal is not to take control of those apps and relationships, but to inventory them so that when they’re no longer needed they can be cleaned up. A side benefit is that if four business units are using similar software outside of IT’s purview, it may be possible to merge those solutions for a corporate approach … But that’s a blog for another time.
Meanwhile, keep rocking it! Just remember there is more on the cloud and in your data center than are dreamt of in your architecture plans (many apologies to Shakespeare). And keep solving the business’ problems.
-
Linux 6.0 is Faster, Cooler | Debian Goes Proprietary | Google Africa Region
Welcome to
The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.
This week: Linux 6.0 promoted to Stable, Debian 12 will include closed-source binaries, and Google Cloud opens its first Africa region. (more…)
-
Driving Organizational Success With the 2022 State of DevOps Report
“Ship it!”
We have just completed the research and analysis for the 2022 Accelerate State of DevOps Report (SODR) and oh boy, do we have some interesting things to share! First, let me take a short step back for those of you that may be new to the DevOps Research and Assessment (DORA) and our annual report, the SODR. DORA is an academically rigorous research program that seeks to answer the questions, “What practices enable teams to be high-performing software teams? And how do these practices impact organizational performance?”
Since 2014, we have surveyed 33,000 practitioners around the world spanning all major industries. The project is intentionally–and staunchly–tool- and platform-agnostic. The research builds on itself, and each year we seek to both evaluate previous findings and expand in other directions or areas of research. This is important to note because many of the core findings, such as “software delivery performance drives organizational performance,” have been validated year after year. I would encourage you to go back and read all the reports from years past.
Security
This year, we doubled down on our 2021 research into software supply chain security by looking at the technical practices that improve software supply chain security and the non-technical practices that impact an organization’s ability to excel at securing their software supply chains. We leveraged two frameworks to focus our research: Supply Chain Levels for Software Artifacts (SLSA) and NIST’s Secure Software Development Framework (SSDF). Below is a summary of some of our key findings in this area:
- Shifting left on security is a broadly adopted practice. Our research shows that two-thirds of respondents are actively pursuing software supply chain security by seamlessly baking security into the development process.
- Culture is the primary driver of security practice adoption. One might expect that technology is the primary driver, but our research showed that a generative organizational culture (eg. performance-oriented, highly cooperative, shared risk) leads to healthier software practices.
- Technical practices around CI/CD predict success in security. Companies that use source control, continuous integration (CI) and continuous delivery (CD) have more established SLSA practices. These practices shift security left to the developers and ensure consistent security scanning.
- Cloud enables secure software practices. The five characteristics of cloud computing as defined by NIST enable the successful adoption of software supply chain security which, in turn, predicts greater organizational performance.
Our research showed that companies who prioritize and excel at securing the software supply chain experienced fewer service outages, anticipate fewer security breaches and demonstrate high levels of both software delivery performance and organizational performance. The data showed that through the use of modern practices like continuous integration, teams can improve their security posture and even amplify the positive impact these security practices have on software delivery metrics (MTTR, deployment frequency, lead time-to-restore service) and overall organizational performance.
Culture
We saw above that the biggest predictor of an organization’s application development security practices was the presence of a generative organizational culture based on shared risk and information sharing. We also see that elements of these types of cultures lead to higher overall organizational performance, as well. Our research showed that high organizational performance can be achieved by fostering environments that are:
- Supportive: Teams that felt supported and that they had buy-in from leadership (e.g., more financial support, more allocation of resources, sponsorships, etc.) were associated with high-performing organizations.
- Stable: Teams that did not experience much change in membership over the last 12 months were more likely to be a part of high-performing organizations.
- Flexible: Organizations with higher levels of working flexibility around where work was performed–remote, in-person or hybrid–were higher-performing overall.
We also looked at burnout again this year and expanded the scope to understand which elements of culture contributed to lower levels of burnout. We found that generative culture, team stability and work flexibility all contributed to a reduction in burnout among employees.
Reliability
In previous years, our research told us that those that excelled in technical practices also excelled at organizational performance. This year, we have more nuanced data about this topic. This year, we saw that software delivery does not predict strong business outcomes unless these practices were paired with reliability. Think about it–will a customer be satisfied with new features if the service isn’t stable? What is the benefit of pushing code quickly into a fragile environment? Reliability is an essential component of driving organizational performance through software delivery performance.
We also see that site reliability engineering’s (SRE’s) impact on organizational performance is non-linear; reliability engineering practices often do not result in additional reliability or organizational performance until a certain maturity is reached. It’s important for teams to know this and approach their SRE practice as an investment. It will likely not be sparkly ponies and unicorns initially as you build the reliability muscle but, as you advance, high performance and success are likely.
Cloud
Public cloud use is up a whopping 36% over 2021, whereas companies reporting no cloud usage at all are down by 50%. The use of hybrid cloud is up by 25%. Unsurprisingly, the use of cloud computing was associated with higher organizational performance. In past years we saw that it wasn’t “using cloud” per se that led to organizational performance, rather it was the achievement of the five essential characteristics of cloud computing–on-demand self-service, broad network access, resource pooling, rapid elasticity and measured service. This year, we saw that cloud computing enabled things like reliability, continuous delivery and improved supply chain security which drive organizational success.
Over 50% of folks who took our survey reported that they leveraged multiple cloud providers. We asked respondents what benefits they realized from using multiple cloud providers. Here are the top three:
- Availability
- Leveraging the unique benefits of each provider
- Trust is spread across multiple providers
Given the fact that reliability seemed to be the key to being a successful software delivery shop as well as a successful organization, it is not surprising to see availability listed by almost 63% of respondents as a benefit derived from using multiple clouds.
Summary
Our research continues to progress and we continue to dig deeper and deeper into the capabilities and practices that impact your business. We see broad adoption on the topics of security and reliability which are both heavily anchored in culture. Good culture leads to success. Period. There is no magic culture button and it takes work, but it is very achievable. Start by defining organization-wide or line-of-business-wide goals and then just start continuously improving. Don’t worry about creating a three-year plan for improvement; create a one-month plan and just get to work instead. At the end of that month, evaluate your learnings, refine your focus and get back to work. With a commitment to hard work and continuous improvement, your investments should start paying off and, as an added bonus, you’ll have a better culture.
We hope you have enjoyed these juicy findings as much as we have and we encourage you to read the entire 2022 State of DevOps Report.
Also, please join us at dora.community to continue the discussion about these findings and to share and discuss your experiences on your journey toward excellent software delivery and operations.