Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.
This week, four things that caught my eye from Amazon Web Services’ re:Invent 2022 conference: SnapStart turbocharges Lambda, Graviton3E ARM HPC SoC, AWS continues hiring, and AWS’s origin story. (more…)
The whole concept of Agile and DevOps was to iterate development faster and deliver results in a more timely manner. As we learned more about both methodologies, processes and policies were put into place that improved the quality of what was created. Early ideas of quality like, “We can just do another iteration,” still exist, but are largely not used. Rollbacks are a thing in most orgs, and though the environment is more complex, rollbacks are easier to manage in the smaller development changes that iterations generally represent.
And once we got past the whole, “Anything that slows the process down is bad!” mentality that permeated early implementations of both DevOps and Agile development, we started seeing amazing things. But including things like security and functional testing were not viable in the fast-iteration world that Agile and DevOps created. And they really weren’t viable at most orgs in the slow-iteration world of waterfall, either. Staffing and time constraints made them fall far short of optimal in the vast majority of organizations. But putting them into the DevOps process challenged vendors to find a way to do things faster and challenged organizations to look at quality and security a little differently.
And now, we have tools that can test—be it functional, integration or security—at the speed of DevOps. Tools that enable Agile development with an eye toward security. A far larger number of developers now acknowledge their part in application security and are creating more secure code.
Keep an eye on how AI/ML is being used in testing—all forms of testing—and how it evolves in the next year or two. The same with full-on automation. I’ve made no secret that I view AI/ML in the testing space as advanced automation because of what it enables, so maybe “Watch what AI/ML-enabled automation is available,” is more accurate. The ability to create, run, process results and even modify code to resolve issues are all out there today, and getting better. Faster, more accurate, more automated.
Much like back when intrusion detection became intrusion prevention, I would not just turn it on and walk away, but think about the ability to upskill developers; “Here is a list of things the automated system found, and a list of fixes it proposes.” They are literally being shown how to do their craft better.
I’ve been a developer, so I’m not being offensive, here. If you get something that tells you, “This is an issue; here is why it happened and here is what to do to fix it,”take the opportunity to get better at coding. Look closely at the what, the why and how to fix it and incorporate that into your work.
And keep rocking it. We’ll be making more stable and secure code, but you’ve kept systems running without that feedback and fixed the errors along the way. This just makes it easier to find the errors early—before they crash production and make a fire drill.
There are plenty of historical examples of good solutions being buried by inadequate or outright wrong sales and marketing tactics. As an analyst, I often talk to the marketing, marcom or AR/PR department of a company and think they could do their solution more justice. Normally, it is little things; sometimes, not so much.
But there is one industry-wide thing that I would like to see changed because it impacts all of us and is endemic.
We don’t want to hear about your five biggest customers. We’re thrilled that you sold into several Fortune 100s. But you spent those five slides of your presentation basically saying the exact same thing five times: “We sold into a Fortune 100!” (or 500, whatever). You lost the opportunity to expand the message to “… and we support all organizations.” The idea that supporting a company with a run rate larger than most nations is in any way relevant to supporting small-to-medium-sized enterprises is mistaken. Talking about an example of a large customer helps us to understand scalability and support levels. Then, we want to know how that works for the rest of the market because most organizations are not Fortune 500s, by definition.
Tell us how the product is used in organizations closer to our size. Instead of five slides about massive multinationals, include the best two that showcase the product and its benefits. Then, use those other three slides to talk about the wider market. People expect that large enterprises like the Fortune 500 pay more in raw dollars and get more support in raw hours. They want to understand how that relates to an organization their size. Because if you don’t work at an F500, information about F500s has limited usefulness.
I recently had a presentation with a vendor; I love the technology, have been following them for years and use their solutions in a small business (not the reason for the presentation, but an expression of my belief that they offer a good solution). And their presentation was all about how well they serve large organizations. Thing is, they do a great job of serving small organizations, too, but unless you were an analyst like me, you would not have known that from the presentation. I get a lot of briefings, so while this one spurred me to write this blog, they are not the only ones, by far.
So do us a favor: Give us presentations that are relevant to the entire market, not the subset someone told you is “most important.” It is, indeed, impressive if the solution can scale to multinationals and it is relevant. Just don’t obsess about it; we need to know if it supports our architecture, tooling, etc.
We’re here rocking it every day, you should bring the broadest focus you have to let us know if you can be part of the solution moving forward. If you don’t, well, we might just think our business isn’t all that important to you. And we’re pretty sure most vendors don’t want that.
Aqua Security this week claimed it is the first software supply chain security platform provider to meet the attestation requirements as defined by an executive order issued to federal agencies last year by the Biden administration.
A supplementary memo issued by the Biden administration required federal agencies to collect compliance attestation letters for all software they use by September 14, 2023. Those attestation letters need to confirm that the software development environments they employ are secure, that sources of code are trusted and that code vulnerabilities have been remediated, provenance for data and code is being tracked using a software bill of materials (SBOM) and that these processes are being consistently maintained.
Eilon Elhadad, senior director of supply chain security for Aqua Security, said Aqua Security was able to meet those requirements within 100 days of the issuance of that memo. The Aqua Security platform for securing software supply chains is based on a platform it gained with the acquisition late last year of Argon, a provider of a set of tools that provides an agentless approach to integrating with continuous integration/continuous delivery (CI/CD) platforms that maps the development environment. Armed with those insights, it then becomes possible to prioritize and automate the remediation of vulnerabilities based on security best practices and compliance rules, noted Elhadad.
The Argon platform also includes code-tampering detection technology that performs validity checks to make sure the integrity of the build process has not been compromised.
Naturally, there’s plenty of time for other security platform providers to comply with the requirements set forth in Biden’s executive order. However, software development teams are also in a race against time. In the wake of a series of high-profile breaches, it’s become apparent how vulnerable software supply chains really are. Cybercriminals have increased their efforts to compromise developer credentials so they can embed malware in software components and activate them downstream if future application incorporate that component, noted Elhadad.
It’s not clear how closely enterprise IT organizations are tracking the efforts of the U.S. government to secure its software supply chain, but any entity that builds software used by any of those agencies will be impacted. Advocates of DevSecOps best practices within enterprise IT organizations will undoubtedly use the Biden administration’s requirements to define a minimum baseline for securing software supply chains.
In the meantime, a new front has clearly been opened in the cybersecurity war. The more applications are targeted, the greater the impetus for shifting responsibility further left toward application development teams becomes. The challenge is not just scanning the code being written by developers, but also making sure the software build created by development teams has not been compromised by bad actors that have inserted themselves into the process, noted Elhadad.
Given the magnitude of the effort required, it’s not likely that every software supply chain will be secured any time soon. However, the responsibility for securing those software supply chains is weighing more heavily on DevOps teams.
At the VMware Explore 2022 Europe conference, Hewlett Packard Enterprise and VMware, Inc. this week extended their alliance to integrate instances of the VMware Cloud platform into the managed HPE GreenLake service.
Hang Tan, chief strategy officer for HPE, said for the first time HPE is managing the entire VMware Cloud stack in addition to its servers and storage platforms via a service that is based on a consumption pricing model. The goal is to reduce the total cost of IT by not requiring IT teams to overprovision IT infrastructure resources to run VMware Cloud, he said.
Previously, IT teams could have deployed VMware Cloud on the HPE GreenLake service in a more disaggregated fashion, but that would have required separate billing processes, added Tan.
Many organizations are currently revisiting their IT strategies during increasingly challenging economic times, noted Tan. The managed HPE GreenLake service enables IT teams to devote more of their internal IT resources to building and deploying applications rather than managing IT infrastructure, he added.
It’s not clear how many organizations are transitioning toward a subscription model for consuming on-premises IT infrastructure, but in the cloud era it’s apparent that more of them are comfortable consuming IT as a service. HPE reported that the HPE GreenLake service supports more than 120,000 users accessing more than one exabyte of data worldwide. Among the top 100 HPE customers, nearly 80% have already adopted the HPE GreenLake platform, the company claims.
The challenge many organizations face is they still need to deploy and update application workloads in on-premises IT environments for both performance reasons and to meet various compliance mandates. The HPE GreenLake service is an effort to bring a cloud operating model to on-premises IT environments running not just VMware Cloud but also a wide range of other rival software stacks that HPE already supports.
Naturally, organizations that have embraced DevOps workflows would need to align the way they build and deploy applications to a managed service. IT teams that find that challenging, however, could opt for a self-service edition of the HPE GreenLake service that provides IT teams with more control over the underlying infrastructure being consumed as a service, noted Tan.
Regardless of approach, it’s apparent that some type of as-a-service approach to consuming IT infrastructure is being more widely employed across what are becoming highly distributed computing environments. Each organization will need to determine for itself to what degree treating IT more as an operating versus capital expense makes the most economic sense.
However, there’s no doubt that infrastructure cycles are generally going to be less disruptive when they are included as part of a multi-year services agreement. IT teams may also find it easier to take advantage of ongoing artificial intelligence (AI) advances when IT environments are managed on their behalf by a vendor.
One way or another, the way IT infrastructure is managed is continuing to evolve at a time when organizations are starting to focus more of their efforts on applications that are becoming more difficult to build, deploy, secure and maintain.
At the VMware Explore 2022 Europe conference, IBM this week announced an IBM Cloud for VMware-as-a-Service offering that integrates instances of VMware Cloud with the infrastructure-as-a-service (IaaS) platform managed by IBM in the cloud.
Rohit Badlaney, vice president of product management and strategy for IBM Cloud, said now that VMware Cloud can be consumed as a subscription service, it’s now possible for IBM to integrate it with its cloud platform to create a fully managed service. Previously, IT organizations would have had to separately license VMware Cloud and deploy it on IBM Cloud themselves, he noted.
In general, Badlaney said IBM is focused on working with organizations in highly regulated industries that need to run complex workloads in the cloud. IBM is not interested in competing against other cloud service providers to run workloads on commodity infrastructure, he added.
The majority of applications that run in the cloud today consume a narrow range of memory and storage resources. The IBM cloud is clearly more focused on providing higher-margin services to complement the cloud infrastructure it makes available via its data centers. Less clear is to what degree monolithic applications running in on-premises IT environments are being lifted and shifted to the cloud. While there is clearly a segment of organizations that no longer want to operate their own data centers, the bulk of applications continue to run in on-premises IT environments. The reasons for this include everything from performance requirements and security concerns to regulatory compliance issues.
However, since most on-premises applications run on VMware virtual machines, much of the focus on cloud migration has revolved around platforms that would not require IT teams to refactor applications to run on a different virtual machine platform. IBM is also betting that, given the current state of the global economy, more organizations will soon be opting to not manage their own data centers. Cloud platforms managed by a third party would enable those organizations to focus more of their own internal IT efforts on application development and deployment.
Each organization will need to determine for itself whether to treat IT as an operating versus a capital expense, depending on which approach makes the most economic sense. Organizations that have embraced DevOps workflows would need to align the way they build and deploy applications to a managed service.
In theory, of course, a managed cloud computing service makes IT teams more agile and efficient. Less IT infrastructure winds up being overprovisioned by developers that can programmatically spin up resources on demand. Of course, there are a lot of IT stacks other than VMware Cloud that can be deployed in the cloud. IBM, for example, makes available the Red Hat OpenShift platform to run cloud-native applications on Kubernetes clusters. IBM has not yet committed to offering the VMware Tanzu platform for running those types of applications.
Not every application is going to wind up running in the cloud, but what is certain is that application environments are becoming more complex to manage in the next era of cloud computing.
Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.
This week: People won’t shut up about Rustlang, and Musk mandates Twitter teams return to the office. (more…)
From Twitter and Netflix to Salesforce and Confluent, modern analytics applications are being built by developers across the world’s leading digital businesses to create digital tools that serve real-time analytics to hundreds or even thousands of concurrent users.
Looking more closely at what these apps do reveals some critical, strategic technology decision-making. In particular, why do developers increasingly turn to Apache Druid instead of the myriad of other databases out there?
In general, they look to Druid to build apps that deliver on a range of objectives, including creating operational visibility at scale, customer-facing analytics, rapid-drill down exploration and real-time ML pipelines. Let’s take a look at each of these in turn:
Building apps for operational visibility can include a wide range of use cases, such as product analytics, digital operations, observability, fraud detection and IoT. In each situation, developers need to minimize the gap between events created and time-to-insight.
From a technology standpoint, there are several databases that can analyze events in real-time, including stream processors like Apache Flink or ksqlDB, time-series databases like InfluxDB or TimeScaleDB and also key-value stores such as Redis. The problem is that all of these technologies share the same inherent limitation for analytics: They aren’t able to process OLAP-style queries effectively at any meaningful scale.
For instance, if the app in question calls for analyzing time and non-time-based dimensions over the past month, assuming ingestion was one million events per second, that would equate to more than 2.6 trillion events. Any database that supports stream ingestion can query the current status (setting aside the challenge to handle ingestion at this scale) and, to an extent, basic aggregation of numbers and counters. Difficulties can arise, however, if the app requires aggregation across all of these events, “group by” on non-time-based attributes or high concurrency. Using the wrong database leaves users looking at the ‘spinning wheel of death’ as they wait for results.
In these circumstances, devs turn to Apache Druid. A real-world example explains how: Salesforce engineers built an analytics app using Druid to monitor their product experience. The app enables engineers, product owners and customer service reps to query any combination of dimensions, filters and aggregations on real-time logs for performance analysis, trend analysis and troubleshooting. As a result, their query results are returned in seconds with billions or even trillions of streaming events ingested every day.
On the face of things, the technologies offered by Twitter, Forescout and Atlassian don’t have a huge amount in common. Look beneath the surface, however, and they all apply analytics to more than just internal decision-making. They have also created analytics apps and processes to provide their customers with powerful insights as part of a value-added service or a core product offering
What does that teach other organizations? When building analytics for external users, a whole different set of tech challenges must be addressed. While concurrency is clearly a key consideration, the database should also support current and future user growth with ease. The thing is, concurrency also has to be delivered economically, or it can have a huge impact on infrastructure costs.
Similarly, delivering a good interactive experience is dependent on sub-second query response performance—it has become a minimum user requirement almost everywhere. Instead of fighting PostgreSQL or Hive or any other analytics database to meet the performance requirements at scale, Druid offers the performance devs and users expect.
Take Confluent, for example, which is a popular event streaming platform. They built their cloud-based monitoring service, Confluent Health+, with Druid. Having initially used a different database, they discovered that, as their data volume grew, their legacy pipeline struggled to keep up with their data ingestion and query loads. By switching to Druid, they have been able to create the impressive user experience developers everywhere are focused on delivering.
Generally speaking, the primary role of reports is to fulfill a simple requirement: Deliver data in a consumable form. What organizations are increasingly looking for, however, is a deeper level of insight and understanding that shows why something happened. They can take this information and apply it to a wide range of possibilities, such as solving, anticipating or preventing future problems or opportunities.
Addressing these ‘why?’ questions means data has to be analyzed at tremendous speed to reveal root causes. While this is a simple proposition when the data set in question is small, when dealing with data drawn from today’s high-volume sources, such as cloud services, clicks and IoT/telemetry, the underlying database has to correlate a few to hundreds of dimensions across highly cardinal data with billions to trillions of rows. That presents a much more difficult challenge.
Indeed, when data is analyzed at a huge scale, full table scans are simply too slow. Exacerbating the problem is the typical query-shaping techniques used to speed up performance have expensive tradeoffs:
Looking at real-world use cases once again helps demonstrate the role of Apache Druid in situations such as these. For example, developers at cybersecurity company HUMAN built an analytics app for internet bot detection using Apache Druid. Previously, their cloud data warehouse was too slow to keep up with their need to drill into massive web traffic quickly. Today, their Apache-powered app gives their data scientists the ability to aggregate and filter across trillions of events instantly, meaning they can classify anomalies and train their machine learning models to detect real-time malicious activity.
For many people, analytics is synonymous with viewing a synthesized data set presented via a UI like a chart, graph, report or dashboard. Increasingly, however, new operational workflows are completely automated without any human intervention or data visualization.
Google Maps is an example of this use case, given it optimizes routing based on real-time and historical traffic flow and patterns. Another example is Reddit’s use of Apache Druid to deliver real-time ad placement within milliseconds of a new visitor arriving on the app. In this context, there is no data visualization to interpret; it has been replaced by automated decisions derived by inference. These performance requirements play particularly well to the capabilities offered by Druid.
Across a growing variety of demanding use cases, devs are adopting Druid to build inference-based capabilities into their apps. From diagnostics, recommendations and automated decisions, today’s organizations need instant query responses of high cardinality, alongside highly dimensional event data at scale and in real-time. Delivering performance levels like this is becoming increasingly important for businesses focused on innovation, quality of service and competitive advantage.
It is pretty amusing, the number of software vendors that are selling us tools to work in our Agile and/or DevOps infrastructure but are not at all agile in their sales cycle implementation.
Top-line vendor hint: If you have a page titled “pricing” and there are no prices on it, you’re doin’ it wrong.
Let’s be serious, here. I should not have to drop everything, call you and go through a whole lengthy, involved process before I can implement a product that you claim makes my development or deployment life cycle more agile.
Beyond the ‘there is no price on the pricing page’ issue, there are the vendors that go to great lengths to keep us from knowing what they cost. Some fail because they have public cloud images with pricing on them, but some even get around that by including “bring your own license” and selling the instance cheaply.
In almost every case, competitor pricing is available, is simple and TCO can be calculated. If yours can’t, you are not winning mindshare.
I probably recounted here that, once, a big-name database vendor tried to charge us insane amounts using the “You have to talk to us” model. I escalated to my VP, who told the salesperson, “I am not buying your Lamborghini,” and we then paid to have the product ported to a competing database with open and easy pricing and minimized use of the product on the offending DB. The moral of the story is, “Keep it simple and make it worthwhile.” We would have paid millions more than we did (which was already a lot) for that project. The vendor got nothing by making convoluted pricing schemes that allowed them to say every endpoint was a user and insisting we should pay the full license price for them.
Top line vendor hint number two: We’re professionals. We install, configure and manage a variety of systems every day. If your system requires services to get up and running, you should consider why that is.
We are not looking to spend months scheduling installation and configuration because you, the vendor, think services are a mandatory part of implementation. We can deploy entire infrastructures without requiring assistance; what is wrong with your product that we cannot deploy it in the same manner?
In general, organizations are willing to pay well for a product that does the job. Hidden pricing, confusing pricing and mandatory services to get the system running all make it harder for an organization to evaluate the value of the product. Saying “Start Free Trial” without explaining pricing is perhaps the funniest vendor move ever. Of course we want to know what we might be getting into before we sign up for that free trial!
Sell us good tools at a clear and reasonable price. We’ll buy them. Because we have a job to do and we want the tools to do it. We don’t want to drop everything to find out what it will cost us to do it and we don’t want to be forced to buy services to get it running.
A survey of 500 DevSecOps professionals in the U.S. found nearly three-quarters (73%) of organizations plan to increase investment in application security in 2023.
The survey, conducted by Wakefield Research on behalf of Invicti, a provider of dynamic application security testing (DAST) tools, also found 97% of DevSecOps teams said they ignored a real vulnerability at least once a month because they assumed it was a false positive.
Invicti CTO and head of security research Frank Catucci said that while it is apparent organizations are under pressure to improve the security of their software supply chains, the tools and processes used to discover vulnerabilities need to be revisited at a time when the security technical debt accrued in production environments is already massive.
Many DevOps teams have routinely shipped code with known vulnerabilities for years, noted Catucci. They are not only under pressure to improve the security of new applications but also remediate applications with known vulnerabilities already running in production environments, he added.
Cybersecurity teams have, of course, been identifying those vulnerabilities for years. The challenge is that the long list of vulnerabilities cybersecurity teams provide to DevSecOps teams tends to lack any context. As such, it becomes impossible for developers to prioritize their remediation efforts based on the actual severity of the vulnerabilities found. DAST tools help address that issue because they provide insights into vulnerabilities in applications already running in a production environment.
Application security, of course, has historically been problematic because many developers assume cybersecurity teams are securing the environments where applications are deployed. Most cybersecurity teams, however, allocate their budgets to tools and platforms they control and manage. Most of those tools and platforms are focused on security operations tasks that tend to have more to do with securing infrastructure rather than applications. The assumption is application development teams are responsible for securing the software that runs on those platforms. That disconnect is now finally being addressed as responsibility for application security shifts left to become an integral part of DevOps workflows.
The Invicti survey suggested that application security budget allocations are now catching up with that transition. In fact, the Invicti survey found that 100% of DevSecOps professionals are tracking the return on investment (ROI) on the application security tools and platforms they employ. Of course, measuring the effectiveness of any cybersecurity tool or platform has always been a challenge. It’s never quite clear whether a breach didn’t occur because a tool or platform was present or if it didn’t happen because of sheer luck and happenstance.
Regardless of the ROI achieved, however, it’s clear that cybercriminals are increasingly targeting software supply chains. The recent executive order issued by the Biden administration requiring federal agencies to review the security of their software supply chains got the attention of traditional enterprise IT organizations, as well. As such, the overall state of application security should steadily improve in the months and years ahead, as more funding is allocated to prevent vulnerabilities from finding their way into production environments in the first place.