Successful new technologies follow a predictable arc, from initial innovation to technical gold rush and, ultimately, to mass market adoption. As a consequence, they achieve strong standardization and productization. Being highly attuned to where a specific field lies on the maturity arc is how you yield the greatest benefits to a business. But what happens if your developers are not thinking about this, too? You end up with what I call a “resume-driven architecture,” and it can be highly costly to your business.
A resume-driven architecture occurs when the interests of developers lead them to designs that no longer align with maximized impacts and outcomes for the organization. Often, the developer clings to a technology that provides them a greater level of control and, at least initially, a higher salary. Meanwhile, the organization gets an architecture that only a handful of people know how to manage and maintain, limiting the available talent pool and hindering future innovation.
To understand why this happens and how to avoid it, it’s helpful to look at this technology arc in more detail. It’s a pattern that has played out up and down the stack, with technologies like big data platforms (now dominated by Snowflake and Databricks), Kubernetes and even the broader move to the cloud. Failing to grasp this concept of a resume-driven architecture can increase costs and hold back innovation.
Here’s how this technology arc typically plays out, using Apache Spark as an example:
- Stage One: A technology breakthrough like Spark emerges, providing a significant advantage to early movers.
- Stage Two: Other companies see the benefits, and the technology receives widespread interest. Specialists are in high demand and command big salaries. Many adjacent professions naturally invest in becoming Spark experts.
- Stage Three: The technology matures and reaches a level of standardization with an ecosystem of tools and services around it. Specialized knowledge, such as how to tune your Spark job parameters, becomes less valuable.
- Stage Four: Mass convergence, productization and simplification occurs. As examples, Databricks becomes the primary Spark vendor and launches SQL warehouses for easier data processing. Snowflake launches Snowpark for Python as a simpler model than classic PySpark jobs. Nobody worries about tuning most parameters anymore as the underlying technology becomes part of the stack: Powerful but almost invisible.
It’s important for businesses to take advantage of this productization stage when it comes along. There’s no sense in investing resources in a bespoke architecture if it’s not providing you with any differentiation—especially when competitors are achieving the same outcome with fewer resources.
Moreover, getting stuck in a Stage Two mindset when the field moves on to Stage Three (or, worse, Stage Four) and cuts you off from the next wave of innovation. Subsequent technology breakthroughs often build on top of—and interoperate with—the previous technology layers. If you’re stuck with a custom architecture when the industry has moved on, you can miss out on the next wave of innovation and fall further behind competitors.
The problem is, developers who’ve made a great career from implementing custom architectures are often reluctant to move on. After all, mastery of that previous innovation is what got them to where they are. They may not have realized that the world has progressed around them, or they may worry a standardized architecture with managed services and off-the-shelf products undermines their power and autonomy. This is how organizations fall victim to a resume-driven architecture. I’ve observed this several times when speaking with engineering leaders, and it’s become more common with the rapid innovation happening around data.
As engineers, our value is in building things and, ideally, in solving innovative, artisanal problems. But this identity and desire can confuse what we want from our own jobs with what our team needs from us. When standard patterns and standard products emerge, we need to adopt them because that’s the way to maximize our impact and ultimately free our time and energy to move further “up stack” and have an even greater impact.
Developers who would rather stick to a custom architecture often argue that they can squeeze out greater performance, but this is rarely a justification. You may end up with an app that’s 10% faster, but rarely does that come close to making up for the time and energy invested, the loss of flexibility and, ultimately, being held hostage by your own technology stack.
So how does an organization avoid the trap of a resume-driven architecture? Strong leadership is key. Look critically at the business and ask which areas of your stack are truly differentiators because it’s only those areas that may—just may—justify a unique architecture. If you do go that route, it needs to provide huge benefits to the business to be worthwhile.
For example, “data” is not in itself strategic to you; every company is now a data company, and the vast majority of things you do with it—how you source, store, process and distribute data—is no different from what everyone else is doing. What does matter are the data products that fuel your business and, most importantly, how fast you can build them.
That’s why, as a developer or engineer, bespoke stacks are not your friend. Instead, architect your platform in a way that allows other developers to onboard quickly; that allows you to upgrade and swap out components easily over time and that doesn’t leave you held hostage to a developer whose main interest is retaining their value by sticking with an unnecessarily complex architecture.
