devopsy.kinsta.cloud

Category: Blogs

  • Pulumi Embeds Terraform Conversion Tool in Platform

    Pulumi Embeds Terraform Conversion Tool in Platform

    At its virtual PulumiUP event, Pulumi this week announced it embedded an ability to convert instances of Terraform into its infrastructure-as-code (IaC) platform via a simple command line interface (CLI) command. In addition, Pulumi added a Review Stacks capability that automatically creates a dedicated cloud environment to review code every time a pull request is opened.

    Pulumi is also making available a beta release of Pulumi Azure Native Provider 2.0, which decreases the size of the software development kit (SDK) by 60%. This is part of an effort to improve deployment times and reduce resource consumption by, for example, removing support for deprecated application programming interfaces (APIs).

    Finally, Pulumi has added support for property names and values to Resource Search for Pulumi Cloud, a search and analytics tool for cloud computing environments. DevOps teams can now additionally search for instances of, for example, versions of databases or specific availability zone as a result.

    Aaron Kao, vice president of marketing for Pulumi, said previously the company made available a separate tool for converting snippets of Terraform code. For the first time, Pulumi is also making available a converter plugin in the Pulumi engine which allows conversion tools from other IaC tools to be used.

    Review Stacks, meanwhile, makes it simpler for DevOps teams to pick up an unfamiliar codebase, make changes to both application and infrastructure code and share a live environment to review that code.

    Pulumi has been making a case for an alternative to Terraform that centralizes the management and provisioning of infrastructure using a set of embedded guardrails. One of the major issues enterprise IT organizations are encountering these days is the number of misconfigurations created when developers use tools like Terraform to provision infrastructure on their own. Those misconfigurations often lead to, for example, ports being left open and cybercriminals exfiltrating data.

    That approach is gaining traction among enterprise customers that are looking to centralize the management of DevOps processes, said Kao.

    In the wake of a series of high-profile breaches, more organizations are now reviewing the security of their software supply chains, including how infrastructure is provisioned. Many of those organizations will need to revisit those processes as pending legislation that requires more stringent requirements for building software eventually become law, noted Kao.

    At the same time, many organizations are also looking to improve the productivity of DevOps teams during an economic downturn that limits the number of full-time employees they might be able to hire. The pace at which software is being developed has not slowed, so there is a greater urgency to do more with less.

    It’s not clear how quickly organizations are centralizing the management of DevOps workflows, but with the rise of platform engineering, there is clearly more interest in defining consistent backend processes that multiple DevOps teams are currently managing individually. The challenge is finding a way to achieve that goal without impeding the pace of innovation enabled by embracing DevOps in the first place.

  • 92% of Devs Use AI, Survey Says ¦ Intel One Mono Font

    92% of Devs Use AI, Survey Says ¦ Intel One Mono Font

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: GitHub touts unbelievable AI stats, and Intel’s beautiful open source typeface. (more…)

  • Four Ways to Modernize Your Enterprise Monoliths

    Four Ways to Modernize Your Enterprise Monoliths

    In a perfect world, every application would run like Netflix–by which I mean every application would be cloud-native, microservices-based, containerized and, in all other senses of the word, modern. Unfortunately, not every application can be transformed in these ways. Some of the most important enterprise applications–like those powered by SAP or Oracle–don’t lend themselves to architectural transformations simply because the vendors don’t support that strategy. As much as organizations might wish that they could re-architect legacy apps from the ground up, the reality is that they just can’t.

    Fortunately, though, all hope is not lost if you have an enterprise app that you can’t containerize or “microservices-ize.” Even if you can’t “break down your monolith”–to quote the standard advice you’ll find if you read articles about how to modernize legacy apps–there are a variety of practical steps you can take to make legacy, monolithic applications look and feel (in most respects) like modern, cloud-native apps.

    Here are four ways to make your enterprise monolithic apps more cloud-friendly, even if they’re not technically cloud-native.

    Use Cloud Automation to Modernize Legacy Application Management

    Most enterprise apps are compatible with the cloud in the sense that you can host them on cloud-based virtual machine (VM) instances using services like Amazon EC2.

    By taking advantage of cloud hosting for enterprise apps, organizations can also leverage cloud automation tools, such as autoscaling policies and rules that automatically start or stop VMs. In this way, it becomes possible to deploy enterprise apps in a highly scalable, automated way. You get something that resembles Kubernetes-based orchestration, except your apps don’t actually run in containers on top of Kubernetes. They’re just monoliths that benefit from cloud automation tools.

    Deploy Load Balancers to Optimize Performance

    Load balancers, which automatically manage network traffic flowing to and from applications, are another critical tool that organizations can easily leverage when they migrate legacy applications to the cloud.

    Although configuring load balancers for enterprise apps hosted in the cloud is not a strict requirement, doing so can substantially improve application performance and reliability because load balancers help to ensure that requests are directed at application instances best able to handle them. They prevent situations where one instance of your application is overwhelmed with traffic while another sits idle.

    Again, this is something that platforms like Kubernetes also do for microservices apps. But, as I’ve explained, you don’t actually need your app to run as microservices to take advantage of load balancing. You can do it with monolithic legacy apps, too.

    Adopt an Everything-as-Code Strategy

    When your app lives in the cloud and is managed by cloud automation tools, it becomes possible to manage most aspects of application operations via code. Specifically, you can:

    ● Use infrastructure-as-code (IaC) tools to automate infrastructure provisioning and setup.
    ● Automate application configuration management using code rather than applying configuration changes manually.
    ● Codify operational processes, such as application updates and backups, instead of relying on manual operations.

    Practices like these, which enable a strategy known as everything-as-code, significantly reduce the amount of time and manual effort that engineers spend managing applications. They also provide consistency and minimize the risk of errors due to human oversight. And there’s no reason why you can’t leverage these practices for legacy apps as well as modern ones.

    Integrate Legacy Apps With Modern Toolchains

    If your enterprise app is hosted on-premises, you probably manage it using tools and processes designed solely for it. But when you move an enterprise app to the cloud, it becomes possible to manage the application using the same integrated toolchain you deploy for other applications and processes.

    You can, for example, take advantage of cloud-based monitoring tools to continuously collect data about application performance and generate automated alerts when something is wrong. Likewise, you can regularly create image-based backups of your application.

    This toolchain-based operational strategy is another way to streamline enterprise application management, even in situations where you can’t rewrite or containerize the application itself.

    Conclusion: Getting More From Legacy Apps

    Just because you have a legacy enterprise app that you can’t transform into microservices doesn’t mean you have to settle for legacy management and operational practices. Modern automation tools combined with modern operational strategies like everything-as-code make it possible to achieve an enterprise application management experience that closely parallels that of fully modern, cloud-native apps, regardless of which architecture your app actually uses.

  • How Open Source Can Enhance Developer Experience

    How Open Source Can Enhance Developer Experience

    Developer experience (DX) is becoming more and more of a competitive advantage among developer-facing tools and platforms. A great developer experience that’s easy to follow and implement can truly help a software tool stand out from the crowd.

    Simultaneously, open source software is all the rage. In fact, a recent StackOverflow survey found open source software (OSS) to be, by far, the most revered and proven technology trend.

    With that in mind, what are the synergies between developer experience and open source? And how can building in an open source format aid the developer experience for your public-facing toolsets and internal development platforms?

    I recently met with Adam Frank, SVP of product and marketing at Armory, to explore the state of developer experience and open source adoption. According to Frank, open source can bring numerous DX benefits. Below, we’ll explore these areas and consider why organizations might benefit from adopting more open source within their developer-facing initiatives.

    The State of Developer Experience and Open Source

    So, what is the state of developer experience, and why is it important now? Well, DX is continually leveraged to help improve agility and competitiveness in the market. In addition to granting differentiation, DX also enables more flexibility, helping users switch from one tool to another if necessary.

    “Developer experience is at the forefront right now because technology and change are moving faster than ever before,” said Frank. “Companies need to focus on DX as a competitive advantage to survive and thrive.” At the same time, open source can help organizations move quickly and iterate more efficiently.

    Yet, historically, open source has not been associated with fantastic DX. Most of us have at least one horror story of an open source project simply going belly-up without warning. It’s common for an OSS tool to lack comprehensive documentationor suffer from a lack of funding for ongoing maintenance, zero customer support and deprecated modules. Some tools that developers directly download and contribute to may not possess a quality DX emphasis, says Frank.

    However, the status quo for OSS has undoubtedly evolved over the years, he adds. Nowadays, there are more established groups, like Linux Foundation’s Cloud Native Computing Foundation (CNCF), with specific Technical Oversight Committees (TOCs) to oversee individual projects. The maturity of OSS ownership now ensures core projects stay relevant and receive continual support—not to mention ongoing developer experience enhancements.

    On the other side, OSS is often absorbed by a platform team to enable their internal developers to move faster or ship more code, says Frank. From this perspective, open source communities give platform engineering the opportunity to contribute back to the project, helping them solve deeper-level problems that are more specific to their environments.

    How Open Source Can Enhance Developer Experience

    Not all IT leaders are sold on the idea of building in a transparent, open source model. Yet, this enhanced transparency can bring dividends in the long run, especially where developer experience is considered. Frank shared some of the benefits of adopting open source.

    Receiving Feedback From The Community

    “As a product manager, you want as much feedback from your users as you can gather,” said Frank. Receiving user feedback is a great way to fix DX issues, and OSS gives you that on a larger scale. By adopting open source, you benefit from collecting insights from your customer base and the developer community at large. This can translate into a broader range of use cases, which can significantly improve projects and overall experience.

    Increasing Ecosystem Transparency

    Building in a more transparent and open fashion so that everyone has the ability to submit pull requests can also help you work together with potential partners. This makes it easier to develop third-party integrations that ensure the broader ecosystem can work together. By decreasing the barrier to entry through OSS, you provide others with the ability to solve more problems around developer experience.

    Streamlining the Developer Toolchain

    “Developers want to innovate; they want to solve problems,” said Frank. Open source can also help expose inefficiencies in the developer toolchain and, notably, many open source CI/CD tools are often leveraged to automate deployment and increase efficiency. The end result is that developers can focus on innovating their own competitive advantage. “Making the deployment aspect of the developer lifecycle really seamless can significantly improve their experience,” he said. Therefore, any tools or extensions around CI/CD are an excellent area to consider open-sourcing.

    Increasing Flexibility

    Another benefit of the open source model is that you have far more flexibility than off-the-shelf proprietary software. As a customer, you don’t have to wait to submit enhancement requests. With OSS, you can fork a project or quickly collaborate with the maintainers. This equates to making fixes far quicker and providing the ability to customize developer experiences to the tasks at hand.

    Best Practices to Ensure Open Source Improves DX

    Open source provides many key benefits that can enhance software development and delivery. But, this should be taken with a grain of salt, as best practices still must be followed to ensure open source enhances, not hinders, developer experience.

    For example, Frank mentions that simplicity should be a guiding light. “Have simplicity for the user experience design in the forefront of everything you’re building and thinking about,” said Frank. Adopting simplicity should help ensure developer-facing materials are structured well and documented in an easy-to-comprehend manner.

    On that note, open source projects will require standards around areas like good documentation and well-written APIs that abide by contracts, said Frank. Anything less will hinder developers and confuse them with disparate data formats. On that note, he has witnessed OpenTelemetry emerge as a common way to unify data that systems produce with standard formatting, which increases the observability of these environments.

    Lastly, it’s critical to hire talent with experience in open source. Bringing in folks that have knowledge of how OSS communities operate will make onboarding easier and encourage safer open source security practices as well.

    What the Future of DX and OSS Holds

    Open source appears to aid developer experience in many ways, from driving more innovation, opening software up to feedback and developing a closer-nit developer community. But what does the future of developer experience and open source look like?

    According to Frank, we will likely see increased data standardization that makes it easy to work with various projects and data formats. OpenTelemetry, for example, makes aggregating logging easier. We’ve also seen standardization around OpenAPI Specification for describing web APIs. Another example is FinOps Foundation’s FOCUS specification to unify cost analysis.

    Open source tends to leverage open source. And as the industry continues to coalesce around open standards, we’ll likely see more robust, compatible layers of abstraction emerge throughout the market, Frank predicts.

  • BMC Adds Generative AI Capabilities to Helix Management Platform

    BMC Adds Generative AI Capabilities to Helix Management Platform

    BMC today revealed it is embedding generative artificial intelligence (AI) capabilities across its BMC Helix service and operations management platforms.

    BMC CTO Ram Chakravarti said the company is incorporating the open source GPT-J large language models (LLMS) to build generative AI models. Those models are optimized for specific domains using data collected by BMC and imported from other data sources, like the Jira project management software from Atlassian. That persistent layer of storage is designed to aggregate multiple types of data within what amounts to a Helix data lake.

    The generative AI capabilities added to BMC’s portfolio include natural language summaries of issues and events captured by BMC Helix Operations Management, along with an ability to create a historical analysis of a user’s specific environment to identify recurring issues more easily.

    BMC Service Management, meanwhile, can now correlate incidents and provide resolution summaries that can be stored. It’s also now possible to run clustering analysis on chat archives to find new or improved utterances that help virtual agents answer queries better and execute workflows.

    BMC initially opted to build its own LLMs because general-purpose LLMs—such as the one created by OpenAI—are subject to “hallucinations.” Hallucinations are created when incorrect data is included in a model. BMC is training its LLMs on a narrow range of trusted data to ensure the recommendations and subsequent automations it creates are valid.

    Over time, BMC will add additional LLMs; the company hasn’t yet decided if there are use cases where it makes sense to rely on an existing general-purpose LLM to add additional capabilities, noted Chakravarti.

    Organizations can also opt to use a containerized version of the BMC LLM if they prefer, but BMC will encourage most organizations to invoke generative AI via its Helix SaaS platform, he added.

    BMC Helix is an IT services management (ITSM) platform that integrates with a range of DevOps platforms. Helix is part of a larger existing BMC Connected Digital Ops initiative that seeks to unify ServiceOps and DevOps with AIOps, DataOps and AutonomousOps. Generative AI will automate a range of tasks from within Helix that can then be extended to DevOps platforms such as Jenkins.

    It’s still early days as far as applying generative AI to processes, so organizations should focus on use cases where it will add some immediate value, said Chakravarti. Once they develop a level of trust in the automation created and executed, they can then pragmatically expand the number of generative AI use cases where they make sense, he added.

    One way or another, generative AI is going to be applied to the management of IT. Many routine tasks that conspire to limit the productivity of IT teams today will be eliminated, but there will always be a need for a human to remain in the loop to ensure the right outcomes are achieved. However, it will also soon become possible for existing teams to manage IT at levels of scale that were previously unattainable.

    Image source: Photo_Credite_Indira-Tjokorda via Unsplash

  • It Is Time for DevOps to Focus on Ops

    It Is Time for DevOps to Focus on Ops

    The advent of both Agile development and DevOps were great steps in improving an IT process that had become ossified, with policies that alternated between resistance to change and trying to do things better/faster. Between them, Agile and DevOps broke the cycle of resistance … but only for development.

    The confusion around CI/CD is a great example. CD is both continuous delivery and continuous deployment. Do you know why? Because up to that point (and even recently), DevOps included Ops in the name but did little with or for operations. I have no doubt that it was originally intended to include everything from build testing to full-on deployment, and we are finally getting there.

    You have no doubt noticed that both operations and security are finally getting DevOps’ attention at critical points. I have been chatting with coworkers about value stream mapping and, if you map the value stream of a software delivery process, you will find that all of the weak points have shifted right. Thankfully, we are also paying attention to them these days.

    From GitOps to infrastructure-as-code (IaC); from cloud-first to policy-as-code (PaC), we are finally approaching the operations side with some level of seriousness. And it is generating results for those involved.

    My two areas of attention these days are DevOps and application security (AppSec). As I’ve written elsewhere, both are benefitting massively from things like policy-as-code. The ability to apply corporate deployment policies, security policies and compliance policies from a centralized location is massive. It is early days, but community contributions to PaC libraries are already massive, making it easier to deploy without recreating that big old list of generally useful policies.

    Given a project, a set of tools and a timeline, you could fully automate the process from the first line of code to running in production, with stops for approval and the specialized steps that still require human interaction (no matter how many public policies are out there, someone with knowledge of your network still has to set up IPs or ranges, for example. Or someone still needs to write the code). Someday, it may be possible to say, “We need a phone app that targets both major platforms and is hosted on Linux that does X, Y, Z,” and it just happens … but we are not there yet, and, IMO, the domain knowledge will be a roadblock for a while. But you can take advantage of what we have to work smarter, not harder. And the organization will be better off for your efforts.

    So keep at it, start to evaluate your processes and consider placing any of the great new DevOps technologies into your path. Let them do some of the grunt work so you can focus on adding value. And for all those coworkers that still don’t get it, thank you. You are keeping them in a job, whether they admit it or not.

  • HashiCorp Releases Slew of Updates to Improve DevSecOps

    HashiCorp Releases Slew of Updates to Improve DevSecOps

    At a HashiDays event this week, HashiCorp launched updates to its Terraform implementation and a software-as-a-service offering, dubbed HCP Vault Secrets, that promise to simplify infrastructure security and ensure that secrets are not leaked.

    In addition, HashiCorp has added a self-hosted edition of its remote access security tool, Boundary, that can now also record SSH sessions. The company is also announcing general availability of Vault Secrets Operator, which provides a high level of abstraction for managing secrets in a Kubernetes environment and which synchronizes with the secrets management interfaces defined by the Kubernetes Technical Oversight Committee.

    Finally, HashiCorp is making available in beta a 1.16 update to the Consul service mesh for containers and virtual machines that adds a Sameness Groups capability to improve the resiliency of microservices by redirecting requests to an identical service. HashiCorp has also added a transparent proxy mode to enable failover between services along with a permissive mutual TLS (mTLS) that enables a service to be configured to accept both mTLS traffic from mesh-enabled services and non-Consul mTLS traffic from external downstream applications.

    Chris Van Wesep, senior director of product marketing for HashiCorp, said Version 1.5 of HashiCorp Terraform adds config-driven import workflow and check blocks capabilities that make it possible to declaratively use code to add existing resources into a Terraform state. Previously, the only way to update a Terraform state was to use an import command that could only be applied to one resource at a time. Terraform 1.5 also adds automatic code-generation capabilities for imported resources.

    In addition, version 1.5 of Terraform also includes a tool to validate that provisioned infrastructure is behaving as intended. That latter capability makes it possible to reduce the number of misconfigurations created by developers that might one day be exploited by cybercriminals to breach an IT environment.

    HCP Vault Secrets, meanwhile, provides a beta release of a SaaS edition of the existing Vault offering specifically designed to enable organizations to better manage application secrets that might inadvertently be exposed to cybercriminals in a production environment, said Van Wesep.

    HashiCorp is moving toward providing a portfolio of tools that enable DevOps teams to securely provision and manage hybrid cloud computing environments using a common workflow, he noted. A Forrester Consulting survey found nearly three-quarters (73%) of respondents are reporting this strategy is helping them reach their business goals, with another 19% adding they expected this strategy to help them achieve their business goals in the next year.

    It’s not clear whether organizations are fully embracing multi-cloud computing, but each additional cloud added to an enterprise IT environment increases the total cost of IT. The only way to mitigate those costs is to define a set of best practices for building and deploying applications that can be applied to any cloud computing environment.

    Unfortunately, many organizations will discover that, as they attempt to define those best practices, they have inadvertently become dependent on a set of proprietary application programming interfaces (APIs) that thwart those efforts.

  • Grafana Labs Slashes Time to Create Observability Dashboards

    Grafana Labs Slashes Time to Create Observability Dashboards

    At its GrafanaCON 2023 event today, Grafana Labs updated its core platform for visualizing data which makes it simpler to set up dashboards in minutes. The updates are part of a celebration of the 10th anniversary of a tool that is now widely used by DevOps teams.

    Ryan McKinley, a distinguished engineer for Grafana Labs, said Grafana 10 should help make DevOps more accessible to a wider range of practitioners with limited experience working with observability tools. The latest release also makes it simpler to share knowledge with external stakeholders via public dashboards, he added.

    In addition, Grafana 10 adds expanded role-based access control (RBAC) capabilities and streamlined implementation of the secure access markup language (SAML) to ensure authentication.

    Grafana Cloud has also been updated to provide a secure way to connect to data residing in an on-premises IT environment or in a cloud with the observability platform the company provides via a software-as-a-service (SaaS) platform.

    Finally, Grafana Labs is also making the engine it created to visualize data more accessible to third-party developers that are embedding visualizations based on Grafana within their applications, noted McKinley.

    Grafana Labs

    There are now more than 150 plugins for various data sources that have been made available to more than 20 million users of the Grafana visualization framework. Not all those users are working in DevOps environments, but Grafana is widely used to enable observability across increasingly complex IT environments. The Grafana Cloud seeks to capitalize on that popularity by adding a service that uses an instance of open source Grafana Mimir software to store data collected in a format originally defined for the open source Prometheus monitoring tool.

    It’s still early days as far as adoption of observability is concerned, but it’s apparent that as application environments become more complex, the plethora of monitoring tools that DevOps teams rely on will need to be streamlined. Observability platforms promise to unify logs, metrics and traces to simplify launching queries to identify the root cause of an issue.

    The rate at which DevOps teams will embrace observability will naturally vary, but the biggest obstacle might not be the platforms themselves. Instead, the issue is understanding what queries can help DevOps teams better understand the root cause of an IT issue before there is a major disruption. In the long term, it’s expected that machine learning algorithms will leverage the data collected by observability platforms to automatically identify issues that might lead to a disruption long before it actually occurs.

    Of course, the result of any query that gets launched needs to be visualized in a way that is easily understood. Grafana Labs is now making a case for a more integrated approach that, at its core, employs its visualization tools to make it simpler to discern what’s occurring in IT environments. After all, no matter how talented any DevOps team may be, it’s still not possible to manage what was never seen in the first place.

  • The Tragedy of the Data Commons: Balancing Innovation With Risk

    The Tragedy of the Data Commons: Balancing Innovation With Risk

    Clean drinking water and fisheries are classic examples of the “tragedy of the commons,” the problem in economics and ecology where an individual is incentivized to consume as much of a shared resource as possible at the expense of the larger society. With fisheries, for example, a logical fisherman would like to make as many catches as possible. However, this can ruin the environment for everyone, as overfishing can endanger an entire species and even threatens whole ecosystems. The modern IT environment has given us a new case to consider related to the tragedy of the commons: Data.

    Data, much like drinking water, is a critical resource. Developers need to use this data and, logically, they want to use it as much as possible to build innovative applications and products. However, too much data flowing around a modern distributed enterprise environment carries risks to the larger company.

    The data developers use often includes customers’ personal data like names and birth dates. The risks of this information being damaged or stolen can be catastrophic. This is where database administrators (DBAs), who are tasked with managing and securing this data, come in. There is an inherent tension between developers, who want as much access as possible, and DBAs, who want to limit that access for security and privacy reasons. How data is used within an enterprise is critical, because it must be used to create new solutions and be kept secure. An overly open data policy could lead to breaches and leaks, while an overly restrictive policy limits innovation.

    Thankfully, there is a solution that is helping ease this tension. Observability can automatically analyze massive amounts of information across a business’s entire IT environment. Users can use it to pinpoint the causes of outages or performance issues and receive actionable insights to resolve these problems quickly. Modern observability solutions even use AI to predict and proactively prevent problems before they occur. DBAs and developers can use observability to strike a balance while boosting innovation and improving productivity. Here are the three biggest ways that observability can solve the tragedy of the data commons and ease the tension between DBAs and developers.

    Ensuring Data is Secure

    One of the biggest concerns that DBAs have regarding data access is security. Sensitive production data are no longer stored exclusively on a company’s on-premise servers, where DBAs have full visibility into how it is being used and who has access. Instead, it is now stored across various private and public clouds as well as on-premises, which has reduced visibility for DBAs.

    This sensitive data is the same information that developers often need to build and launch applications in a continuous delivery model, where developers build software in short cycles so it can be released at any time. This involves working across different clouds, including AWS and Azure, while leveraging complicated cloud-enabled technologies like microservices and Docker containers. When data is deployed across multiple clouds, the attack surface increases while visibility decreases, leading to new security concerns.

    Observability provides DBAs with a better understanding of how data is being used by developer teams so they can more easily ensure that the data remains secure and protected. Additionally, many modern IT environments deploy threat intelligence solutions that observe actions and detect threat signals to help predict security incidents and respond before they happen. Some threat intelligence applications are powered by AI, which can automatically gather and organize complex log data from the entire network. This data is then compared with an out-of-the-box threat database, helping to detect suspicious activity early. Observability and threat intelligence working in tandem provide a comprehensive overview to help relieve some of the security concerns for DBAs and create a more secure environment for data to be shared between teams.

    Remaining Data Compliant

    Ensuring that data is compliant is top of mind for every organization. Compliance regulations such as GDPR, CCPA and HIPAA carry severe penalties for data leaks, meaning that companies must maintain the integrity and availability of sensitive and regulated data.

    Ensuring compliance while managing requests for information from developers working across geographies and clouds can be a daunting task for DBAs. Not only must they remain compliant, but many businesses must demonstrate continuous compliance, which is only possible using complex, dynamic and automated systems.

    Observability allows teams to clearly see and follow the complex flow of data as it moves between DBAs, developers and other teams. Additionally, observability can be deployed to help ensure that any sensitive data that is required to be anonymized is done so before it is shared. DBAs can further use observability to track the lineage of data as it moves between teams to monitor that it is not changed or transformed in any way that risks a compliance violation. Making compliance a regular part of the IT environment should not be viewed as a burden. In fact, it provides many benefits, including improving data quality, which can be used to further innovate and build better products.

    Making Teams More Productive

    Observability has shown it can create harmony between DBAs and developers by helping mitigate security and compliance concerns. Not only is it addressing these issues, but it is being used to make each team more productive on their own.

    For DBAs, this means gaining additional insights that help ensure applications and services are running properly. Observability collects information from across hybrid and multi-cloud environments and presents it in a single view to allow DBAs to respond quickly and resolve issues. When DBAs gain the insights that observability provides, they can better manage critical data and anticipate future problems before they arise.

    Observability is key to helping developers build applications as productively as possible. The ongoing shift left has necessitated the use of observability as a way to gain single-pane-of-glass visibility into the build process. Developers are also provided insights into the performance of web applications and can move quickly to address any problems. Finally, observability is improving the speed at which teams can innovate, as it provides live code profiling that can be used to predict and prevent future problems.

    Observability provides a real solution to ensure that data is not a resource that falls victim to the tragedy of commons, where its use can harm an entire organization. Instead, it is powering teams to use data securely to work more productively and build innovative applications.

  • Avoiding the Pain of a ‘Resume-Driven Architecture’

    Avoiding the Pain of a ‘Resume-Driven Architecture’

    Successful new technologies follow a predictable arc, from initial innovation to technical gold rush and, ultimately, to mass market adoption. As a consequence, they achieve strong standardization and productization. Being highly attuned to where a specific field lies on the maturity arc is how you yield the greatest benefits to a business. But what happens if your developers are not thinking about this, too? You end up with what I call a “resume-driven architecture,” and it can be highly costly to your business.

    A resume-driven architecture occurs when the interests of developers lead them to designs that no longer align with maximized impacts and outcomes for the organization. Often, the developer clings to a technology that provides them a greater level of control and, at least initially, a higher salary. Meanwhile, the organization gets an architecture that only a handful of people know how to manage and maintain, limiting the available talent pool and hindering future innovation.

    To understand why this happens and how to avoid it, it’s helpful to look at this technology arc in more detail. It’s a pattern that has played out up and down the stack, with technologies like big data platforms (now dominated by Snowflake and Databricks), Kubernetes and even the broader move to the cloud. Failing to grasp this concept of a resume-driven architecture can increase costs and hold back innovation.

    Here’s how this technology arc typically plays out, using Apache Spark as an example:

    • Stage One: A technology breakthrough like Spark emerges, providing a significant advantage to early movers.
    • Stage Two: Other companies see the benefits, and the technology receives widespread interest. Specialists are in high demand and command big salaries. Many adjacent professions naturally invest in becoming Spark experts.
    • Stage Three: The technology matures and reaches a level of standardization with an ecosystem of tools and services around it. Specialized knowledge, such as how to tune your Spark job parameters, becomes less valuable.
    • Stage Four: Mass convergence, productization and simplification occurs. As examples, Databricks becomes the primary Spark vendor and launches SQL warehouses for easier data processing. Snowflake launches Snowpark for Python as a simpler model than classic PySpark jobs. Nobody worries about tuning most parameters anymore as the underlying technology becomes part of the stack: Powerful but almost invisible.

    It’s important for businesses to take advantage of this productization stage when it comes along. There’s no sense in investing resources in a bespoke architecture if it’s not providing you with any differentiation—especially when competitors are achieving the same outcome with fewer resources.

    Moreover, getting stuck in a Stage Two mindset when the field moves on to Stage Three (or, worse, Stage Four) and cuts you off from the next wave of innovation. Subsequent technology breakthroughs often build on top of—and interoperate with—the previous technology layers. If you’re stuck with a custom architecture when the industry has moved on, you can miss out on the next wave of innovation and fall further behind competitors.

    The problem is, developers who’ve made a great career from implementing custom architectures are often reluctant to move on. After all, mastery of that previous innovation is what got them to where they are. They may not have realized that the world has progressed around them, or they may worry a standardized architecture with managed services and off-the-shelf products undermines their power and autonomy. This is how organizations fall victim to a resume-driven architecture. I’ve observed this several times when speaking with engineering leaders, and it’s become more common with the rapid innovation happening around data.

    As engineers, our value is in building things and, ideally, in solving innovative, artisanal problems. But this identity and desire can confuse what we want from our own jobs with what our team needs from us. When standard patterns and standard products emerge, we need to adopt them because that’s the way to maximize our impact and ultimately free our time and energy to move further “up stack” and have an even greater impact.

    Developers who would rather stick to a custom architecture often argue that they can squeeze out greater performance, but this is rarely a justification. You may end up with an app that’s 10% faster, but rarely does that come close to making up for the time and energy invested, the loss of flexibility and, ultimately, being held hostage by your own technology stack.

    So how does an organization avoid the trap of a resume-driven architecture? Strong leadership is key. Look critically at the business and ask which areas of your stack are truly differentiators because it’s only those areas that may—just may—justify a unique architecture. If you do go that route, it needs to provide huge benefits to the business to be worthwhile.

    For example, “data” is not in itself strategic to you; every company is now a data company, and the vast majority of things you do with it—how you source, store, process and distribute data—is no different from what everyone else is doing. What does matter are the data products that fuel your business and, most importantly, how fast you can build them.

    That’s why, as a developer or engineer, bespoke stacks are not your friend. Instead, architect your platform in a way that allows other developers to onboard quickly; that allows you to upgrade and swap out components easily over time and that doesn’t leave you held hostage to a developer whose main interest is retaining their value by sticking with an unnecessarily complex architecture.