devopsy.kinsta.cloud

Category: DevOps Practice

  • Sonatype Acquires MuseDev to Add Code Analysis

    Sonatype Acquires MuseDev to Add Code Analysis

    Sonatype today revealed it has acquired MuseDev, a provider of a code analysis tool, in addition to updating its Nexus platform for discovering vulnerabilities in software supply chains.

    Muse analyzes code each time a pull request is made, which makes it easier for developers to discover and address issues long before they commit code. Muse is integrated with 24 pre-configured code analyzers for GitHub, GitLab and Bitbucket repositories. It then analyzes each pull request to surface bugs, as part of automated code review, and includes analysis of interprocedural information flow and thread safety issue analysis, along with guidance to fix any bugs discovered.

    In addition to adding support for Muse, the latest edition of the Nexus platform integrates a container firewall from NueVector along with a set of tools for identifying and providing guidance on how to address misconfiguration issues that frequently arise when developers employ open source Terraform software to programmatically manage infrastructure as code.

    Finally, Sonatype added tools to provide better guidance on the implications of open source licensing terms attached to various projects, while at the same time starting its own community for organizations migrating from Bintray and JCenter, now that those tools have been sunsetted.

    Brian Fox, Sonatype CTO, said the expectation is Muse will create an opportunity to introduce the complete DevSecOps portfolio to developers as they analyze code. Most developers are keenly interested in both optimizing and securing their code. They would just prefer to achieve that goal on their own versus being informed of an issue after code has been committed to a repository managed by a larger DevOps team, noted Fox.

    An advanced development pack was added to the Nexus platform last year to make it easier to surface dependencies in code that are vulnerable to threats. It applies machine and deep learning algorithms to automatically identify and block software supply chain attacks based on typosquatting and malicious code injections. Collectively, these tools give developers control over third-party open source code, first-party source code, infrastructure as code and containerized code, Fox said.

    Sonatype claims it now counts 70% of the Fortune 100 as customers, and supports more than 2,000 commercial engineering teams. In 2020, it claimed it saw a 35% annual growth in Nexus repository installs, which now stands at more than 250,000 instances employed by nearly 15 million developers.

    Overall, it’s not clear how much progress is being made toward embracing DevSecOps best practices. Sonatype, however. is making a case for putting DevSecOps tools directly into the hands of developers to address software supply chain security. In the wake of some recent high-profile breaches involving software supply chains, that approach may soon gain additional traction. After all, for all the encouragement being provided by application development team leaders these days, there’s nothing quite as effective as providing developers with a set of tools that enables them to achieve that goal on their own terms versus requiring them to follow a set of procedures dictated by a DevOps management team.

  • Measuring the Value of DevOps-as-a-Service (DaaS)

    Measuring the Value of DevOps-as-a-Service (DaaS)

    The late great W. Edwards Deming, statistician, professor, author, lecturer, and consultant said, “In God we trust. All others must bring data.” The point is, if you want people to trust your information, then you need to present measured data. This is true for countless circumstances. As a DevOps consultant, I am often asked by my clients what data should be used to show the value of their DevOps capabilities, in a way that is acceptable to the stakeholders of those DevOps capabilities. A lot of excellent research results, published by DORA and others, describes metrics that demonstrate the value that DevOps provides to application development. More and more enterprises are embracing the idea of providing DevOps capabilities, such as pipelines, tools, hardened containers and tech stacks, as a centralized service to support their many application teams. The question then becomes, “What metrics should be used to show the value of DevOps-as-a-Service (DaaS)?”

    Before measuring the value of anything, you need to know two things: “What is it?” and “Why is it important?”

    What is DevOps-as-a-Service?

    In high-level terms, DaaS is DevOps capabilities made available to the users of those capabilities through portals and APIs. There is no standard set of services; DevOps capabilities offered vary from organization to organization. In my SlideShare, DevOps-as-a-Service (DaaS) Values, I illustrate a generic blueprint for DaaS.

    The DaaS blueprint has four layers, as follows:

    • Stakeholders – There are three types of stakeholders: business, app-dev users and DaaS teams.
    • Service access layer – This layer consists of user portals, APIs, and access security.
    • Service layer – The service layer consists of user services and management services. User services include things such as repositories, CI/CD pipelines, sandboxes, test data, configuration data, guides and training. Management services include service catalogs, issues and change requests, SLOs, SLIs, governance policies, experience monitoring and automated tasks.
    • Resources layer – This layer includes application stacks, infrastructure resources, tools, infrastructure code and topology data.

    Why is DaaS Important?

    The value of DaaS, as perceived by different stakeholders, can be summarized as follows:

    • Business – DaaS makes the business competitive.
    • App-dev Users – DaaS improves application development.
    • DaaS team(s) – DaaS improves management of DevOps capabilities.

    What Service Level Objectives (SLOs) Demonstrate the Value of DaaS?

    The SLOs need to be chosen to best fit the requirements of each application. The following are examples.

    Business value can be demonstrated by:
    ROI based on tracking cost savings relative to a benchmark comparing prior non-DaaS, trend cost per developer and per release.
    SLO agility of services – Lead time for changes, and frequency of releases.
    SLO reliability of services – change failure rate, time to restore services.
    SLO governance, compliance and security scores.

    App-dev user value can be demonstrated by:
    SLO availability of DaaS, service level indicators (SLIs) depend on service components and architecture.
    SLO service response time.
    SLO experience scores.
    SLO change request response time.

    DaaS team value can be demonstrated by:
    SLO scope of applications – number of apps supported, number of users per month.
    SLO number of requests served per month.
    SLO critical events per month.

    How Does an Organization Develop DaaS Metrics?

    Start by picking an application that is supported by DaaS. Do not try to boil the ocean and measure all applications at once. Then, appoint a DaaS SLO/SLI team to work on metrics development for the application. The team should include stakeholders from business, app-dev and DaaS support teams associated with the application.

    DaaS SLO/SLI team members determine the SLOs of DaaS that each class of stakeholder values the most, and then decomposes the services to identify sources of data that will form the most important service level indicators (SLIs).

    With the SLIs in hand, instrumentation, telemetry and analysis requirements can be determined and used to design monitoring agents, communication, analyzers, alerts and dashboards.

    Then, roles, policies, playbooks and responsibilities will complete the solution. Test the solution for the chosen application. As you gain confidence in the solution, adapt and expand the solution for use with other applications.

    Continue to monitor the progress and performance of the SLO metrics, and continuously improve them with experience and as newer technologies become available.

    What This Means

    DevOps-as-a-Service provides many valuable capabilities to business, app-dev users and DaaS teams. Metrics that demonstrate value to each class of stakeholder should be determined for each application that consumes the capabilities offered by each unique DaaS. A DaaS SLO/SLI team can determine the best metrics and system design for the implementation for an example application. The SLOs can be adapted and evolved for other applications in the organization, and used as a basis for continuous improvement.

  • The Importance of Incorporating DesignOps Into DevOps

    The Importance of Incorporating DesignOps Into DevOps

    Design is a critical software project element that often goes unnoticed by those writing code. After all, coders and implementers tend to take the design process for granted and leave the look and feel of applications to the designers. In the days of waterfall based development, design was one of the first steps in the development sequence of creating a new application. However, with Agile and DevOps development practices, design became disconnected from the overall process.

    “If you look at the full spectrum of AppDev, it becomes obvious that there are many elements that should be part of DevOps. Take, for example, cyber – today, there is a movement to integrate cyber into the DevOps pipeline, which the industry has started to call DevSecOps,” said Mark Troester, VP of strategy at Progress, a software company offering tools for developers and IT. “To make design a part of the iterative process of DevOps, teams need to think about including designers during the complete development process, something along the lines of how cyber has become part of the iterative process. In other words, teams should be embracing the concept of DesignOps,” Troester said.

    To understand the challenges this creates for those looking to include design as part of the DevOps process, one only needs to look at how design, specifically user experience/customer experience (UX/CX) design, was done in the past. Designers would use tools such as InVision Studio, Axure, Proto.IO, Adobe XD, or one of the many other tools available, to prototype the look and feel of an application, or, more specifically, the end-user experience. The term DesignOps was coined to encapsulate the processes around design. However, DesignOps solely focused on the process of design, such as design workflows, design tools, design governance, team management, budgets and so on. 

    The ultimate goal of DesignOps was design creation; that design was then passed on to those who would write the code and develop the logic to bring that design to fruition in the form of an application. DesignOps worked rather well when an organization was adhering to waterfall development methodologies, where each step was clearly defined and the goal was to deliver a complete, finalized application to those requesting it.

    “Larger organizations were able to field large design teams and could afford to implement a complete design system. When you have 50 or more designers, a complete design system becomes critical to manage design,” said Troester. “However, as AppDev has evolved into a more agile process, especially under DevOps, the disconnect between design and development became a significant problem.”

    What Troester is alluding to is the reiterative development cycle used by DevOps teams, where applications are delivered in pieces and then further modified based on user feedback or business operational changes. Businesses today are embracing agile concepts, so they can react much faster to changing market conditions. This, in turn, means modifying applications as quickly as possible to maintain a competitive advantage. Currently, design remains somewhat siloed, and can become a bottleneck for implementing changes if the UX/CX must be modified to address those needed changes. 

    “With the rise of DevOps, teams need to start thinking of reiterative processes, where DesignOps integrates with the DevOps team, so there is collaboration between the designers and the developers, allowing design changes to happen quickly,” Troester said. “Yet, there are still remaining challenges – some created by the COVID-19 crisis – where designers and developers may not even be in the same building, much less the same time zone,” he said.

    To Troester’s point, the COVID-19 crisis has forced businesses to evolve in ways that were once thought impossible. Developers, designers and even operations teams and individuals may no longer be sharing office space, and might be distributed across multiple locations. That complication will drive the need for better collaboration tools that bring design into the DevOps cycle.

    “DevOps teams, as well as DesignOps teams, should be looking into technologies that can bridge the designer/developer divide. It will be critical in the near future to facilitate collaboration between design and development,” Troester said. “Perhaps, the answer lies with building some type of bi-directional API into design systems, which can play a key role in the implementation and support of design-first experiences,” he added.

    Existing solutions to the current divide seem to be few and far between. But, that does not mean DesignOps and DevOps teams should not be currently looking for a solution to what will become a major problem as DevOps accelerates in the enterprise.                  

  • LinearB Raises $16M for Software Delivery Intelligence

    LinearB Raises $16M for Software Delivery Intelligence

    LinearB announced today it has raised an additional $16 million in funding to advance adoption of a software delivery intelligence platform.

    Ori Keren, LinearB CEO, said rather than relying on antiquated processes that require software development teams to track progress by reentering data into a project management application, it’s now possible to automatically collect all the relevant data required from the tools and platforms being employed to build software.

    The LinearB platform correlates DevOps pipeline data gathered from code, Git, projects and releases to provide visibility and context to every member of the development team. LinearB employs machine learning algorithms to construct and correlate that data, in real-time, to provide accurate insights into project status. That’s critical, because it empowers every individual on that team to make informed decisions about which tasks to prioritize without waiting for an explicit directive, Keren said.

    LinearB will, in part, use the funding to build integrations with a wider range of DevOps platforms and tools, Keren added.

    Software delivery intelligence differs from value stream management (VSM) platforms; though they provide similar insights, those surfaced by LinearB are aimed at developers rather than C-level executives trying to optimize deployment of limited resources across multiple teams, said Keren. In many ways, the data collected by LinearB could be employed to better inform a VSM platform, Keren added.

    LinearB, in contrast, makes it easier to identify which teams and projects need additional help in near-real-time using metrics surfaced in a dashboard that track, for example, cycle times. The goal is to provide insights that make it easier to predict and eliminate project delays, remediate high-risk code before it is included in a production environment, eliminate bottlenecks and make certain the focus of the DevOps team remains on tasks of the highest priority.

    More than 1,500 software development teams are already using LinearB as an alternative to traditional project management applications. Most of those customers are building commercial applications, so the goal now is to extend the reach of LinearB into the enterprise, said Keren.

    LinearB, of course, doesn’t eliminate the need for project managers. However, it does eliminate the need for project managers to nag developers to update a project management application. Given the rate at which application development projects are now updated, much of the data in a project management application is outdated shortly after it is entered.

    There’s a lot of pressure on application development teams these days. Many projects have been accelerated as part of an effort to more rapidly embrace digital business transformation in the wake of the economic downturn brought on by the COVID-19 pandemic. The challenge for many developers is they, too, are working from home to help combat the spread of the virus. That often results in interactions with managers over video conferencing calls that lack context. Having access to a software development intelligence platform should enable everyone with a vested interest in application deployment speed to make more informed decisions faster.

  • 5 Steps to Successful DevOps Culture

    5 Steps to Successful DevOps Culture

    The late business management guru Peter Drucker coined the phrase, “Culture eats strategy for breakfast,” and the phrase has endured through the years. Your people must be passionate and enthusiastic about executing business strategy, or it – and your business – is doomed to fail. But remember that company culture itself needs a strategy to complement the business.

    DevOps, too, will not succeed without the right culture. As I said in my book “Engineering DevOps,” “Culture is a door to Engineering DevOps,” because if leaders and people do not work in accordance with DevOps principles, then no tool or engineering process will be sufficient to succeed with DevOps. In my (and others’) experience, culture challenges are the No. 1 obstacle that many organizations need to overcome to succeed with DevOps. But changing culture is not easy. Values, beliefs, behaviors and work habits are deeply ingrained in organizations. Organizational inertia and deep-rooted habits are much harder to change than tools or process changes. Transforming organizational culture for success with DevOps requires a strategy, goals and sustained commitment.

    In her article, “5 Steps to Change Your Company Culture,” Kate Heinz prescribes five practical steps for cultural change which also are highly applicable to DevOps culture transformation.

    Step 1: Determine Core Values

    Start by clarifying the values that are most important to accomplishing the long-term strategic goals of the organization. With DevOps, typical strategic core values could be responsiveness to customer needs, speed of innovation, technology leadership and software products and services resiliency, for example. Collaborate with all the key stakeholders including executives, long-term employees, sales teams, key suppliers and HR to obtain alignment on core values before executing the next steps.

    Step 2: Set Your Culture Goals

    In his article, “7 Key Principles for a Successful DevOps Culture,” Kaya Ismail identified seven principles that can form a framework for determining your DevOps culture goals.

    1. Foster a Collaborative Environment – Break down silos to bring people, processes and tools towards a unified focus on the customer.
    2. End-to-End Responsibility – All people in the value stream need to be fully accountable for the application and outcomes from beginning to end.
    3. Encourage Continuous Improvement – Optimize value streams using integration toolchains and automated processes to realize quick lead times and fast customer feedback to continuously improve performance.
    4. Automate Almost Everything – Automated CI/CD processes, infrastructure provisioning, testing, deployment and security compliance require a skilled workforce.
    5. Focus on Customer Needs – DevOps teams must continuously pay close attention to ever-changing consumer demands and be ready to respond quickly.
    6. Embrace Failure and Learn – A “climate for learning” in which failures are viewed as opportunities to learn makes people feel safe and empowered to transform their work.
    7. Unite Teams and Expertise – The scope and complexity of DevOps requires team members to share knowledge and cross-train so they are well-rounded and have a balanced set of skills.

    Step 3: Assess Your Existing Culture

    Conduct interviews and surveys with employees and key stakeholders to determine where there are deviances from the culture goals set in Step 2. Communicate an analysis of the assessment findings to all stakeholders, pointing out important gaps, areas for improvement and the general plan to close the gaps.

    Step 4: Map Out the Plan

    Develop a roadmap to address the gaps identified in Step 3. Include specific tasks, timelines, benchmarks and responsibilities. Tasks and items may include targeted workshops, training, selective new hires, organizational changes and, most importantly, rewards and consequences – the real drivers of behavior and culture. One caution about training: as Thane Bellomo explained in his article, “Training Doesn’t Change Culture,” training alone does not have the power to fundamentally change culture.

    Step 5: Evaluate Your Progress

    Communication, surveys and interviews are important so individuals can provide feedback and track progress. Use the feedback you receive to verify that culture goals are being improved upon, and to reinforce to everyone that the culture changes are important. Use DevOps metrics such as lead time, release frequency and time-to-restore service to demonstrate that the culture changes are helping the organization achieve the strategic values set in Step 1. As you monitor progress, be mindful of pitfalls. As Larry Senn explained, “The Top 4 Reasons Culture Change Initiatives Fail” are: lack of consistent leadership from top management, inadequate personal involvement, disconnected initiatives and lack of top-to-bottom momentum. Be prepared to make corrections based on the evaluations to ensure your culture transformation stays true to your DevOps strategic values and culture goals.

    What This All Means

    Success with DevOps requires the right kind of culture. Changing culture is no small feat, but it can be done if approached strategically. The five steps described in this article provide a framework for organizational culture change that supports DevOps.

  • Linux Foundation Project Secures Software Supply Chains

    Linux Foundation Project Secures Software Supply Chains

    The Linux Foundation today embraced a sigstore project founded by Red Hat, Google and Purdue University to make it simpler for developers to employ cryptographic software, enabled by transparency log technologies, to secure software supply chains.

    Luke Hinds, security engineering lead in the office of the CTO at Red Hat, said he and Dan Lorenc, a staff software engineer for Google, led a small team creating a lightweight mechanism for signing software artifacts, such as release files, container images and binaries, in way that creates an immutable, tamper-proof log.

    Now, the goal is to take that project to the next level by contributing it to the Linux Foundation, at a time when the focus on software supply chain security has increased in the wake of a spate of high-profile breaches. As part of that effort, a free service, based on sigstore, will also be made available by the Linux Foundation to make the cryptographic software easily accessible to a wide range of developers.

    Hinds said one of the reasons that artifacts are not signed by developers is that the current processes employed are too cumbersome. The cryptographic software is designed to be integrated as an extension of existing workflows, Hinds noted.

    Because the log created is immutable, everyone participating in the software development process, including auditors, can now transparently track chain of custody for any software artifact, Hinds said. At its core, sigstore creates a set of short-lived ephemeral keys with a trust root based on an auditable public log.

    Existing approaches rely on digests stored on insecure systems that are susceptible to tampering. Cybercriminals can, for example, swap out digests if they can gain access to compromised credentials. Those types of targeted attacks can be thwarted using sigstore, because the log management system that underpins it is immutable, said Hinds.

    Of course, no system for securing software supply chains is going to be useful if developers won’t employ it. Key management, key compromise/revocation and the distribution of public keys and artifact digests are challenging to manage. Developers, then, need to determine which keys to trust in addition to learning how to sign them when different platforms are employed. Digests and public keys are also often distributed across vulnerable websites susceptible to hacks, or stored as a README file on a public git repository.

    As organizations embrace DevSecOps best practices, it’s only a matter of time before the entire software supply chain comes under greater scrutiny. Ultimately, the issue will come down to finding a way to secure those supply chains, either from the top down by deploying and maintaining a platform, or simply making it easier for developers to better secure artifacts from the bottom up.

    Regardless of approach, the signing of software artifacts should, one way or another, become more routine in the weeks and months ahead. The biggest challenge will be overcoming the simple inertia that currently exists among developers who routinely ignore the whole process.

  • GitGuardian Reports Careless Handling of Application Secrets

    GitGuardian Reports Careless Handling of Application Secrets

    A new report, the 2021 State of Secrets Sprawl on GitHub, published today by GitGuardian, a provider of a tool for monitoring usage of application secrets, suggests developers are not especially good at keeping those secrets safe.

    Based on an analysis of every single commit made to GitHub, the report finds there has been a 20% year-over-year increase in the number of secrets – such as application programming interface (API) keys, private keys, certificates, usernames and passwords –  discovered on a public repository.

    Developers, as they build an application, make secrets easily accessible as plain text for the sake of convenience. However, when those applications are deployed in a production environment, developers often forget to delete the secrets they stored in GitHub.

    GitGuardian CEO Jeremey Thomas noted that cybercriminals are now actively scanning repositories looking for secrets that would enable them to compromise applications.

    As the number of application development projects being launched increased, so too did the number of application secrets being stored in repositories like GitHub. The GitGuardian report notes 60 million repositories were created in the last year, representing a 35% increase over the previous year. GitGuardian claims it detected more than 5,000 application secrets per day. Most of those secrets have been detected in repositories that belong to developers (85%), while the other 15% belonged to organizations. The top four countries where secrets were disclosed were India, Brazil, the United States and Nigeria.

    GitGuardian also notes it sent out 937,539 pro bono alerts to 558,086 developers last year that informed developers that application secrets were exposed. The mean time for GitGuardian’s algorithm to detect a leak is four seconds. The average mean time to response of a developer when informed of the issues is about 25 minutes, the report noted.

    As a best practice, GitGuardian recommends not storing unencrypted secrets in git repositories, and that unencrypted secrets should never be shared via messaging systems like Slack. DevOps teams should also restrict API access and permissions.

    As more attention is paid to how software supply chains are managed in the wake of some recent, high-profile breaches, there is no doubt the number of audits conducted will increase. Many of those auditors will specifically be trying to ascertain how application secrets are being managed. Developers may have to get used to a new, higher level of scrutiny, especially as DevSecOps practices become more widely adopted.

    In the meantime, DevOps teams may want to review their dependency on public repositories. There are plenty of private repository options that provide most of the same source code sharing capabilities as a public repository. That may mean someone on the DevOps team is required to manage the private repository, but, given the fact that many regulatory bodies will soon be looking to make an example of organizations that don’t secure application secrets, chances are good those costs will be trivial compared to a fine that would be levied.

  • Kong Integrates Service Mesh with OPA

    Kong Integrates Service Mesh with OPA

    Kong, Inc. today announced an update to Kong Mesh, built on top of open source Kuma service mesh, that adds support for Open Policy Agent (OPA) software that enables DevOps teams to manage compliance as code.

    Kuma and OPA are both being developed under the auspices of the Cloud Native Computing Foundation (CNCF). Kuma, unlike other service mesh offerings, provides connectivity across both microservices-based applications running on Kubernetes platform and monolithic applications deployed on virtual machines.

    In addition to adding support for OPA, the 1.2 release of Kong Mesh adds authentication mechanisms between the global and remote control planes operating across multiple zones. That functionality extends the Kuma Discovery Service (KDS) to support multiple authentication mechanisms.

    Finally, Kong has added support for the federal information processing standard (FIPS).

    Like the Istio service mesh, advanced by Google and IBM for Kubernetes environments, Kuma is based on open source Envoy proxy software. In contrast, however, Kuma is both lighter-weight and designed to span both cloud-native and legacy application environments.

    Marco Palladino, CTO at Kong, said OPA support will be a significant advance for both classes of workloads as organizations start to employ Kuma within the context of zero-trust computing environments. As DevSecOps processes are extended to include managing compliance as code, OPA will provide a programmable mechanism for ensuring that policies are not only properly implemented, but can also be easily audited, Palladino said.

    In general, an effort led by Kong, VMware and others to elevate the service mesh conversations to more broadly address application connectivity is underway. A service mesh, at its core, provides the means for integrating hundreds of application programming interfaces (APIs) at scale. It also provides a level of abstraction above the network layer that makes it easier to connect applications across a heterogeneous networking environment.

    Kong is making a case for a lighter-weight approach that spans multiple platforms and is highly extensible. As such, third-party vendors and IT teams can add extensions to Kuma as they see fit.

    Ultimately, Palladino said, the time has come to take cloud computing to the next level. If all organizations wind up doing is lifting and shifting monolithic applications to public clouds, an opportunity to advance the state of IT will have been lost.

    Of course, it’s still early days as far as adoption of service meshes are concerned. Most organizations today are relying on API gateways, proxy software or ingress controllers to manage APIs. It’s not clear to what degree a service mesh may remove the need for some of that existing application connectivity middleware.

    Regardless of which approach is ultimately favored, it’s apparent that, as microservices-based applications become more widely deployed, a different approach to application connectivity will be required. Monolithic applications, however, are not going away. In some cases, a monolithic application might be carved up into a smaller set of microservices, but it’s unlikely that every existing monolithic application will lend itself to that architecture any time soon.

  • Introduction to Materialized Views

    Introduction to Materialized Views

    More often than not, the types of queries that use materialized views are both faster, and use fewer resources than those that are still processing queries directly to base tables. But the key to understanding the potential that materialized views bring with them involves first understanding as much about them as possible.

    What is a Materialized View?

    In the world of computing, a materialized view is a database object made up of any results that were obtained from a query. Note that this could be a local copy of data that is also stored in a separate remote location, or it could have been derived from a join result. Sometimes, a materialized view could even be created after a user performed an aggregate function, thus allowing it to act as a summary of that database information.

    Oracle Database first made materialized views available in version 8i of their software, and it has been a part of every release since. Just a few of the environments that support materialized views include, but are not limited to, others like SQL Server, BigQuery, PostgreSQL and Sybase SQL Anywhere.

    View vs. Materialized View

    What’s the difference between a regular view versus a materialized view? There are a number of important things you’ll need to keep in mind.

    Chief among them, the fact that a view itself is a precise type of virtual table created via the “Create View” command. As the name suggests, it collects all data obtained from any relevant query that you’ve executed. A view will always be compiled every time it is either used or accessed in any way. This is how a view is able to make sure that you’re always getting the most updated data, no matter what – it’s literally designed to do that.

    If you make any changes to the data contained inside that view, it will get pushed out to the table in which it originated. Along the same lines, any changes made to that base table are automatically pushed to the view so that the same information is in both places at once.

    Because of this extra step, a view will always be a bit slower than a materialized view when it comes to performance. The upside is that this really doesn’t require much in the way of storage space. But it does mean that you’ll be sacrificing a bit in terms of performance.

    With a materialized view, you’re talking about a physical copy of the information contained in that original table. This means that it isn’t updated every time you interact with it – you have to go out of your way to update it manually. Or, if you’d prefer, you can have it update using pre-defined triggers.

    Because of this, a materialized view is always going to respond much faster than its traditional counterpart. However, if you don’t make an effort to confirm that it has been updated, you also run the risk of retrieving stale data.

    Why Use a Materialized View?

    So at this point, you’re likely asking yourself, “Why use a materialized view in the first place? What can this do for me that a simple view cannot?” The key thing to understand has to do with the costs that you accrue whenever a database is queried.

    Even a simple query still needs to be parsed, validated, planned, optimized and executed. In an operational sense, this equates to CPU time. This means memory usage. This even directly impacts your opportunity costs as time goes on. All of this adds up – especially as the application you’re working with continues to evolve and become more resource-intensive.

    Savvy developers are always looking for a way to cut costs, and materialized views are a big part of how they’re able to do it. Remember that the results obtained are always kept in memory – meaning that they are only updated when absolutely needed. This is the opposite of what happens when you query tables or use logical views; those results are updated constantly. Therefore, using materialized views is a great way to significantly reduce your costs without impacting performance in a way that will also affect your end users.

    Materialized Views Best Practices

    For best results, you should always make sure that your materialized view reflects query patterns against the original table you’re working with. Don’t just create a materialized view for every iteration of a query. Instead, create one to help focus on a broader set of queries.

    Likewise, if the base table you’re working with is partitioned, your materialized view may grow to a size larger than you’re comfortable with. Therefore, it should be partitioned, as well, to preserve the performance benefits it is supposed to bring with it.

  • IaC Frameworks: Vendor-Specific or Multi-Cloud?

    IaC Frameworks: Vendor-Specific or Multi-Cloud?

    Over the past few years, I have been watching the shift to infrastructure as code (IaC) frameworks closely. With the shift from monolithic to microservices, and with the shift from mostly virtual machines (VMs) to cloud-native architecture, applications are much more complicated, making it much more important to automate infrastructure to respond with speed.

    In that sense, I refer to IaC as “The third cloud revolution.” Looking at the history of cloud transformation, the shift from physical servers to virtual machines (led by VMware about 20 years ago) would be the first cloud revolution. The move from virtualized data centers to the public cloud (led by AWS about 10 years ago) would be the second cloud revolution. With the shift to public clouds, people were suddenly able to click buttons to provision new resources, including “hardware.”

    In the last few years however, we see fewer clicks and more code. Engineers write and maintain that IaC, and something executes that code. Engineers influence the infrastructure much more than they used to – that is the third cloud revolution. We mainly see four IaC frameworks that drove this revolution – Terraform and Pulumi (both multi-cloud), CloudFormation and ARM (vendor specific).

    I recently talked with Joe Duffy, CEO, Pulumi, Mark Russinovich, CTO, Azure and Anton Babenko, AWS community hero and Hashicorp ambassador, the head of Terraform at Microsoft, the head of Terraform at IBM Cloud and one of the founders at Gruntwork. After our brainstorming session, it’s clear we all agree that IaC will have a huge effect on cloud operations. In this blog, I’ll share my point of view regarding multi-vendor IaC versus single-vendor IaC frameworks; discuss how they compare to the differences between iOS and Android and which direction I believe will win the day.

    The big cloud vendors seem to go “all-in” and create full ecosystem, iOS-style, and they have everything working nicely with the relevant services and tools. AWS just released AWS Proton, which works with CloudFormation. They say that Terraform is on their roadmap, but time will tell. Microsoft seems to offer more solutions that are ARM-based, like Bicep. When I talk with executives at those companies, they strongly believe in the great value of this vendor lock-in/closed approach. They truly believe that this will help their customers in the best way possible. Perhaps they are right.

    On the other hand, when I talk with the community, I hear different opinions. DevOps engineers at Azure customers do not debate whether to choose Terraform or ARM. DevOps engineers at AWS customers do not debate whether to choose Cloudformation or Terraform. While this was the case three years ago, it is not, anymore. Now, the real debate is whether to choose Terraform or Pulumi. Engineers prefer a language that is multi-vendor, open-source, no vendor lock-in and that is compatible with other open source solutions built on top, like Open Policy Agent and Terragrunt, to help Terraform users. This is the Android-style approach – open platform.

    Moreover, I hear new phrases from the community that make me think the future is going in this direction. Erik Osterman at Cloud Posse started using the acronym  TACOS – Terraform Automation and Collaboration Software (see the video here). It takes time, but, going forward, developers and the community will hold the key to the future of IaC, not executives and large vendors.

    You can, of course, challenge this by replying that, “iOS is a great OS.” This is obviously true, but I think there is one big difference here as to why the Android approach is winning the IaC market. The answer lies with the end user; the Apple ecosystem is a solution for the end user and consumer, and IaC is about larger-scale infrastructure decisions. Like the shift from monolith to microservices, developers prefer small and isolated parts, not one big system. With Terraform and Pulumi, developers get more freedom. Maybe the company will move from AWS to GCP or Azure or vice versa. Maybe IT will need an Auth0 provider, as well, and will want to manage it the same way. Maybe the use of Open Policy Agent to govern deployments becomes the status quo. Developers need freedom, whereas consumers just need a great product.

    But the question has still not been answered: Terraform or Pulumi? It is still too early to tell, and, frankly, there is more than enough room in the IaC space for both. Maybe more open-source and multi-vendor frameworks will emerge and one that’s yet unknown will rule the day. Terraform provides a great ecosystem of providers, support and maintainers. Pulumi has started to provide all of those, together with its imperative approach, which seems to be a bit more appealing to developers.