devopsy.kinsta.cloud

Category: DevOps and Open Technologies

Editorial Calendar for February 2016

  • GitHub Adds Tools to Simplify Management of Software Development

    GitHub Adds Tools to Simplify Management of Software Development

    GitHub has made generally available Projects powered by GitHub Issues, a set of tools for managing application development projects based on a spreadsheet interface.

    Mario Rodriguez, vice president of product management at GitHub, said these tools enable DevOps teams to more easily filter, sort and group issues and pull requests. That capability makes it possible to, for example, configure and track cycle velocity and current work status and create visualizations using cumulative flow diagrams.

    Over the course of the rest of this year, Rodriguez said GitHub also plans to add additional capabilities including custom triggers, conditionals and action logic along with a mobile edition of Projects powered by GitHub Issues.

    The overall goal is to provide DevOps teams with a set of project management tools that enable organizations to plan where they code, said Rodriguez. In many cases, that approach eliminates the need to deploy a separate set of project management applications, he added.

    As more organizations become dependent on software to drive digital business processes, it’s become apparent that managing all the ongoing application development projects has become a significant challenge. GitHub has been developing Projects powered by GitHub Issues to provide organizations with a set of tools that can be accessed not just by DevOps teams but also by application owners and business leaders to track the status of various projects, said Rodriguez.

    Most organizations have not reached the point where revenue growth is completely dependent on the rate at which software is developed and deployed, he noted. However, as DevOps teams more regularly engage other departments in the business, Projects powered by GitHub Issues can provide anyone with access to a spreadsheet-like application that is simple to use, said Rodriguez.

    In general, GitHub is committed to improving developer experiences at a time when organizations are competing for the best talent available, added Rodriguez. If salary and benefits are equal, the decision to work for one company versus another comes down to the quality of the application development experience, he noted.

    As the provider of the most widely-used code repository, GitHub has been extending the reach and capabilities of its platform to address a range of DevOps workflow issues. The issue that DevOps teams need to address is how to anchor those workflows to a code repository versus the continuous integration/continuous delivery (CI/CD) platforms they employ to build and deploy applications.

    Regardless of approach, there’s no doubt that DevOps teams can now construct more scalable workflows. That’s critical, because the number of concurrent application development initiatives that organizations are managing simultaneously has increased substantially in recent years. The underlying platforms used to manage those projects need to be extensible to prevent workflows from becoming overly rigid, said Rodriguez.

    It’s not clear to what degree business leaders appreciate the value of DevOps workflows, but as business processes continue to evolve, the ability to effectively compete could rest on the agility of an application development team.

  • Armory Extends Scope of Spinnaker CD Distribution

    Armory Extends Scope of Spinnaker CD Distribution

    Armory today updated both the self-hosted and managed editions of its distribution of the open source Spinnaker continuous delivery (CD) platform. The updates support concurrent pipelines and provide tighter integration with open source Terraform infrastructure-as-code (IaC) to better enforce policies.

    Previously, the two platforms were known as Armory Enterprise and Armory Enterprise Managed. They are now being rechristened; Armory Continuous Deployment Self-Hosted becomes Armory Continuous Deployment Managed in part to provide more differentiation from a CD cloud service the company rolled out last month that is not based on Spinnaker.

    Adam Frank, vice president of product for Armory, said other capabilities added to the platform include additional testing tools for contributions to the Spinnaker platform as well as enhanced plug-ins and remediation capabilities for Common Vulnerabilities and Exposures (CVEs) that might impact the Spinnaker platform.

    Despite launching a declarative CD-as-a-service platform that is not based on Spinnaker, Armory remains committed to supporting organizations that prefer to continue to rely on the open source platform, said Frank.

    The number of organizations that routinely make use of a dedicated CD platform is still relatively small. While organizations have been employing continuous integration (CI) within a CI/CD platform, making use of the CD element of those platforms has proven challenging. Each platform used to run applications has been unique, so automating delivery has been problematic. CI platforms, in some cases, have been extended using custom scripts to automate application delivery, but those efforts don’t typically scale across an extended enterprise. Many organizations that attempted to extend the open source Jenkins CI/CD platform, for example, are now looking to more loosely couple CI and CD processes to make DevOps workflows less brittle, noted Frank.

    Armory is making a case for a distinct CD platform at a time when more organizations are starting to build microservices-based applications on Kubernetes clusters that expose a standard set of application programming interfaces (APIs) regardless of what infrastructure they are deployed on. In many cases, those organizations now need to manage multiple pipelines in parallel as application environments become more complex rather than relying on a single pipeline, noted Frank.

    There’s no doubt that as the pace of application development continues to accelerate, the need to automate CD will become a more pressing issue. DevOps teams are releasing a larger number of smaller updates to applications and there’s also a need to be able to easily roll back those updates in case an unexpected issue is encountered, said Frank.

    It’s not clear yet how many organizations will be adopting dedicated CD platforms, but as application development continues to evolve many DevOps teams will, at the very least, be revisiting their existing processes. At the same time, there are also many organizations that are just beginning their DevOps journey. Most of those organizations are not committed to one CI/CD approach versus another.

    Regardless of approach, however, the expectation in the era of digital business transformation is organizations will be able to deliver software updates in a timely manner. The only way to achieve that goal, however, will be to have some type of CD capability that reliably operates at the scale required by organizations that need to operate like a true software company.

  • Palo Alto Networks Extends Checkov Tool for Securing Infrastructure

    Palo Alto Networks Extends Checkov Tool for Securing Infrastructure

    Palo Alto Networks has added support for GitHub Actions, GitLab Runners, CircleCI and Argo Workflows to Checkov, an open source tool that scans programmatically provisioned infrastructure for misconfigurations.

    Guy Eisenkot, senior director of product at Bridgecrew by Prisma Cloud at Palo Alto Networks, said the goal is to make it easier to secure configurations created using infrastructure-as-code (IaC) tools such as Terraform.

    Those additions are now available as part of a Checkov library of policies, including graph-based checks, that provide a context-aware way of identifying risks within infrastructure and application code within a DevSecOps workflow using a tool that enables IT teams to manage policy-as-code, he noted.

    Misconfigurations of cloud infrastructure have become a major issue. Typically, that infrastructure is programmatically provisioned by developers that have little to no cybersecurity expertise. As a result, cybercriminals are now more aggressively scanning for misconfigurations that they can exploit to, for example, exfiltrate data or illicitly access services via application programming interfaces (APIs). Checkov makes it easier to identify those potential security issues within the context of a DevOps workflow before cloud infrastructure is provisioned, noted Eisenkot.

    There is a lot more focus on securing software supply chains in the wake of a series of high-profile breaches. The Biden administration last year even went so far as to issue an executive order requiring federal agencies to review the security of their software supply chains. The challenge is that most organizations have yet to implement a truly developer-centric approach to ensuring application security, said Eisenkot.

    In general, cloud platforms are more secure than on-premises IT environments; however, the processes used to build and deploy cloud applications today are clearly problematic. A chronic shortage of cybersecurity staff further aggravates the issue, because most organizations are not able to keep pace with the rate at which workloads are being deployed in the cloud.

    As more organizations are also starting to embrace DevSecOps best practices, the overall state of cybersecurity should improve. The challenge is that no matter how much time and effort is made to educate developers, there will always be mistakes a cybercriminal can exploit. Policy-as-code tools like Chekhov make it much less likely those mistakes will carry over into a production environment.

    In the meantime, organizations need to work on work on bridging the long-standing divide between application development and cybersecurity teams. Historically, cybersecurity teams would aggregate vulnerabilities they discovered in spreadsheets that developers would then be asked to remediate. The issue is not only the lack of time to address those vulnerabilities but also the lack of context provided. Many of those vulnerabilities often turn out to not be applicable to the way an application is deployed. Over time, application developers start to ignore many of those requests in favor of focusing their efforts on writing additional code. Of course, the more code that is written the greater the number of vulnerabilities that need to theoretically be remediated until a vicious cycle is created.

    Of course, it’s only a matter of time before one vulnerability winds up being a critical exploit and becomes the exception to that rule.

  • API Gateway Vs. Service Mesh: What’s the Difference?

    API Gateway Vs. Service Mesh: What’s the Difference?

    In the average microservices architecture, app programs trade the stability and rigidity a call stack offers for the network’s flexibility. Consequently, latency, security and traceability issues that were previously absent become a service call concern. 

    Service mesh is the resulting solution to eliminate these issues and allow developers to focus on more important business solutions. The caveat is the functionality overlap that exists between a service mesh and an API gateway. But while the two are similar, there are also key differences between both options. Here’s what you need to know.

    Defining a Service Mesh

    A service mesh is a comprehensive architectural pattern set designed to deal with enterprise concerns like instability, debugging and security. These concerns arise when a team moves from a call stack to network calls. 

    For instance, a function call via a call stack recognizes the constant availability of the requested function. On the other end of the equation, a network call doesn’t offer this functionality. 

    A service mesh is the preferred option because it assists the system’s client endpoint in managing network instability. The primary method for solving this network instability is frontrunning retries transparent from the client app’s viewpoint. 

    The service mesh also assists the server’s endpoint by rerouting incoming requests to the server node that’s best able to deal with the incoming request. The choice of server node is a function of previously determined policies for routing incoming traffic.

    The ideal service mesh is installed with two layers: The control plane and the data plane. The latter is a proxy for any connection’s server and client endpoints. 

    The data plane also enforces policies from the former while running operation metrics to a monitoring tool. Conversely, the control plane layer manages all attending service policies alongside the data plane’s operation. 

    service mesh

    Image Source

    Key Features of a Service Mesh

    The average service mesh has a number of primary features. 

    But there are a few things that bear mentioning. Say your app development team is working on a small business phone system. This particular service implementation doesn’t need every feature listed below. It depends on your team’s specific needs and traffic setup.

    Traffic Routing

    This feature enables the service mesh to route requests based on preset configuration or a predetermined policy. The system may prioritize traffic from specific client applications. Alternatively, a service mesh can re-route traffic to different service versions for:

    • A/B testing
    • Canary release
    • Service versioning

    Observability

    Service mesh implementations feature proxies that record service calls for individual clients and services. This feature takes the manual burden of the call-logging process away from the developers. 

    Plus, teams can leverage downstream analysis and monitoring tools to report on system availability and overall performance. These monitoring tools also provide a framework for basic tracing across all call chains. 

    The observability capabilities of a service mesh extend beyond the above. For example, some coding work can allow developers to improve call chain monitoring to allow for accurate tracing for enterprise business transactions.  

    Common service mesh observability features include: 

    • Latency, error rate and throughput alerts
    • On-demand service dashboards and graphs depicting the connection between individual services
    • Tracing the journey a business transaction or request makes through the service mesh

    Image Source

    Security Policies

    An application with multiple independent services has a huge attack surface. Essentially, each independent service is a vulnerable entry point that requires protection. A service mesh has proxies on both the server and client endpoints to secure any communication between both terminals. 

    A service mesh also improves security by eliminating the dependence on developers to manually program security into every service. Instead, the server proxies will handle authorization, authentication and encryption for each independent service. The resultant effect is a zero-trust security framework.  

    Identity

    A service mesh manages who can access individual services. It also maintains a real-time log of stakeholders accessing each service and when they did so. The system validates identity through JSON Web Tokens (JWTs) while allowing authorization depending on the requesting service and the stakeholder. 

    Encryption

    Compliance, security and data protection are key considerations. Well, communication between all services in a service mesh is fully encrypted. The control plane is responsible for certificate management functionality including certificate rotation and generation. 

    Furthermore, a service mesh uses mutual TLS where both server and client endpoints whitelist the certificates that can operate on the other side of any connection. A service mesh provides robust mutual TLS authentication, allowing for seamless system authentication and encryption. 

    Resilience

    This feature defines the ability of a service mesh to insulate developers in scenarios where the principal service function goes offline. In some instances, the proxy may follow an alternative service path or switch to a backup framework. 

    For illustration, streaming giant Netflix has a personalized user recommendation feature that suggests titles to its users. If this service goes offline, a service mesh system can ensure a switch to a default recommendation feature with zero personalization. Only when this backup fails will the system return an error code. 

    A service mesh assures developers that when a service fails, the proxy has done all it can to manage communication errors. In some cases, the mesh can ensure optimal performance by switching to a different service with the lowest latency.

    Examples of configurable resilience patterns for a service mesh include 

    • Circuit breaker patterns
    • Rate limiting
    • Retry policies

    Another illustration of a service mesh’s resilience is the ability to build chaos engineering capabilities. The service mesh can inject itself into the endpoints within the network, providing the observability required to monitor a chaos test. 

    Image Source

    What an API Gateway Entails

    Whether the team is working on contract management software for small business or an expense calculator app, the choice of API can determine the results of your team’s app development efforts. But more than the choice of API, proper management is another crucial talking point. The best results come with a system that identifies incoming calls, directing them to the appropriate resource. 

    API gateway is the future of DevOps and is the solution to that need. It’s the technology framework for routing API client calls to the ideal application. It also processes the call, forwarding resulting responses to the client’s network.

    At the technical level, an API gateway isn’t a must-have. Find a developer with a digital business card, hire them, and they can always program an embedded code responsible for call request routing. The only caveat to this alternative is the manpower and resources necessary to program and insert codes for individual call requests. As such, it’s always better to have an API gateway to handle this task. 

    An API gateway is a more innovative solution because extensive coding always holds the possibility of risks and errors. Furthermore, these errors can complicate app development stages like maintenance, bug fixing and application testing. 

    The API gateway ensures all API requests reach their intended destination with zero diversion. Plus, it also offers the following added functionalities:

    • API request monitoring
    • Shifting API requests between individual nodes
    • Request balancing for multiple instance situations

    The Major Differences Between API Gateway and Service Mesh

    An important digital workplace definition that differentiates both frameworks is the placement location. The API gateway is necessarily present on public-facing systems. Other key areas where an API gateway and service mesh differ include:

    Observability

    Observing the performance of the average API gateway is a more concrete task. The framework can provide actionable data insights like how long the API takes to respond to requests, problematic features, and their impact on network traffic. 

    With the service mesh, the ability to spot performance issues is a given. However, the system can’t ascertain any effects of performance lags on end-user response and experience. Therefore, while troubleshooting is easy, gauging the extent of the problem’s impact is more complicated. 

    The Communications They Handle

    Both frameworks manage different communication nodes—one handles internal discussions and the other external ones. The API gateway is the component responsible for routing external communications. For example, the API gateway handles chatbot connections, purchase orders and visits to specific pages. 

    Conversely, the service mesh is responsible for internal communications in the system. For illustration, a discussion between multiple microservices under the same app environment will fall under the service mesh’s purview. 

    In other words, the API gateway manages client-server discussions. Meanwhile, service-service communications are for the service mesh. 

    Management

    Both frameworks may have some areas of overlap, but the amount of management each one requires isn’t one of them. API gateways are simpler to manage and develop. As such, they’re easier for your app development team to manage. 

    The deployment for an API gateway happens just once, regardless of the service app. Plus, post-deployment centralization and monitoring are easy tasks. 

    In contrast, a service mesh is more complicated and tedious to manage. Your app development team must deploy the service mesh function for individual app systems. 

    Compatible Tools

    Most API gateway tools and software resources are highly expensive. Plus, they’re compatible with every popular app architecture. 

    The other end of the equation offers more accessibility. There are countless service mesh tools and resources for varied functionality. Plus, the majority of these support tools come at zero cost because they are open source.  For example, there’s Envoy and Istio, both of which are free and open source. 

    However, you shouldn’t expect to enjoy cross-platform compatibility across these service mesh tools. For example, Istio is only compatible with Google. On the other hand, AWS App Mesh—another service mesh tool—is compatible with AWS Cloud. 

    Image Source

     

    Difference  API Gateway  Service Mesh
    Existing Functionalities
    • Useful for API requests between the client and server
    • Compatible with external API requests and internal calls. 
    • Handles internal communications 
    • Best for improving the portability of service-to-service calls 
    Digital Transformation Functionality
    • Longer app delivery cycles 
    • Zero security risks
    • Useful for effective microservices management 
    • Speeds up the network’s delivery cycle 
    • Security risks due to open loopholes when the program runs alone
    Operations Reroutes API calls present outside of the given application environment  Works within the application architecture 
    Maturity Level API gateway is mature technology. It has been a core developer tool for several years now Service mesh is still a relatively new technology and requires more research and development
    Security Processing Automated security protocol Manual security systems 
    Usage Complexity Application complexity remains constant with endpoints remaining unchanged  Business scalability can be complex as each update comes with new endpoints 

    Can Service Mesh and an API Gateway Be Used Together?

    The unlikeliest of brands often form marketing partnerships. In the same way, there are instances where using both a service mesh and an API gateway is a logical and feasible idea. 

    While differences exist between an API gateway and a service mesh, it’s not an “either/or” situation.

    Cloud contact center teams looking to streamline communications at both the internal and external levels are already leveraging both technologies. And you can also leverage both systems to achieve the following:

    Improved Innovation

    Effective API use, especially during external interactions, requires internal resource optimization, which is better when teams deploy a service mesh alongside an API gateway. 

     

    Image Source

    Accelerated Digital Transformation

    Leveraging both resources improves enterprise digital transformation rates. Plus, any application with both technologies will enjoy a unified API gateway for internal service mesh management. If there are any linked applications, the unified API makes them simpler to handle.

    Increased Security Scalability

    Scalable container security is achievable when you deploy a service mesh and API gateway together. The service mech will boost the system’s interservice connectivity while the API gateway acts as a central point for all client-server requests. 

    The API gateway also provides the much-needed technical support for increased proxy security. It also fosters quick threat detection and resolution at the proxy level. 

    API Gateway and Service Mesh are Different

    Both the API gateway and service mesh are designed to simplify communication while also reducing any coding responsibility on the app development team. While both systems can ensure network calls through an enterprise contact center reach the target destination, they’re structurally different.

    Above, we’ve highlighted the key differences between both frameworks. But despite these differences, both systems are compatible and can work together for improved app development. We recommend leveraging a service mesh and an API gateway together for better security, innovation and app scalability.

  • Majority of Open Source Devs Open to Change Jobs in 2022

    Majority of Open Source Devs Open to Change Jobs in 2022

    Open source is the foundation of most modern software applications. Now more than ever, retaining developers with experience working on and with open source projects is critical to maintaining software development agility. Yet amid the Great Resignation, it can be challenging to keep talent happy—organizations must demonstrate a commitment to employee benefits, mentorship and cutting-edge technologies to remain competitive.

    The EDB Open Source Talent Survey 2022 surveyed over 1,400 developers and DevOps professionals working on open source projects. Below, we’ll highlight the key takeaways about their volume of work, the quality of on-the-job training and top motivators that could influence workers to take a job elsewhere.

    The Great Resignation Persists

    It’s not news that IT is under mounting pressure to deliver new features and applications to meet surging digital demands. As a result, the amount of work has increased across the board. The report found that 67% of respondents said their volume of assigned work has increased in the past 12 months.

    Employer favorability has also declined slightly—28% have a less favorable view of their employer over the last 12 months. Although 46% are satisfied with their current job, 44% are satisfied but would be open to new opportunities. And 10% said they were not satisfied with their current position. Taken together, that means 54% are open to or actively looking for new roles.

    Mentoring: An Enticing Benefit

    So, what are the main drivers that would influence IT workers to change jobs? It’s no surprise that increased pay and benefits are a top reason behind most resignations. However, salary is not the only factor—43% say improved career opportunities influence a job shift, a steep rise from 21% in 2021.

    On-the-job mentorship and training are also becoming more desirable. A little over a third (38%) said that better access to mentorship would make them consider another job opportunity. This was followed by working with more cutting-edge technologies (32%), greater access to training and certifications (30%) and allotting more time for open source projects (19%).

    Interestingly, one-fifth of responders who said they were satisfied with their current employer said their organization introduced a mentoring program in the past year. This indicates that investment in employee career advancement helps increase morale and retain skilled workers.

    Career Advancement And Training

    Out of the above reasons developers are open to switching jobs, one factor in particular is increasingly important: Working with cutting-edge technologies. This figure doubled in the last year, rising from 16% to 32%. Most engineers prefer not to work with cumbersome legacy technologies or manage rising technical debt. And if an organization isn’t evolving its software infrastructure, employee skillsets become stale.

    Working with new open source or cloud-native technologies can help retain workers, but will likely require training. In terms of specific categories, 41% said that training in emerging technologies, like cloud, is a favorable factor in considering another job opportunity.

    From a security perspective, ongoing training is also vital to staying on top of emerging exploits. As a result, 43% said security training is an interesting perk when considering a job shift. These areas are followed by training on core technologies like database and infrastructure (38%) and training on compliance (37%).

    Final Thoughts

    Open source software is pervasive and at the heart of today’s digital innovation. Nearly all applications depend on some form of open source software, but it’s no less prevalent in the DevOps sphere, where open source projects are used in continuous delivery, container orchestration, observability and a multitude of other stages of the software life cycle.

    Supporting the needs of skilled labor will be necessary to leverage the benefits of open source projects. As such, organizations will require not only competitive wages but also relevant training and investment into cutting-edge technologies to retain such developers.

    Another factor that could drive workers away is unnecessary toil, which can negatively affect morale. A company’s environmental, social and governance (ESG) initiatives can also affect its talent pool. Furthermore, a commitment to high-quality developer experience can ensure the needs of developers are heard and respected.

    The EDB Open Source Talent Survey 2022 surveyed over 1,400 application developers and IT professionals from around the world. It targeted open source technology users, specifically those holding application development, information technology and operations titles. For additional insights, you can read the full report here.

  • VW CEO Fired for Dev Fails | Fiber Shortage Hits | Google Fires Blake Lemoine

    VW CEO Fired for Dev Fails | Fiber Shortage Hits | Google Fires Blake Lemoine

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: Herbert Diess is out at VW because software is hard (yo), fiber optic cable is hard to find, and the guy who said LaMDA was sentient is dumped by Google. (more…)

  • 3 New WFH and Hybrid Work Trends That YOU Need to Grok

    3 New WFH and Hybrid Work Trends That YOU Need to Grok

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week, a remote/hybrid work special: Bribes to relocate you, Fridays are forgotten, and big tech firms stop building offices. (more…)

  • Red Hat CEO: Out | Blind Users: Revolt | ARM: Google Joins Party

    Red Hat CEO: Out | Blind Users: Revolt | ARM: Google Joins Party

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: Matt Hicks is the new Red Hat CEO, visual accessibility is in focus, and Google Cloud rolls out ARM instances. (more…)

  • Styra Unfurls Cloud Service for Implementing Compliance-as-Code

    Styra Unfurls Cloud Service for Implementing Compliance-as-Code

    Styra, Inc. today launched an authorization service based on the Open Policy Agent (OPA) software that can be invoked via an application programming interface (API).

    Torin Sandall, vice president of open source for Styra, said the Styra Run cloud service will make it much simpler to embed enterprise-grade authorization capabilities within applications. Today, developers spend too much time building, maintaining and operating their own authorization platforms to manage permissions across multiple applications.

    In addition to enabling developers to access pre-built OPA logic, Sandall noted the service also offloads storage and global replication of end-user permissions data to a cloud service.

    The service itself currently runs on Amazon Web Services (AWS), but Styra plans to make the service available on other cloud platforms in the future, he added.

    OPA makes use of a general-purpose policy enforcement engine that developers can embed within applications using the Rego language. OPA is now being advanced under the auspices of the Cloud Native Computing Foundation (CNCF) and is gaining traction among cloud-native application developers. OPA provides a means to enforce compliance-as-code across a wide range of microservices that now routinely access a wide range of distributed data.

    However, Styra Run should make OPA more accessible to a wider range of organizations, especially those that are looking to shift responsibility for security and compliance further left toward developers as part of a set of DevSecOps processes, noted Sandall. Developers that don’t have a lot of compliance expertise can now rely on a cloud service that enables them to focus more of their time and effort on writing code, he added.

    The arrival of Styra Run also comes at a time when more organizations are reviewing their software supply chain processes in the wake of a series of high-profile security breaches. It’s not clear to what degree compliance and cybersecurity teams have developed an appreciation for OPA, but as more organizations embrace compliance-as-code, the pressure on those teams should decrease as those applications are inherently more secure than legacy applications.

    There will, of course, be some developers that prefer to embed OPA into applications themselves, but a cloud service accessed via API will be the path of least resistance for most developers building software-as-a-service (SaaS) applications, said Sandall. The issue may simply be making developers aware that another low-latency option exists; one that doesn’t require them to deeply master the Rego programming language for the sole purpose of achieving compliance, he added.

    Regardless of approach, the general expectation going forward is that applications will more easily be able to meet a wide range of compliance mandates without requiring IT teams to deploy and maintain dedicated compliance infrastructure. As such, there may soon come a day when any application that doesn’t use some type of compliance-as-code framework will simply be rejected out of hand.

  • The Quest To Unify With GraphQL, The Modern Data API

    The Quest To Unify With GraphQL, The Modern Data API

    There is an ongoing trend in enterprise software: Using GraphQL as an aggregation layer to unify access to underlying data sources and REST APIs. Some call this the supergraph, and others refer to it as a meta layer. The idea is that by providing a more consumable layer to a wide range of internal sources, frontend developers are better equipped to access whatever data they need when constructing applications. But the quest to unify all data sources into a single schema doesn’t come without its share of trials.

    GraphQL adoption is steadily increasing, with significant large enterprises making up some of the early adopters. For example, Netflix engineers recently shared how one approach gaining traction is “GraphQL microservices (GQLMS) as a backend platform facilitating rapid application development.” As the use of GraphQL increases across a large enterprise, though, scalability and security hurdles present themselves. These include retaining proper authorization throughout multiple layers and mitigating the risks of an inherently open schema.

    I recently met with Hasura CEO Tanmai Gopal to discuss why GraphQL should be used as a middleware element to access and expose underlying datasets and APIs. According to Gopal, GraphQL is becoming the data API for modern applications—this could be a massive positive shift. However, the potential downside is around the time involved in integration. Thus, the key will be automating the integration of data sources so that developers and DevOps teams can focus on the apps and services they’re building instead of maintaining the data model itself.

    The Underlying Issue: Data Access

    “Data access is a bottleneck,” said Gopal. “It needs to be self-service.”

    On one side, there are data producers, such as databases, internal services and external SaaS services. On the other side, there are the clients that consume this data, such as mobile apps, web apps, microservices and APIs externalized for third-party developers.

    The problem, according to Gopal, is that the link between both sides is increasingly challenging to implement. It takes time to integrate with HTTP-based APIs and parse JSON payloads. The exchange protocols are dissimilar and you must solve for performance issues, caching and concurrency in data access. What’s more, the authorization context must be maintained for all data sources. These issues are exacerbated since not all developers are accustomed to the nuances of maintaining integrations.

    Frontend developers want to communicate with a centralized data store, but also need to mix and match various sources. At this point, Gopal explained, a data access layer is necessary to handle the communication between both sides. Such a layer requires some integration automation to truly get APIs to developers at the edge. And it also requires a simple, unified schema—this is where GraphQL comes in.

    Solution: A GraphQL-Based Data API

    “GraphQL is an absolutely delightful API to consume,” said Rajoshi Ghosh, co-founder of Hasura. “Using GraphQL as a data API can remove a lot of requirements of legacy knowledge of integration systems and how to securely fetch data.”

    The concept involves using GraphQL as an intermediary data API—this would consume all sources, whether they are RESTful APIs, GraphQL services, an Amazon Aurora database or a PostgresSQL-style database—and unify them in a more accessible, pleasing construct. To create a unified API over these disparate sources, a middleware solution must map API resources to logical models and methods and add authorization rules to logical models, explained Gopal.

    Although GraphQL enables developer agility, it’s the part in the middle that’s difficult to assemble. Building these connectors from scratch requires a lot of relearning and tedium to maintain, Ghosh said. And to truly cater to all data sources, organizations will likely need to craft custom plugins for legacy sources, he added.

    Solving Unified Graph Problems

    Data-driven initiatives are a common technology goal, but the breadth of data sources is rising. GraphQL can play an important role in uniting these sources and providing flexibility for frontend developers to call precisely what they need. A unified graph is thus a boon for developer experience but will be challenging to set up from scratch.

    Arguably, authorization is the biggest challenge when approaching GraphQL as an aggregation layer for multiple APIs. Any system that unifies multiple backend APIs must ensure that only the person with the correct permissions can access underlying data. A unified GraphQL layer must retain granular permissions that are endpoint-centric and unique to the specific caller.

    “Most enterprises don’t have a good authorization story,” Gopal said. Implementing authorization for data APIs is hard, Gopal said, as it’s unlike REST where the user ID can literally be specified within the URL request. These kinds of fine-grained attribute level entitlements must be recreated as a policy engine and applied to the graph API as well. For example, Hasura implements authorization by using a token that scopes the entire experience to the particular user.

    The Future of Unified API Strategies

    Looking to the future of this area, Ghosh noted that frontend stacks are becoming more robust and code in the backend is essentially becoming business logic deployed as serverless functions. And in this new reality of increasing abstractions, the role of modeling experts will likely become more important.

    Furthermore, GraphQL must go beyond read-only capabilities to embrace another trend—real-time. Although most GraphQL APIs retrieve persistent data, GraphQL-based data brokers must also evolve to implement streaming, said Gopal, as real-time data and event-based subscriptions are becoming more common.

    Today, you probably don’t need to unify all your sources into a unified data model all at once. This activity will likely be relegated to particular business domains or products to help them accelerate and demonstrate a proof-of-concept. But in the coming years, there will be mounting pressure to adopt a unified approach, Gopal said. “In a few years, if you don’t have a unified API strategy, you’ll have a massive problem.”